sam-t2/devtools/v8plus/html/python/lib/module-parser.html

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<link rel="STYLESHEET" href="lib.css" type='text/css' />
<link rel="SHORTCUT ICON" href="../icons/pyfav.png" type="image/png" />
<link rel='start' href='../index.html' title='Python Documentation Index' />
<link rel="first" href="lib.html" title='Python Library Reference' />
<link rel='contents' href='contents.html' title="Contents" />
<link rel='index' href='genindex.html' title='Index' />
<link rel='last' href='about.html' title='About this document...' />
<link rel='help' href='about.html' title='About this document...' />
<link rel="next" href="module-symbol.html" />
<link rel="prev" href="language.html" />
<link rel="parent" href="language.html" />
<link rel="next" href="node767.html" />
<meta name='aesop' content='information' />
<title>18.1 parser -- Access Python parse trees</title>
</head>
<body>
<DIV CLASS="navigation">
<div id='top-navigation-panel' xml:id='top-navigation-panel'>
<table align="center" width="100%" cellpadding="0" cellspacing="2">
<tr>
<td class='online-navigation'><a rel="prev" title="18. Python Language Services"
  href="language.html"><img src='../icons/previous.png'
  border='0' height='32'  alt='Previous Page' width='32' /></A></td>
<td class='online-navigation'><a rel="parent" title="18. Python Language Services"
  href="language.html"><img src='../icons/up.png'
  border='0' height='32'  alt='Up One Level' width='32' /></A></td>
<td class='online-navigation'><a rel="next" title="18.1.1 Creating AST Objects"
  href="node767.html"><img src='../icons/next.png'
  border='0' height='32'  alt='Next Page' width='32' /></A></td>
<td align="center" width="100%">Python Library Reference</td>
<td class='online-navigation'><a rel="contents" title="Table of Contents"
  href="contents.html"><img src='../icons/contents.png'
  border='0' height='32'  alt='Contents' width='32' /></A></td>
<td class='online-navigation'><a href="modindex.html" title="Module Index"><img src='../icons/modules.png'
  border='0' height='32'  alt='Module Index' width='32' /></a></td>
<td class='online-navigation'><a rel="index" title="Index"
  href="genindex.html"><img src='../icons/index.png'
  border='0' height='32'  alt='Index' width='32' /></A></td>
</tr></table>
<div class='online-navigation'>
<b class="navlabel">Previous:</b>
<a class="sectref" rel="prev" href="language.html">18. Python Language Services</A>
<b class="navlabel">Up:</b>
<a class="sectref" rel="parent" href="language.html">18. Python Language Services</A>
<b class="navlabel">Next:</b>
<a class="sectref" rel="next" href="node767.html">18.1.1 Creating AST Objects</A>
</div>
<hr /></div>
</DIV>
<!--End of Navigation Panel-->

<H1><A NAME="SECTION0020100000000000000000">
18.1 <tt class="module">parser</tt> --
         Access Python parse trees</A>
</H1>

<P>
<A NAME="module-parser"></A>

<P>
<a id='l2h-4937' xml:id='l2h-4937'></a>

<P>
The <tt class="module">parser</tt> module provides an interface to Python's internal
parser and byte-code compiler.  The primary purpose for this interface
is to allow Python code to edit the parse tree of a Python expression
and create executable code from this.  This is better than trying
to parse and modify an arbitrary Python code fragment as a string
because parsing is performed in a manner identical to the code
forming the application.  It is also faster.

<P>
There are a few things to note about this module which are important
to making use of the data structures created.  This is not a tutorial
on editing the parse trees for Python code, but some examples of using
the <tt class="module">parser</tt> module are presented.

<P>
Most importantly, a good understanding of the Python grammar processed
by the internal parser is required.  For full information on the
language syntax, refer to the <em class="citetitle"><a
 href="../ref/ref.html"
 title="Python
Language Reference"
 >Python
Language Reference</a></em>.  The parser itself is created from a grammar
specification defined in the file <span class="file">Grammar/Grammar</span> in the
standard Python distribution.  The parse trees stored in the AST
objects created by this module are the actual output from the internal
parser when created by the <tt class="function">expr()</tt> or <tt class="function">suite()</tt>
functions, described below.  The AST objects created by
<tt class="function">sequence2ast()</tt> faithfully simulate those structures.  Be
aware that the values of the sequences which are considered
``correct'' will vary from one version of Python to another as the
formal grammar for the language is revised.  However, transporting
code from one Python version to another as source text will always
allow correct parse trees to be created in the target version, with
the only restriction being that migrating to an older version of the
interpreter will not support more recent language constructs.  The
parse trees are not typically compatible from one version to another,
whereas source code has always been forward-compatible.

<P>
Each element of the sequences returned by <tt class="function">ast2list()</tt> or
<tt class="function">ast2tuple()</tt> has a simple form.  Sequences representing
non-terminal elements in the grammar always have a length greater than
one.  The first element is an integer which identifies a production in
the grammar.  These integers are given symbolic names in the C header
file <span class="file">Include/graminit.h</span> and the Python module
<tt class="module"><a href="module-symbol.html">symbol</a></tt>.  Each additional element of the sequence represents
a component of the production as recognized in the input string: these
are always sequences which have the same form as the parent.  An
important aspect of this structure which should be noted is that
keywords used to identify the parent node type, such as the keyword
<tt class="keyword">if</tt> in an <tt class="constant">if_stmt</tt>, are included in the node tree without
any special treatment.  For example, the <tt class="keyword">if</tt> keyword is
represented by the tuple <code>(1, 'if')</code>, where <code>1</code> is the
numeric value associated with all <tt class="constant">NAME</tt> tokens, including
variable and function names defined by the user.  In an alternate form
returned when line number information is requested, the same token
might be represented as <code>(1, 'if', 12)</code>, where the <code>12</code>
represents the line number at which the terminal symbol was found.

<P>
Terminal elements are represented in much the same way, but without
any child elements and the addition of the source text which was
identified.  The example of the <tt class="keyword">if</tt> keyword above is
representative.  The various types of terminal symbols are defined in
the C header file <span class="file">Include/token.h</span> and the Python module
<tt class="module"><a href="module-token.html">token</a></tt>.

<P>
The AST objects are not required to support the functionality of this
module, but are provided for three purposes: to allow an application
to amortize the cost of processing complex parse trees, to provide a
parse tree representation which conserves memory space when compared
to the Python list or tuple representation, and to ease the creation
of additional modules in C which manipulate parse trees.  A simple
``wrapper'' class may be created in Python to hide the use of AST
objects.

<P>
The <tt class="module">parser</tt> module defines functions for a few distinct
purposes.  The most important purposes are to create AST objects and
to convert AST objects to other representations such as parse trees
and compiled code objects, but there are also functions which serve to
query the type of parse tree represented by an AST object.

<P>
<div class="seealso">
  <p class="heading">See Also:</p>

  <dl compact="compact" class="seemodule">
    <dt>Module <b><tt class="module"><a href="module-symbol.html">symbol</a></tt>:</b>
    <dd>Useful constants representing internal nodes of
                     the parse tree.
  </dl>
  <dl compact="compact" class="seemodule">
    <dt>Module <b><tt class="module"><a href="module-token.html">token</a></tt>:</b>
    <dd>Useful constants representing leaf nodes of the
                    parse tree and functions for testing node values.
  </dl>
</div>

<P>

<p><br /></p><hr class='online-navigation' />
<div class='online-navigation'>
<!--Table of Child-Links-->
<A NAME="CHILD_LINKS"><STRONG>Subsections</STRONG></a>

<UL CLASS="ChildLinks">
<LI><A href="node767.html">18.1.1 Creating AST Objects</a>
<LI><A href="node768.html">18.1.2 Converting AST Objects</a>
<LI><A href="node769.html">18.1.3 Queries on AST Objects</a>
<LI><A href="node770.html">18.1.4 Exceptions and Error Handling</a>
<LI><A href="node771.html">18.1.5 AST Objects</a>
<LI><A href="node772.html">18.1.6 Examples</a>
<UL>
<LI><A href="node773.html">18.1.6.1 Emulation of <tt class="function">compile()</tt></a>
<LI><A href="node774.html">18.1.6.2 Information Discovery</a>
</ul></ul>
<!--End of Table of Child-Links-->
</div>

<DIV CLASS="navigation">
<div class='online-navigation'>
<p></p><hr />
<table align="center" width="100%" cellpadding="0" cellspacing="2">
<tr>
<td class='online-navigation'><a rel="prev" title="18. Python Language Services"
  href="language.html"><img src='../icons/previous.png'
  border='0' height='32'  alt='Previous Page' width='32' /></A></td>
<td class='online-navigation'><a rel="parent" title="18. Python Language Services"
  href="language.html"><img src='../icons/up.png'
  border='0' height='32'  alt='Up One Level' width='32' /></A></td>
<td class='online-navigation'><a rel="next" title="18.1.1 Creating AST Objects"
  href="node767.html"><img src='../icons/next.png'
  border='0' height='32'  alt='Next Page' width='32' /></A></td>
<td align="center" width="100%">Python Library Reference</td>
<td class='online-navigation'><a rel="contents" title="Table of Contents"
  href="contents.html"><img src='../icons/contents.png'
  border='0' height='32'  alt='Contents' width='32' /></A></td>
<td class='online-navigation'><a href="modindex.html" title="Module Index"><img src='../icons/modules.png'
  border='0' height='32'  alt='Module Index' width='32' /></a></td>
<td class='online-navigation'><a rel="index" title="Index"
  href="genindex.html"><img src='../icons/index.png'
  border='0' height='32'  alt='Index' width='32' /></A></td>
</tr></table>
<div class='online-navigation'>
<b class="navlabel">Previous:</b>
<a class="sectref" rel="prev" href="language.html">18. Python Language Services</A>
<b class="navlabel">Up:</b>
<a class="sectref" rel="parent" href="language.html">18. Python Language Services</A>
<b class="navlabel">Next:</b>
<a class="sectref" rel="next" href="node767.html">18.1.1 Creating AST Objects</A>
</div>
</div>
<hr />
<span class="release-info">Release 2.4.2, documentation updated on 28 September 2005.</span>
</DIV>
<!--End of Navigation Panel-->
<ADDRESS>
See <i><a href="about.html">About this document...</a></i> for information on suggesting changes.
</ADDRESS>
</BODY>
</HTML>