projects / OpenSPARC-T2-DV / blame
| Commit | Line | Data |
|---|---|---|
| 86530b38 AT |
1 | <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> |
| 2 | <html> | |
| 3 | <head> | |
| 4 | <link rel="STYLESHEET" href="lib.css" type='text/css' /> | |
| 5 | <link rel="SHORTCUT ICON" href="../icons/pyfav.png" type="image/png" /> | |
| 6 | <link rel='start' href='../index.html' title='Python Documentation Index' /> | |
| 7 | <link rel="first" href="lib.html" title='Python Library Reference' /> | |
| 8 | <link rel='contents' href='contents.html' title="Contents" /> | |
| 9 | <link rel='index' href='genindex.html' title='Index' /> | |
| 10 | <link rel='last' href='about.html' title='About this document...' /> | |
| 11 | <link rel='help' href='about.html' title='About this document...' /> | |
| 12 | <link rel="next" href="module-symbol.html" /> | |
| 13 | <link rel="prev" href="language.html" /> | |
| 14 | <link rel="parent" href="language.html" /> | |
| 15 | <link rel="next" href="node767.html" /> | |
| 16 | <meta name='aesop' content='information' /> | |
| 17 | <title>18.1 parser -- Access Python parse trees</title> | |
| 18 | </head> | |
| 19 | <body> | |
| 20 | <DIV CLASS="navigation"> | |
| 21 | <div id='top-navigation-panel' xml:id='top-navigation-panel'> | |
| 22 | <table align="center" width="100%" cellpadding="0" cellspacing="2"> | |
| 23 | <tr> | |
| 24 | <td class='online-navigation'><a rel="prev" title="18. Python Language Services" | |
| 25 | href="language.html"><img src='../icons/previous.png' | |
| 26 | border='0' height='32' alt='Previous Page' width='32' /></A></td> | |
| 27 | <td class='online-navigation'><a rel="parent" title="18. Python Language Services" | |
| 28 | href="language.html"><img src='../icons/up.png' | |
| 29 | border='0' height='32' alt='Up One Level' width='32' /></A></td> | |
| 30 | <td class='online-navigation'><a rel="next" title="18.1.1 Creating AST Objects" | |
| 31 | href="node767.html"><img src='../icons/next.png' | |
| 32 | border='0' height='32' alt='Next Page' width='32' /></A></td> | |
| 33 | <td align="center" width="100%">Python Library Reference</td> | |
| 34 | <td class='online-navigation'><a rel="contents" title="Table of Contents" | |
| 35 | href="contents.html"><img src='../icons/contents.png' | |
| 36 | border='0' height='32' alt='Contents' width='32' /></A></td> | |
| 37 | <td class='online-navigation'><a href="modindex.html" title="Module Index"><img src='../icons/modules.png' | |
| 38 | border='0' height='32' alt='Module Index' width='32' /></a></td> | |
| 39 | <td class='online-navigation'><a rel="index" title="Index" | |
| 40 | href="genindex.html"><img src='../icons/index.png' | |
| 41 | border='0' height='32' alt='Index' width='32' /></A></td> | |
| 42 | </tr></table> | |
| 43 | <div class='online-navigation'> | |
| 44 | <b class="navlabel">Previous:</b> | |
| 45 | <a class="sectref" rel="prev" href="language.html">18. Python Language Services</A> | |
| 46 | <b class="navlabel">Up:</b> | |
| 47 | <a class="sectref" rel="parent" href="language.html">18. Python Language Services</A> | |
| 48 | <b class="navlabel">Next:</b> | |
| 49 | <a class="sectref" rel="next" href="node767.html">18.1.1 Creating AST Objects</A> | |
| 50 | </div> | |
| 51 | <hr /></div> | |
| 52 | </DIV> | |
| 53 | <!--End of Navigation Panel--> | |
| 54 | ||
| 55 | <H1><A NAME="SECTION0020100000000000000000"> | |
| 56 | 18.1 <tt class="module">parser</tt> -- | |
| 57 | Access Python parse trees</A> | |
| 58 | </H1> | |
| 59 | ||
| 60 | <P> | |
| 61 | <A NAME="module-parser"></A> | |
| 62 | ||
| 63 | <P> | |
| 64 | <a id='l2h-4937' xml:id='l2h-4937'></a> | |
| 65 | ||
| 66 | <P> | |
| 67 | The <tt class="module">parser</tt> module provides an interface to Python's internal | |
| 68 | parser and byte-code compiler. The primary purpose for this interface | |
| 69 | is to allow Python code to edit the parse tree of a Python expression | |
| 70 | and create executable code from this. This is better than trying | |
| 71 | to parse and modify an arbitrary Python code fragment as a string | |
| 72 | because parsing is performed in a manner identical to the code | |
| 73 | forming the application. It is also faster. | |
| 74 | ||
| 75 | <P> | |
| 76 | There are a few things to note about this module which are important | |
| 77 | to making use of the data structures created. This is not a tutorial | |
| 78 | on editing the parse trees for Python code, but some examples of using | |
| 79 | the <tt class="module">parser</tt> module are presented. | |
| 80 | ||
| 81 | <P> | |
| 82 | Most importantly, a good understanding of the Python grammar processed | |
| 83 | by the internal parser is required. For full information on the | |
| 84 | language syntax, refer to the <em class="citetitle"><a | |
| 85 | href="../ref/ref.html" | |
| 86 | title="Python | |
| 87 | Language Reference" | |
| 88 | >Python | |
| 89 | Language Reference</a></em>. The parser itself is created from a grammar | |
| 90 | specification defined in the file <span class="file">Grammar/Grammar</span> in the | |
| 91 | standard Python distribution. The parse trees stored in the AST | |
| 92 | objects created by this module are the actual output from the internal | |
| 93 | parser when created by the <tt class="function">expr()</tt> or <tt class="function">suite()</tt> | |
| 94 | functions, described below. The AST objects created by | |
| 95 | <tt class="function">sequence2ast()</tt> faithfully simulate those structures. Be | |
| 96 | aware that the values of the sequences which are considered | |
| 97 | ``correct'' will vary from one version of Python to another as the | |
| 98 | formal grammar for the language is revised. However, transporting | |
| 99 | code from one Python version to another as source text will always | |
| 100 | allow correct parse trees to be created in the target version, with | |
| 101 | the only restriction being that migrating to an older version of the | |
| 102 | interpreter will not support more recent language constructs. The | |
| 103 | parse trees are not typically compatible from one version to another, | |
| 104 | whereas source code has always been forward-compatible. | |
| 105 | ||
| 106 | <P> | |
| 107 | Each element of the sequences returned by <tt class="function">ast2list()</tt> or | |
| 108 | <tt class="function">ast2tuple()</tt> has a simple form. Sequences representing | |
| 109 | non-terminal elements in the grammar always have a length greater than | |
| 110 | one. The first element is an integer which identifies a production in | |
| 111 | the grammar. These integers are given symbolic names in the C header | |
| 112 | file <span class="file">Include/graminit.h</span> and the Python module | |
| 113 | <tt class="module"><a href="module-symbol.html">symbol</a></tt>. Each additional element of the sequence represents | |
| 114 | a component of the production as recognized in the input string: these | |
| 115 | are always sequences which have the same form as the parent. An | |
| 116 | important aspect of this structure which should be noted is that | |
| 117 | keywords used to identify the parent node type, such as the keyword | |
| 118 | <tt class="keyword">if</tt> in an <tt class="constant">if_stmt</tt>, are included in the node tree without | |
| 119 | any special treatment. For example, the <tt class="keyword">if</tt> keyword is | |
| 120 | represented by the tuple <code>(1, 'if')</code>, where <code>1</code> is the | |
| 121 | numeric value associated with all <tt class="constant">NAME</tt> tokens, including | |
| 122 | variable and function names defined by the user. In an alternate form | |
| 123 | returned when line number information is requested, the same token | |
| 124 | might be represented as <code>(1, 'if', 12)</code>, where the <code>12</code> | |
| 125 | represents the line number at which the terminal symbol was found. | |
| 126 | ||
| 127 | <P> | |
| 128 | Terminal elements are represented in much the same way, but without | |
| 129 | any child elements and the addition of the source text which was | |
| 130 | identified. The example of the <tt class="keyword">if</tt> keyword above is | |
| 131 | representative. The various types of terminal symbols are defined in | |
| 132 | the C header file <span class="file">Include/token.h</span> and the Python module | |
| 133 | <tt class="module"><a href="module-token.html">token</a></tt>. | |
| 134 | ||
| 135 | <P> | |
| 136 | The AST objects are not required to support the functionality of this | |
| 137 | module, but are provided for three purposes: to allow an application | |
| 138 | to amortize the cost of processing complex parse trees, to provide a | |
| 139 | parse tree representation which conserves memory space when compared | |
| 140 | to the Python list or tuple representation, and to ease the creation | |
| 141 | of additional modules in C which manipulate parse trees. A simple | |
| 142 | ``wrapper'' class may be created in Python to hide the use of AST | |
| 143 | objects. | |
| 144 | ||
| 145 | <P> | |
| 146 | The <tt class="module">parser</tt> module defines functions for a few distinct | |
| 147 | purposes. The most important purposes are to create AST objects and | |
| 148 | to convert AST objects to other representations such as parse trees | |
| 149 | and compiled code objects, but there are also functions which serve to | |
| 150 | query the type of parse tree represented by an AST object. | |
| 151 | ||
| 152 | <P> | |
| 153 | <div class="seealso"> | |
| 154 | <p class="heading">See Also:</p> | |
| 155 | ||
| 156 | <dl compact="compact" class="seemodule"> | |
| 157 | <dt>Module <b><tt class="module"><a href="module-symbol.html">symbol</a></tt>:</b> | |
| 158 | <dd>Useful constants representing internal nodes of | |
| 159 | the parse tree. | |
| 160 | </dl> | |
| 161 | <dl compact="compact" class="seemodule"> | |
| 162 | <dt>Module <b><tt class="module"><a href="module-token.html">token</a></tt>:</b> | |
| 163 | <dd>Useful constants representing leaf nodes of the | |
| 164 | parse tree and functions for testing node values. | |
| 165 | </dl> | |
| 166 | </div> | |
| 167 | ||
| 168 | <P> | |
| 169 | ||
| 170 | <p><br /></p><hr class='online-navigation' /> | |
| 171 | <div class='online-navigation'> | |
| 172 | <!--Table of Child-Links--> | |
| 173 | <A NAME="CHILD_LINKS"><STRONG>Subsections</STRONG></a> | |
| 174 | ||
| 175 | <UL CLASS="ChildLinks"> | |
| 176 | <LI><A href="node767.html">18.1.1 Creating AST Objects</a> | |
| 177 | <LI><A href="node768.html">18.1.2 Converting AST Objects</a> | |
| 178 | <LI><A href="node769.html">18.1.3 Queries on AST Objects</a> | |
| 179 | <LI><A href="node770.html">18.1.4 Exceptions and Error Handling</a> | |
| 180 | <LI><A href="node771.html">18.1.5 AST Objects</a> | |
| 181 | <LI><A href="node772.html">18.1.6 Examples</a> | |
| 182 | <UL> | |
| 183 | <LI><A href="node773.html">18.1.6.1 Emulation of <tt class="function">compile()</tt></a> | |
| 184 | <LI><A href="node774.html">18.1.6.2 Information Discovery</a> | |
| 185 | </ul></ul> | |
| 186 | <!--End of Table of Child-Links--> | |
| 187 | </div> | |
| 188 | ||
| 189 | <DIV CLASS="navigation"> | |
| 190 | <div class='online-navigation'> | |
| 191 | <p></p><hr /> | |
| 192 | <table align="center" width="100%" cellpadding="0" cellspacing="2"> | |
| 193 | <tr> | |
| 194 | <td class='online-navigation'><a rel="prev" title="18. Python Language Services" | |
| 195 | href="language.html"><img src='../icons/previous.png' | |
| 196 | border='0' height='32' alt='Previous Page' width='32' /></A></td> | |
| 197 | <td class='online-navigation'><a rel="parent" title="18. Python Language Services" | |
| 198 | href="language.html"><img src='../icons/up.png' | |
| 199 | border='0' height='32' alt='Up One Level' width='32' /></A></td> | |
| 200 | <td class='online-navigation'><a rel="next" title="18.1.1 Creating AST Objects" | |
| 201 | href="node767.html"><img src='../icons/next.png' | |
| 202 | border='0' height='32' alt='Next Page' width='32' /></A></td> | |
| 203 | <td align="center" width="100%">Python Library Reference</td> | |
| 204 | <td class='online-navigation'><a rel="contents" title="Table of Contents" | |
| 205 | href="contents.html"><img src='../icons/contents.png' | |
| 206 | border='0' height='32' alt='Contents' width='32' /></A></td> | |
| 207 | <td class='online-navigation'><a href="modindex.html" title="Module Index"><img src='../icons/modules.png' | |
| 208 | border='0' height='32' alt='Module Index' width='32' /></a></td> | |
| 209 | <td class='online-navigation'><a rel="index" title="Index" | |
| 210 | href="genindex.html"><img src='../icons/index.png' | |
| 211 | border='0' height='32' alt='Index' width='32' /></A></td> | |
| 212 | </tr></table> | |
| 213 | <div class='online-navigation'> | |
| 214 | <b class="navlabel">Previous:</b> | |
| 215 | <a class="sectref" rel="prev" href="language.html">18. Python Language Services</A> | |
| 216 | <b class="navlabel">Up:</b> | |
| 217 | <a class="sectref" rel="parent" href="language.html">18. Python Language Services</A> | |
| 218 | <b class="navlabel">Next:</b> | |
| 219 | <a class="sectref" rel="next" href="node767.html">18.1.1 Creating AST Objects</A> | |
| 220 | </div> | |
| 221 | </div> | |
| 222 | <hr /> | |
| 223 | <span class="release-info">Release 2.4.2, documentation updated on 28 September 2005.</span> | |
| 224 | </DIV> | |
| 225 | <!--End of Navigation Panel--> | |
| 226 | <ADDRESS> | |
| 227 | See <i><a href="about.html">About this document...</a></i> for information on suggesting changes. | |
| 228 | </ADDRESS> | |
| 229 | </BODY> | |
| 230 | </HTML> |