sam-t2/devtools/v9/html/python/ref/lexical.html

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<link rel="STYLESHEET" href="ref.css" type='text/css' />
<link rel="SHORTCUT ICON" href="../icons/pyfav.png" type="image/png" />
<link rel='start' href='../index.html' title='Python Documentation Index' />
<link rel="first" href="ref.html" title='Python Reference Manual' />
<link rel='contents' href='contents.html' title="Contents" />
<link rel='index' href='genindex.html' title='Index' />
<link rel='last' href='about.html' title='About this document...' />
<link rel='help' href='about.html' title='About this document...' />
<link rel="next" href="datamodel.html" />
<link rel="prev" href="introduction.html" />
<link rel="parent" href="ref.html" />
<link rel="next" href="line-structure.html" />
<meta name='aesop' content='information' />
<title>2. Lexical analysis</title>
</head>
<body>
<DIV CLASS="navigation">
<div id='top-navigation-panel' xml:id='top-navigation-panel'>
<table align="center" width="100%" cellpadding="0" cellspacing="2">
<tr>
<td class='online-navigation'><a rel="prev" title="1.2 Notation"
  href="notation.html"><img src='../icons/previous.png'
  border='0' height='32'  alt='Previous Page' width='32' /></A></td>
<td class='online-navigation'><a rel="parent" title="Python Reference Manual"
  href="ref.html"><img src='../icons/up.png'
  border='0' height='32'  alt='Up One Level' width='32' /></A></td>
<td class='online-navigation'><a rel="next" title="2.1 Line structure"
  href="line-structure.html"><img src='../icons/next.png'
  border='0' height='32'  alt='Next Page' width='32' /></A></td>
<td align="center" width="100%">Python Reference Manual</td>
<td class='online-navigation'><a rel="contents" title="Table of Contents"
  href="contents.html"><img src='../icons/contents.png'
  border='0' height='32'  alt='Contents' width='32' /></A></td>
<td class='online-navigation'><img src='../icons/blank.png'
  border='0' height='32'  alt='' width='32' /></td>
<td class='online-navigation'><a rel="index" title="Index"
  href="genindex.html"><img src='../icons/index.png'
  border='0' height='32'  alt='Index' width='32' /></A></td>
</tr></table>
<div class='online-navigation'>
<b class="navlabel">Previous:</b>
<a class="sectref" rel="prev" href="notation.html">1.2 Notation</A>
<b class="navlabel">Up:</b>
<a class="sectref" rel="parent" href="ref.html">Python Reference Manual</A>
<b class="navlabel">Next:</b>
<a class="sectref" rel="next" href="line-structure.html">2.1 Line structure</A>
</div>
<hr /></div>
</DIV>
<!--End of Navigation Panel-->

<H1><A NAME="SECTION004000000000000000000"></A><A NAME="lexical"></A><a id='l2h-2' xml:id='l2h-2'></a>
<BR>
2. Lexical analysis
</H1>

<P>
A Python program is read by a <em>parser</em>.  Input to the parser is a
stream of <em>tokens</em>, generated by the <em>lexical analyzer</em>.  This
chapter describes how the lexical analyzer breaks a file into tokens.

<P>
Python uses the 7-bit ASCII character set for program text.

<span class="versionnote">New in version 2.3:
An encoding declaration can be used to indicate that
string literals and comments use an encoding different from ASCII..</span>

For compatibility with older versions, Python only warns if it finds
8-bit characters; those warnings should be corrected by either declaring
an explicit encoding, or using escape sequences if those bytes are binary
data, instead of characters.

<P>
The run-time character set depends on the I/O devices connected to the
program but is generally a superset of ASCII.

<P>
<strong>Future compatibility note:</strong> It may be tempting to assume that the
character set for 8-bit characters is ISO Latin-1 (an ASCII
superset that covers most western languages that use the Latin
alphabet), but it is possible that in the future Unicode text editors
will become common.  These generally use the UTF-8 encoding, which is
also an ASCII superset, but with very different use for the
characters with ordinals 128-255.  While there is no consensus on this
subject yet, it is unwise to assume either Latin-1 or UTF-8, even
though the current implementation appears to favor Latin-1.  This
applies both to the source character set and the run-time character
set.

<P>

<p><br /></p><hr class='online-navigation' />
<div class='online-navigation'>
<!--Table of Child-Links-->
<A NAME="CHILD_LINKS"><STRONG>Subsections</STRONG></a>

<UL CLASS="ChildLinks">
<LI><A href="line-structure.html">2.1 Line structure</a>
<UL>
<LI><A href="logical.html">2.1.1 Logical lines</a>
<LI><A href="physical.html">2.1.2 Physical lines</a>
<LI><A href="comments.html">2.1.3 Comments</a>
<LI><A href="encodings.html">2.1.4 Encoding declarations</a>
<LI><A href="explicit-joining.html">2.1.5 Explicit line joining</a>
<LI><A href="implicit-joining.html">2.1.6 Implicit line joining</a>
<LI><A href="blank-lines.html">2.1.7 Blank lines</a>
<LI><A href="indentation.html">2.1.8 Indentation</a>
<LI><A href="whitespace.html">2.1.9 Whitespace between tokens</a>
</ul>
<LI><A href="other-tokens.html">2.2 Other tokens</a>
<LI><A href="identifiers.html">2.3 Identifiers and keywords</a>
<UL>
<LI><A href="keywords.html">2.3.1 Keywords</a>
<LI><A href="id-classes.html">2.3.2 Reserved classes of identifiers</a>
</ul>
<LI><A href="literals.html">2.4 Literals</a>
<UL>
<LI><A href="strings.html">2.4.1 String literals</a>
<LI><A href="string-catenation.html">2.4.2 String literal concatenation</a>
<LI><A href="numbers.html">2.4.3 Numeric literals</a>
<LI><A href="integers.html">2.4.4 Integer and long integer literals</a>
<LI><A href="floating.html">2.4.5 Floating point literals</a>
<LI><A href="imaginary.html">2.4.6 Imaginary literals</a>
</ul>
<LI><A href="operators.html">2.5 Operators</a>
<LI><A href="delimiters.html">2.6 Delimiters</a>
</ul>
<!--End of Table of Child-Links-->
</div>

<DIV CLASS="navigation">
<div class='online-navigation'>
<p></p><hr />
<table align="center" width="100%" cellpadding="0" cellspacing="2">
<tr>
<td class='online-navigation'><a rel="prev" title="1.2 Notation"
  href="notation.html"><img src='../icons/previous.png'
  border='0' height='32'  alt='Previous Page' width='32' /></A></td>
<td class='online-navigation'><a rel="parent" title="Python Reference Manual"
  href="ref.html"><img src='../icons/up.png'
  border='0' height='32'  alt='Up One Level' width='32' /></A></td>
<td class='online-navigation'><a rel="next" title="2.1 Line structure"
  href="line-structure.html"><img src='../icons/next.png'
  border='0' height='32'  alt='Next Page' width='32' /></A></td>
<td align="center" width="100%">Python Reference Manual</td>
<td class='online-navigation'><a rel="contents" title="Table of Contents"
  href="contents.html"><img src='../icons/contents.png'
  border='0' height='32'  alt='Contents' width='32' /></A></td>
<td class='online-navigation'><img src='../icons/blank.png'
  border='0' height='32'  alt='' width='32' /></td>
<td class='online-navigation'><a rel="index" title="Index"
  href="genindex.html"><img src='../icons/index.png'
  border='0' height='32'  alt='Index' width='32' /></A></td>
</tr></table>
<div class='online-navigation'>
<b class="navlabel">Previous:</b>
<a class="sectref" rel="prev" href="notation.html">1.2 Notation</A>
<b class="navlabel">Up:</b>
<a class="sectref" rel="parent" href="ref.html">Python Reference Manual</A>
<b class="navlabel">Next:</b>
<a class="sectref" rel="next" href="line-structure.html">2.1 Line structure</A>
</div>
</div>
<hr />
<span class="release-info">Release 2.4.2, documentation updated on 28 September 2005.</span>
</DIV>
<!--End of Navigation Panel-->
<ADDRESS>
See <i><a href="about.html">About this document...</a></i> for information on suggesting changes.
</ADDRESS>
</BODY>
</HTML>