The user must supply a lexical analyzer to read the input stream and communicate tokens
(with values, if desired) to the parser.
The lexical analyzer is an integer-valued function called
The function returns an integer, the
representing the kind of token read.
If there is a value associated with that token, it should be assigned
The parser and the lexical analyzer must agree on these token numbers in order for
communication between them to take place.
The numbers may be chosen by Yacc, or chosen by the user.
In either case, the ``# define'' mechanism of C is used to allow the lexical analyzer
to return these numbers symbolically.
For example, suppose that the token name DIGIT has been defined in the declarations section of the
The relevant portion of the lexical analyzer might look like:
The intent is to return a token number of DIGIT, and a value equal to the numerical value of the
Provided that the lexical analyzer code is placed in the programs section of the specification file,
the identifier DIGIT will be defined as the token number associated
This mechanism leads to clear,
easily modified lexical analyzers; the only pitfall is the need
to avoid using any token names in the grammar that are reserved
or significant in C or the parser; for example, the use of
will almost certainly cause severe
difficulties when the lexical analyzer is compiled.
is reserved for error handling, and should not be used naively
As mentioned above, the token numbers may be chosen by Yacc or by the user.
In the default situation, the numbers are chosen by Yacc.
The default token number for a literal
character is the numerical value of the character in the local character set.
Other names are assigned token numbers
To assign a token number to a token (including literals),
the first appearance of the token name or literal
in the declarations section
can be immediately followed by
This integer is taken to be the token number of the name or literal.
Names and literals not defined by this mechanism retain their default definition.
It is important that all token numbers be distinct.
For historical reasons, the endmarker must have token
This token number cannot be redefined by the user; thus, all
lexical analyzers should be prepared to return 0 or negative as a token number
upon reaching the end of their input.
A very useful tool for constructing lexical analyzers is
program developed by Mike Lesk.
These lexical analyzers are designed to work in close
harmony with Yacc parsers.
The specifications for these lexical analyzers
use regular expressions instead of grammar rules.
Lex can be easily used to produce quite complicated lexical analyzers,
but there remain some languages (such as FORTRAN) which do not
fit any theoretical framework, and whose lexical analyzers