+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+NAME
+ flex - fast lexical analyzer generator
+
+SYNOPSIS
+ flex [-bcdfhilnpstvwBFILTV78+? -C[aefFmr] -ooutput -Pprefix
+ -Sskeleton] [--help --version] [filename ...]
+
+OVERVIEW
+ This manual describes flex, a tool for generating programs
+ that perform pattern-matching on text. The manual includes
+ both tutorial and reference sections:
+
+ Description
+ a brief overview of the tool
+
+ Some Simple Examples
+
+ Format Of The Input File
+
+ Patterns
+ the extended regular expressions used by flex
+
+ How The Input Is Matched
+ the rules for determining what has been matched
+
+ Actions
+ how to specify what to do when a pattern is matched
+
+ The Generated Scanner
+ details regarding the scanner that flex produces;
+ how to control the input source
+
+ Start Conditions
+ introducing context into your scanners, and
+ managing "mini-scanners"
+
+ Multiple Input Buffers
+ how to manipulate multiple input sources; how to
+ scan from strings instead of files
+
+ End-of-file Rules
+ special rules for matching the end of the input
+
+ Miscellaneous Macros
+ a summary of macros available to the actions
+
+ Values Available To The User
+ a summary of values available to the actions
+
+ Interfacing With Yacc
+ connecting flex scanners together with yacc parsers
+
+
+
+
+Version 2.5 Last change: April 1995 1
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ Options
+ flex command-line options, and the "%option"
+ directive
+
+ Performance Considerations
+ how to make your scanner go as fast as possible
+
+ Generating C++ Scanners
+ the (experimental) facility for generating C++
+ scanner classes
+
+ Incompatibilities With Lex And POSIX
+ how flex differs from AT&T lex and the POSIX lex
+ standard
+
+ Diagnostics
+ those error messages produced by flex (or scanners
+ it generates) whose meanings might not be apparent
+
+ Files
+ files used by flex
+
+ Deficiencies / Bugs
+ known problems with flex
+
+ See Also
+ other documentation, related tools
+
+ Author
+ includes contact information
+
+
+DESCRIPTION
+ flex is a tool for generating scanners: programs which
+ recognized lexical patterns in text. flex reads the given
+ input files, or its standard input if no file names are
+ given, for a description of a scanner to generate. The
+ description is in the form of pairs of regular expressions
+ and C code, called rules. flex generates as output a C
+ source file, lex.yy.c, which defines a routine yylex(). This
+ file is compiled and linked with the -lfl library to produce
+ an executable. When the executable is run, it analyzes its
+ input for occurrences of the regular expressions. Whenever
+ it finds one, it executes the corresponding C code.
+
+SOME SIMPLE EXAMPLES
+ First some simple examples to get the flavor of how one uses
+ flex. The following flex input specifies a scanner which
+ whenever it encounters the string "username" will replace it
+ with the user's login name:
+
+ %%
+
+
+
+Version 2.5 Last change: April 1995 2
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ username printf( "%s", getlogin() );
+
+ By default, any text not matched by a flex scanner is copied
+ to the output, so the net effect of this scanner is to copy
+ its input file to its output with each occurrence of "user-
+ name" expanded. In this input, there is just one rule.
+ "username" is the pattern and the "printf" is the action.
+ The "%%" marks the beginning of the rules.
+
+ Here's another simple example:
+
+ int num_lines = 0, num_chars = 0;
+
+ %%
+ \n ++num_lines; ++num_chars;
+ . ++num_chars;
+
+ %%
+ main()
+ {
+ yylex();
+ printf( "# of lines = %d, # of chars = %d\n",
+ num_lines, num_chars );
+ }
+
+ This scanner counts the number of characters and the number
+ of lines in its input (it produces no output other than the
+ final report on the counts). The first line declares two
+ globals, "num_lines" and "num_chars", which are accessible
+ both inside yylex() and in the main() routine declared after
+ the second "%%". There are two rules, one which matches a
+ newline ("\n") and increments both the line count and the
+ character count, and one which matches any character other
+ than a newline (indicated by the "." regular expression).
+
+ A somewhat more complicated example:
+
+ /* scanner for a toy Pascal-like language */
+
+ %{
+ /* need this for the call to atof() below */
+ #include <math.h>
+ %}
+
+ DIGIT [0-9]
+ ID [a-z][a-z0-9]*
+
+ %%
+
+ {DIGIT}+ {
+ printf( "An integer: %s (%d)\n", yytext,
+ atoi( yytext ) );
+
+
+
+Version 2.5 Last change: April 1995 3
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ }
+
+ {DIGIT}+"."{DIGIT}* {
+ printf( "A float: %s (%g)\n", yytext,
+ atof( yytext ) );
+ }
+
+ if|then|begin|end|procedure|function {
+ printf( "A keyword: %s\n", yytext );
+ }
+
+ {ID} printf( "An identifier: %s\n", yytext );
+
+ "+"|"-"|"*"|"/" printf( "An operator: %s\n", yytext );
+
+ "{"[^}\n]*"}" /* eat up one-line comments */
+
+ [ \t\n]+ /* eat up whitespace */
+
+ . printf( "Unrecognized character: %s\n", yytext );
+
+ %%
+
+ main( argc, argv )
+ int argc;
+ char **argv;
+ {
+ ++argv, --argc; /* skip over program name */
+ if ( argc > 0 )
+ yyin = fopen( argv[0], "r" );
+ else
+ yyin = stdin;
+
+ yylex();
+ }
+
+ This is the beginnings of a simple scanner for a language
+ like Pascal. It identifies different types of tokens and
+ reports on what it has seen.
+
+ The details of this example will be explained in the follow-
+ ing sections.
+
+FORMAT OF THE INPUT FILE
+ The flex input file consists of three sections, separated by
+ a line with just %% in it:
+
+ definitions
+ %%
+ rules
+ %%
+ user code
+
+
+
+Version 2.5 Last change: April 1995 4
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ The definitions section contains declarations of simple name
+ definitions to simplify the scanner specification, and
+ declarations of start conditions, which are explained in a
+ later section.
+
+ Name definitions have the form:
+
+ name definition
+
+ The "name" is a word beginning with a letter or an under-
+ score ('_') followed by zero or more letters, digits, '_',
+ or '-' (dash). The definition is taken to begin at the
+ first non-white-space character following the name and con-
+ tinuing to the end of the line. The definition can subse-
+ quently be referred to using "{name}", which will expand to
+ "(definition)". For example,
+
+ DIGIT [0-9]
+ ID [a-z][a-z0-9]*
+
+ defines "DIGIT" to be a regular expression which matches a
+ single digit, and "ID" to be a regular expression which
+ matches a letter followed by zero-or-more letters-or-digits.
+ A subsequent reference to
+
+ {DIGIT}+"."{DIGIT}*
+
+ is identical to
+
+ ([0-9])+"."([0-9])*
+
+ and matches one-or-more digits followed by a '.' followed by
+ zero-or-more digits.
+
+ The rules section of the flex input contains a series of
+ rules of the form:
+
+ pattern action
+
+ where the pattern must be unindented and the action must
+ begin on the same line.
+
+ See below for a further description of patterns and actions.
+
+ Finally, the user code section is simply copied to lex.yy.c
+ verbatim. It is used for companion routines which call or
+ are called by the scanner. The presence of this section is
+ optional; if it is missing, the second %% in the input file
+ may be skipped, too.
+
+ In the definitions and rules sections, any indented text or
+ text enclosed in %{ and %} is copied verbatim to the output
+
+
+
+Version 2.5 Last change: April 1995 5
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ (with the %{}'s removed). The %{}'s must appear unindented
+ on lines by themselves.
+
+ In the rules section, any indented or %{} text appearing
+ before the first rule may be used to declare variables which
+ are local to the scanning routine and (after the declara-
+ tions) code which is to be executed whenever the scanning
+ routine is entered. Other indented or %{} text in the rule
+ section is still copied to the output, but its meaning is
+ not well-defined and it may well cause compile-time errors
+ (this feature is present for POSIX compliance; see below for
+ other such features).
+
+ In the definitions section (but not in the rules section),
+ an unindented comment (i.e., a line beginning with "/*") is
+ also copied verbatim to the output up to the next "*/".
+
+PATTERNS
+ The patterns in the input are written using an extended set
+ of regular expressions. These are:
+
+ x match the character 'x'
+ . any character (byte) except newline
+ [xyz] a "character class"; in this case, the pattern
+ matches either an 'x', a 'y', or a 'z'
+ [abj-oZ] a "character class" with a range in it; matches
+ an 'a', a 'b', any letter from 'j' through 'o',
+ or a 'Z'
+ [^A-Z] a "negated character class", i.e., any character
+ but those in the class. In this case, any
+ character EXCEPT an uppercase letter.
+ [^A-Z\n] any character EXCEPT an uppercase letter or
+ a newline
+ r* zero or more r's, where r is any regular expression
+ r+ one or more r's
+ r? zero or one r's (that is, "an optional r")
+ r{2,5} anywhere from two to five r's
+ r{2,} two or more r's
+ r{4} exactly 4 r's
+ {name} the expansion of the "name" definition
+ (see above)
+ "[xyz]\"foo"
+ the literal string: [xyz]"foo
+ \X if X is an 'a', 'b', 'f', 'n', 'r', 't', or 'v',
+ then the ANSI-C interpretation of \x.
+ Otherwise, a literal 'X' (used to escape
+ operators such as '*')
+ \0 a NUL character (ASCII code 0)
+ \123 the character with octal value 123
+ \x2a the character with hexadecimal value 2a
+ (r) match an r; parentheses are used to override
+ precedence (see below)
+
+
+
+Version 2.5 Last change: April 1995 6
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ rs the regular expression r followed by the
+ regular expression s; called "concatenation"
+
+
+ r|s either an r or an s
+
+
+ r/s an r but only if it is followed by an s. The
+ text matched by s is included when determining
+ whether this rule is the "longest match",
+ but is then returned to the input before
+ the action is executed. So the action only
+ sees the text matched by r. This type
+ of pattern is called trailing context".
+ (There are some combinations of r/s that flex
+ cannot match correctly; see notes in the
+ Deficiencies / Bugs section below regarding
+ "dangerous trailing context".)
+ ^r an r, but only at the beginning of a line (i.e.,
+ which just starting to scan, or right after a
+ newline has been scanned).
+ r$ an r, but only at the end of a line (i.e., just
+ before a newline). Equivalent to "r/\n".
+
+ Note that flex's notion of "newline" is exactly
+ whatever the C compiler used to compile flex
+ interprets '\n' as; in particular, on some DOS
+ systems you must either filter out \r's in the
+ input yourself, or explicitly use r/\r\n for "r$".
+
+
+ <s>r an r, but only in start condition s (see
+ below for discussion of start conditions)
+ <s1,s2,s3>r
+ same, but in any of start conditions s1,
+ s2, or s3
+ <*>r an r in any start condition, even an exclusive one.
+
+
+ <<EOF>> an end-of-file
+ <s1,s2><<EOF>>
+ an end-of-file when in start condition s1 or s2
+
+ Note that inside of a character class, all regular expres-
+ sion operators lose their special meaning except escape
+ ('\') and the character class operators, '-', ']', and, at
+ the beginning of the class, '^'.
+
+ The regular expressions listed above are grouped according
+ to precedence, from highest precedence at the top to lowest
+ at the bottom. Those grouped together have equal pre-
+ cedence. For example,
+
+
+
+Version 2.5 Last change: April 1995 7
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ foo|bar*
+
+ is the same as
+
+ (foo)|(ba(r*))
+
+ since the '*' operator has higher precedence than concatena-
+ tion, and concatenation higher than alternation ('|'). This
+ pattern therefore matches either the string "foo" or the
+ string "ba" followed by zero-or-more r's. To match "foo" or
+ zero-or-more "bar"'s, use:
+
+ foo|(bar)*
+
+ and to match zero-or-more "foo"'s-or-"bar"'s:
+
+ (foo|bar)*
+
+
+ In addition to characters and ranges of characters, charac-
+ ter classes can also contain character class expressions.
+ These are expressions enclosed inside [: and :] delimiters
+ (which themselves must appear between the '[' and ']' of the
+ character class; other elements may occur inside the charac-
+ ter class, too). The valid expressions are:
+
+ [:alnum:] [:alpha:] [:blank:]
+ [:cntrl:] [:digit:] [:graph:]
+ [:lower:] [:print:] [:punct:]
+ [:space:] [:upper:] [:xdigit:]
+
+ These expressions all designate a set of characters
+ equivalent to the corresponding standard C isXXX function.
+ For example, [:alnum:] designates those characters for which
+ isalnum() returns true - i.e., any alphabetic or numeric.
+ Some systems don't provide isblank(), so flex defines
+ [:blank:] as a blank or a tab.
+
+ For example, the following character classes are all
+ equivalent:
+
+ [[:alnum:]]
+ [[:alpha:][:digit:]
+ [[:alpha:]0-9]
+ [a-zA-Z0-9]
+
+ If your scanner is case-insensitive (the -i flag), then
+ [:upper:] and [:lower:] are equivalent to [:alpha:].
+
+ Some notes on patterns:
+
+ - A negated character class such as the example "[^A-Z]"
+
+
+
+Version 2.5 Last change: April 1995 8
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ above will match a newline unless "\n" (or an
+ equivalent escape sequence) is one of the characters
+ explicitly present in the negated character class
+ (e.g., "[^A-Z\n]"). This is unlike how many other reg-
+ ular expression tools treat negated character classes,
+ but unfortunately the inconsistency is historically
+ entrenched. Matching newlines means that a pattern
+ like [^"]* can match the entire input unless there's
+ another quote in the input.
+
+ - A rule can have at most one instance of trailing con-
+ text (the '/' operator or the '$' operator). The start
+ condition, '^', and "<<EOF>>" patterns can only occur
+ at the beginning of a pattern, and, as well as with '/'
+ and '$', cannot be grouped inside parentheses. A '^'
+ which does not occur at the beginning of a rule or a
+ '$' which does not occur at the end of a rule loses its
+ special properties and is treated as a normal charac-
+ ter.
+
+ The following are illegal:
+
+ foo/bar$
+ <sc1>foo<sc2>bar
+
+ Note that the first of these, can be written
+ "foo/bar\n".
+
+ The following will result in '$' or '^' being treated
+ as a normal character:
+
+ foo|(bar$)
+ foo|^bar
+
+ If what's wanted is a "foo" or a bar-followed-by-a-
+ newline, the following could be used (the special '|'
+ action is explained below):
+
+ foo |
+ bar$ /* action goes here */
+
+ A similar trick will work for matching a foo or a bar-
+ at-the-beginning-of-a-line.
+
+HOW THE INPUT IS MATCHED
+ When the generated scanner is run, it analyzes its input
+ looking for strings which match any of its patterns. If it
+ finds more than one match, it takes the one matching the
+ most text (for trailing context rules, this includes the
+ length of the trailing part, even though it will then be
+ returned to the input). If it finds two or more matches of
+ the same length, the rule listed first in the flex input
+
+
+
+Version 2.5 Last change: April 1995 9
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ file is chosen.
+
+ Once the match is determined, the text corresponding to the
+ match (called the token) is made available in the global
+ character pointer yytext, and its length in the global
+ integer yyleng. The action corresponding to the matched pat-
+ tern is then executed (a more detailed description of
+ actions follows), and then the remaining input is scanned
+ for another match.
+
+ If no match is found, then the default rule is executed: the
+ next character in the input is considered matched and copied
+ to the standard output. Thus, the simplest legal flex input
+ is:
+
+ %%
+
+ which generates a scanner that simply copies its input (one
+ character at a time) to its output.
+
+ Note that yytext can be defined in two different ways:
+ either as a character pointer or as a character array. You
+ can control which definition flex uses by including one of
+ the special directives %pointer or %array in the first
+ (definitions) section of your flex input. The default is
+ %pointer, unless you use the -l lex compatibility option, in
+ which case yytext will be an array. The advantage of using
+ %pointer is substantially faster scanning and no buffer
+ overflow when matching very large tokens (unless you run out
+ of dynamic memory). The disadvantage is that you are res-
+ tricted in how your actions can modify yytext (see the next
+ section), and calls to the unput() function destroys the
+ present contents of yytext, which can be a considerable
+ porting headache when moving between different lex versions.
+
+ The advantage of %array is that you can then modify yytext
+ to your heart's content, and calls to unput() do not destroy
+ yytext (see below). Furthermore, existing lex programs
+ sometimes access yytext externally using declarations of the
+ form:
+ extern char yytext[];
+ This definition is erroneous when used with %pointer, but
+ correct for %array.
+
+ %array defines yytext to be an array of YYLMAX characters,
+ which defaults to a fairly large value. You can change the
+ size by simply #define'ing YYLMAX to a different value in
+ the first section of your flex input. As mentioned above,
+ with %pointer yytext grows dynamically to accommodate large
+ tokens. While this means your %pointer scanner can accommo-
+ date very large tokens (such as matching entire blocks of
+ comments), bear in mind that each time the scanner must
+
+
+
+Version 2.5 Last change: April 1995 10
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ resize yytext it also must rescan the entire token from the
+ beginning, so matching such tokens can prove slow. yytext
+ presently does not dynamically grow if a call to unput()
+ results in too much text being pushed back; instead, a run-
+ time error results.
+
+ Also note that you cannot use %array with C++ scanner
+ classes (the c++ option; see below).
+
+ACTIONS
+ Each pattern in a rule has a corresponding action, which can
+ be any arbitrary C statement. The pattern ends at the first
+ non-escaped whitespace character; the remainder of the line
+ is its action. If the action is empty, then when the pat-
+ tern is matched the input token is simply discarded. For
+ example, here is the specification for a program which
+ deletes all occurrences of "zap me" from its input:
+
+ %%
+ "zap me"
+
+ (It will copy all other characters in the input to the out-
+ put since they will be matched by the default rule.)
+
+ Here is a program which compresses multiple blanks and tabs
+ down to a single blank, and throws away whitespace found at
+ the end of a line:
+
+ %%
+ [ \t]+ putchar( ' ' );
+ [ \t]+$ /* ignore this token */
+
+
+ If the action contains a '{', then the action spans till the
+ balancing '}' is found, and the action may cross multiple
+ lines. flex knows about C strings and comments and won't be
+ fooled by braces found within them, but also allows actions
+ to begin with %{ and will consider the action to be all the
+ text up to the next %} (regardless of ordinary braces inside
+ the action).
+
+ An action consisting solely of a vertical bar ('|') means
+ "same as the action for the next rule." See below for an
+ illustration.
+
+ Actions can include arbitrary C code, including return
+ statements to return a value to whatever routine called
+ yylex(). Each time yylex() is called it continues processing
+ tokens from where it last left off until it either reaches
+ the end of the file or executes a return.
+
+
+
+
+
+Version 2.5 Last change: April 1995 11
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ Actions are free to modify yytext except for lengthening it
+ (adding characters to its end--these will overwrite later
+ characters in the input stream). This however does not
+ apply when using %array (see above); in that case, yytext
+ may be freely modified in any way.
+
+ Actions are free to modify yyleng except they should not do
+ so if the action also includes use of yymore() (see below).
+
+ There are a number of special directives which can be
+ included within an action:
+
+ - ECHO copies yytext to the scanner's output.
+
+ - BEGIN followed by the name of a start condition places
+ the scanner in the corresponding start condition (see
+ below).
+
+ - REJECT directs the scanner to proceed on to the "second
+ best" rule which matched the input (or a prefix of the
+ input). The rule is chosen as described above in "How
+ the Input is Matched", and yytext and yyleng set up
+ appropriately. It may either be one which matched as
+ much text as the originally chosen rule but came later
+ in the flex input file, or one which matched less text.
+ For example, the following will both count the words in
+ the input and call the routine special() whenever
+ "frob" is seen:
+
+ int word_count = 0;
+ %%
+
+ frob special(); REJECT;
+ [^ \t\n]+ ++word_count;
+
+ Without the REJECT, any "frob"'s in the input would not
+ be counted as words, since the scanner normally exe-
+ cutes only one action per token. Multiple REJECT's are
+ allowed, each one finding the next best choice to the
+ currently active rule. For example, when the following
+ scanner scans the token "abcd", it will write "abcdab-
+ caba" to the output:
+
+ %%
+ a |
+ ab |
+ abc |
+ abcd ECHO; REJECT;
+ .|\n /* eat up any unmatched character */
+
+ (The first three rules share the fourth's action since
+ they use the special '|' action.) REJECT is a
+
+
+
+Version 2.5 Last change: April 1995 12
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ particularly expensive feature in terms of scanner per-
+ formance; if it is used in any of the scanner's actions
+ it will slow down all of the scanner's matching.
+ Furthermore, REJECT cannot be used with the -Cf or -CF
+ options (see below).
+
+ Note also that unlike the other special actions, REJECT
+ is a branch; code immediately following it in the
+ action will not be executed.
+
+ - yymore() tells the scanner that the next time it
+ matches a rule, the corresponding token should be
+ appended onto the current value of yytext rather than
+ replacing it. For example, given the input "mega-
+ kludge" the following will write "mega-mega-kludge" to
+ the output:
+
+ %%
+ mega- ECHO; yymore();
+ kludge ECHO;
+
+ First "mega-" is matched and echoed to the output.
+ Then "kludge" is matched, but the previous "mega-" is
+ still hanging around at the beginning of yytext so the
+ ECHO for the "kludge" rule will actually write "mega-
+ kludge".
+
+ Two notes regarding use of yymore(). First, yymore() depends
+ on the value of yyleng correctly reflecting the size of the
+ current token, so you must not modify yyleng if you are
+ using yymore(). Second, the presence of yymore() in the
+ scanner's action entails a minor performance penalty in the
+ scanner's matching speed.
+
+ - yyless(n) returns all but the first n characters of the
+ current token back to the input stream, where they will
+ be rescanned when the scanner looks for the next match.
+ yytext and yyleng are adjusted appropriately (e.g.,
+ yyleng will now be equal to n ). For example, on the
+ input "foobar" the following will write out "foobar-
+ bar":
+
+ %%
+ foobar ECHO; yyless(3);
+ [a-z]+ ECHO;
+
+ An argument of 0 to yyless will cause the entire
+ current input string to be scanned again. Unless
+ you've changed how the scanner will subsequently pro-
+ cess its input (using BEGIN, for example), this will
+ result in an endless loop.
+
+
+
+
+Version 2.5 Last change: April 1995 13
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ Note that yyless is a macro and can only be used in the flex
+ input file, not from other source files.
+
+ - unput(c) puts the character c back onto the input
+ stream. It will be the next character scanned. The
+ following action will take the current token and cause
+ it to be rescanned enclosed in parentheses.
+
+ {
+ int i;
+ /* Copy yytext because unput() trashes yytext */
+ char *yycopy = strdup( yytext );
+ unput( ')' );
+ for ( i = yyleng - 1; i >= 0; --i )
+ unput( yycopy[i] );
+ unput( '(' );
+ free( yycopy );
+ }
+
+ Note that since each unput() puts the given character
+ back at the beginning of the input stream, pushing back
+ strings must be done back-to-front.
+
+ An important potential problem when using unput() is that if
+ you are using %pointer (the default), a call to unput() des-
+ troys the contents of yytext, starting with its rightmost
+ character and devouring one character to the left with each
+ call. If you need the value of yytext preserved after a
+ call to unput() (as in the above example), you must either
+ first copy it elsewhere, or build your scanner using %array
+ instead (see How The Input Is Matched).
+
+ Finally, note that you cannot put back EOF to attempt to
+ mark the input stream with an end-of-file.
+
+ - input() reads the next character from the input stream.
+ For example, the following is one way to eat up C com-
+ ments:
+
+ %%
+ "/*" {
+ register int c;
+
+ for ( ; ; )
+ {
+ while ( (c = input()) != '*' &&
+ c != EOF )
+ ; /* eat up text of comment */
+
+ if ( c == '*' )
+ {
+ while ( (c = input()) == '*' )
+
+
+
+Version 2.5 Last change: April 1995 14
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ ;
+ if ( c == '/' )
+ break; /* found the end */
+ }
+
+ if ( c == EOF )
+ {
+ error( "EOF in comment" );
+ break;
+ }
+ }
+ }
+
+ (Note that if the scanner is compiled using C++, then
+ input() is instead referred to as yyinput(), in order
+ to avoid a name clash with the C++ stream by the name
+ of input.)
+
+ - YY_FLUSH_BUFFER flushes the scanner's internal buffer
+ so that the next time the scanner attempts to match a
+ token, it will first refill the buffer using YY_INPUT
+ (see The Generated Scanner, below). This action is a
+ special case of the more general yy_flush_buffer()
+ function, described below in the section Multiple Input
+ Buffers.
+
+ - yyterminate() can be used in lieu of a return statement
+ in an action. It terminates the scanner and returns a
+ 0 to the scanner's caller, indicating "all done". By
+ default, yyterminate() is also called when an end-of-
+ file is encountered. It is a macro and may be rede-
+ fined.
+
+THE GENERATED SCANNER
+ The output of flex is the file lex.yy.c, which contains the
+ scanning routine yylex(), a number of tables used by it for
+ matching tokens, and a number of auxiliary routines and mac-
+ ros. By default, yylex() is declared as follows:
+
+ int yylex()
+ {
+ ... various definitions and the actions in here ...
+ }
+
+ (If your environment supports function prototypes, then it
+ will be "int yylex( void )".) This definition may be
+ changed by defining the "YY_DECL" macro. For example, you
+ could use:
+
+ #define YY_DECL float lexscan( a, b ) float a, b;
+
+ to give the scanning routine the name lexscan, returning a
+
+
+
+Version 2.5 Last change: April 1995 15
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ float, and taking two floats as arguments. Note that if you
+ give arguments to the scanning routine using a K&R-
+ style/non-prototyped function declaration, you must ter-
+ minate the definition with a semi-colon (;).
+
+ Whenever yylex() is called, it scans tokens from the global
+ input file yyin (which defaults to stdin). It continues
+ until it either reaches an end-of-file (at which point it
+ returns the value 0) or one of its actions executes a return
+ statement.
+
+ If the scanner reaches an end-of-file, subsequent calls are
+ undefined unless either yyin is pointed at a new input file
+ (in which case scanning continues from that file), or yyres-
+ tart() is called. yyrestart() takes one argument, a FILE *
+ pointer (which can be nil, if you've set up YY_INPUT to scan
+ from a source other than yyin), and initializes yyin for
+ scanning from that file. Essentially there is no difference
+ between just assigning yyin to a new input file or using
+ yyrestart() to do so; the latter is available for compati-
+ bility with previous versions of flex, and because it can be
+ used to switch input files in the middle of scanning. It
+ can also be used to throw away the current input buffer, by
+ calling it with an argument of yyin; but better is to use
+ YY_FLUSH_BUFFER (see above). Note that yyrestart() does not
+ reset the start condition to INITIAL (see Start Conditions,
+ below).
+
+ If yylex() stops scanning due to executing a return state-
+ ment in one of the actions, the scanner may then be called
+ again and it will resume scanning where it left off.
+
+ By default (and for purposes of efficiency), the scanner
+ uses block-reads rather than simple getc() calls to read
+ characters from yyin. The nature of how it gets its input
+ can be controlled by defining the YY_INPUT macro.
+ YY_INPUT's calling sequence is
+ "YY_INPUT(buf,result,max_size)". Its action is to place up
+ to max_size characters in the character array buf and return
+ in the integer variable result either the number of charac-
+ ters read or the constant YY_NULL (0 on Unix systems) to
+ indicate EOF. The default YY_INPUT reads from the global
+ file-pointer "yyin".
+
+ A sample definition of YY_INPUT (in the definitions section
+ of the input file):
+
+ %{
+ #define YY_INPUT(buf,result,max_size) \
+ { \
+ int c = getchar(); \
+ result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \
+
+
+
+Version 2.5 Last change: April 1995 16
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ }
+ %}
+
+ This definition will change the input processing to occur
+ one character at a time.
+
+ When the scanner receives an end-of-file indication from
+ YY_INPUT, it then checks the yywrap() function. If yywrap()
+ returns false (zero), then it is assumed that the function
+ has gone ahead and set up yyin to point to another input
+ file, and scanning continues. If it returns true (non-
+ zero), then the scanner terminates, returning 0 to its
+ caller. Note that in either case, the start condition
+ remains unchanged; it does not revert to INITIAL.
+
+ If you do not supply your own version of yywrap(), then you
+ must either use %option noyywrap (in which case the scanner
+ behaves as though yywrap() returned 1), or you must link
+ with -lfl to obtain the default version of the routine,
+ which always returns 1.
+
+ Three routines are available for scanning from in-memory
+ buffers rather than files: yy_scan_string(),
+ yy_scan_bytes(), and yy_scan_buffer(). See the discussion of
+ them below in the section Multiple Input Buffers.
+
+ The scanner writes its ECHO output to the yyout global
+ (default, stdout), which may be redefined by the user simply
+ by assigning it to some other FILE pointer.
+
+START CONDITIONS
+ flex provides a mechanism for conditionally activating
+ rules. Any rule whose pattern is prefixed with "<sc>" will
+ only be active when the scanner is in the start condition
+ named "sc". For example,
+
+ <STRING>[^"]* { /* eat up the string body ... */
+ ...
+ }
+
+ will be active only when the scanner is in the "STRING"
+ start condition, and
+
+ <INITIAL,STRING,QUOTE>\. { /* handle an escape ... */
+ ...
+ }
+
+ will be active only when the current start condition is
+ either "INITIAL", "STRING", or "QUOTE".
+
+ Start conditions are declared in the definitions (first)
+ section of the input using unindented lines beginning with
+
+
+
+Version 2.5 Last change: April 1995 17
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ either %s or %x followed by a list of names. The former
+ declares inclusive start conditions, the latter exclusive
+ start conditions. A start condition is activated using the
+ BEGIN action. Until the next BEGIN action is executed,
+ rules with the given start condition will be active and
+ rules with other start conditions will be inactive. If the
+ start condition is inclusive, then rules with no start con-
+ ditions at all will also be active. If it is exclusive,
+ then only rules qualified with the start condition will be
+ active. A set of rules contingent on the same exclusive
+ start condition describe a scanner which is independent of
+ any of the other rules in the flex input. Because of this,
+ exclusive start conditions make it easy to specify "mini-
+ scanners" which scan portions of the input that are syntac-
+ tically different from the rest (e.g., comments).
+
+ If the distinction between inclusive and exclusive start
+ conditions is still a little vague, here's a simple example
+ illustrating the connection between the two. The set of
+ rules:
+
+ %s example
+ %%
+
+ <example>foo do_something();
+
+ bar something_else();
+
+ is equivalent to
+
+ %x example
+ %%
+
+ <example>foo do_something();
+
+ <INITIAL,example>bar something_else();
+
+ Without the <INITIAL,example> qualifier, the bar pattern in
+ the second example wouldn't be active (i.e., couldn't match)
+ when in start condition example. If we just used <example>
+ to qualify bar, though, then it would only be active in
+ example and not in INITIAL, while in the first example it's
+ active in both, because in the first example the example
+ startion condition is an inclusive (%s) start condition.
+
+ Also note that the special start-condition specifier <*>
+ matches every start condition. Thus, the above example
+ could also have been written;
+
+ %x example
+ %%
+
+
+
+
+Version 2.5 Last change: April 1995 18
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ <example>foo do_something();
+
+ <*>bar something_else();
+
+
+ The default rule (to ECHO any unmatched character) remains
+ active in start conditions. It is equivalent to:
+
+ <*>.|\n ECHO;
+
+
+ BEGIN(0) returns to the original state where only the rules
+ with no start conditions are active. This state can also be
+ referred to as the start-condition "INITIAL", so
+ BEGIN(INITIAL) is equivalent to BEGIN(0). (The parentheses
+ around the start condition name are not required but are
+ considered good style.)
+
+ BEGIN actions can also be given as indented code at the
+ beginning of the rules section. For example, the following
+ will cause the scanner to enter the "SPECIAL" start condi-
+ tion whenever yylex() is called and the global variable
+ enter_special is true:
+
+ int enter_special;
+
+ %x SPECIAL
+ %%
+ if ( enter_special )
+ BEGIN(SPECIAL);
+
+ <SPECIAL>blahblahblah
+ ...more rules follow...
+
+
+ To illustrate the uses of start conditions, here is a
+ scanner which provides two different interpretations of a
+ string like "123.456". By default it will treat it as three
+ tokens, the integer "123", a dot ('.'), and the integer
+ "456". But if the string is preceded earlier in the line by
+ the string "expect-floats" it will treat it as a single
+ token, the floating-point number 123.456:
+
+ %{
+ #include <math.h>
+ %}
+ %s expect
+
+ %%
+ expect-floats BEGIN(expect);
+
+ <expect>[0-9]+"."[0-9]+ {
+
+
+
+Version 2.5 Last change: April 1995 19
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ printf( "found a float, = %f\n",
+ atof( yytext ) );
+ }
+ <expect>\n {
+ /* that's the end of the line, so
+ * we need another "expect-number"
+ * before we'll recognize any more
+ * numbers
+ */
+ BEGIN(INITIAL);
+ }
+
+ [0-9]+ {
+ printf( "found an integer, = %d\n",
+ atoi( yytext ) );
+ }
+
+ "." printf( "found a dot\n" );
+
+ Here is a scanner which recognizes (and discards) C comments
+ while maintaining a count of the current input line.
+
+ %x comment
+ %%
+ int line_num = 1;
+
+ "/*" BEGIN(comment);
+
+ <comment>[^*\n]* /* eat anything that's not a '*' */
+ <comment>"*"+[^*/\n]* /* eat up '*'s not followed by '/'s */
+ <comment>\n ++line_num;
+ <comment>"*"+"/" BEGIN(INITIAL);
+
+ This scanner goes to a bit of trouble to match as much text
+ as possible with each rule. In general, when attempting to
+ write a high-speed scanner try to match as much possible in
+ each rule, as it's a big win.
+
+ Note that start-conditions names are really integer values
+ and can be stored as such. Thus, the above could be
+ extended in the following fashion:
+
+ %x comment foo
+ %%
+ int line_num = 1;
+ int comment_caller;
+
+ "/*" {
+ comment_caller = INITIAL;
+ BEGIN(comment);
+ }
+
+
+
+
+Version 2.5 Last change: April 1995 20
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ ...
+
+ <foo>"/*" {
+ comment_caller = foo;
+ BEGIN(comment);
+ }
+
+ <comment>[^*\n]* /* eat anything that's not a '*' */
+ <comment>"*"+[^*/\n]* /* eat up '*'s not followed by '/'s */
+ <comment>\n ++line_num;
+ <comment>"*"+"/" BEGIN(comment_caller);
+
+ Furthermore, you can access the current start condition
+ using the integer-valued YY_START macro. For example, the
+ above assignments to comment_caller could instead be written
+
+ comment_caller = YY_START;
+
+ Flex provides YYSTATE as an alias for YY_START (since that
+ is what's used by AT&T lex).
+
+ Note that start conditions do not have their own name-space;
+ %s's and %x's declare names in the same fashion as
+ #define's.
+
+ Finally, here's an example of how to match C-style quoted
+ strings using exclusive start conditions, including expanded
+ escape sequences (but not including checking for a string
+ that's too long):
+
+ %x str
+
+ %%
+ char string_buf[MAX_STR_CONST];
+ char *string_buf_ptr;
+
+
+ \" string_buf_ptr = string_buf; BEGIN(str);
+
+ <str>\" { /* saw closing quote - all done */
+ BEGIN(INITIAL);
+ *string_buf_ptr = '\0';
+ /* return string constant token type and
+ * value to parser
+ */
+ }
+
+ <str>\n {
+ /* error - unterminated string constant */
+ /* generate error message */
+ }
+
+
+
+
+Version 2.5 Last change: April 1995 21
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ <str>\\[0-7]{1,3} {
+ /* octal escape sequence */
+ int result;
+
+ (void) sscanf( yytext + 1, "%o", &result );
+
+ if ( result > 0xff )
+ /* error, constant is out-of-bounds */
+
+ *string_buf_ptr++ = result;
+ }
+
+ <str>\\[0-9]+ {
+ /* generate error - bad escape sequence; something
+ * like '\48' or '\0777777'
+ */
+ }
+
+ <str>\\n *string_buf_ptr++ = '\n';
+ <str>\\t *string_buf_ptr++ = '\t';
+ <str>\\r *string_buf_ptr++ = '\r';
+ <str>\\b *string_buf_ptr++ = '\b';
+ <str>\\f *string_buf_ptr++ = '\f';
+
+ <str>\\(.|\n) *string_buf_ptr++ = yytext[1];
+
+ <str>[^\\\n\"]+ {
+ char *yptr = yytext;
+
+ while ( *yptr )
+ *string_buf_ptr++ = *yptr++;
+ }
+
+
+ Often, such as in some of the examples above, you wind up
+ writing a whole bunch of rules all preceded by the same
+ start condition(s). Flex makes this a little easier and
+ cleaner by introducing a notion of start condition scope. A
+ start condition scope is begun with:
+
+ <SCs>{
+
+ where SCs is a list of one or more start conditions. Inside
+ the start condition scope, every rule automatically has the
+ prefix <SCs> applied to it, until a '}' which matches the
+ initial '{'. So, for example,
+
+ <ESC>{
+ "\\n" return '\n';
+ "\\r" return '\r';
+ "\\f" return '\f';
+ "\\0" return '\0';
+
+
+
+Version 2.5 Last change: April 1995 22
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ }
+
+ is equivalent to:
+
+ <ESC>"\\n" return '\n';
+ <ESC>"\\r" return '\r';
+ <ESC>"\\f" return '\f';
+ <ESC>"\\0" return '\0';
+
+ Start condition scopes may be nested.
+
+ Three routines are available for manipulating stacks of
+ start conditions:
+
+ void yy_push_state(int new_state)
+ pushes the current start condition onto the top of the
+ start condition stack and switches to new_state as
+ though you had used BEGIN new_state (recall that start
+ condition names are also integers).
+
+ void yy_pop_state()
+ pops the top of the stack and switches to it via BEGIN.
+
+ int yy_top_state()
+ returns the top of the stack without altering the
+ stack's contents.
+
+ The start condition stack grows dynamically and so has no
+ built-in size limitation. If memory is exhausted, program
+ execution aborts.
+
+ To use start condition stacks, your scanner must include a
+ %option stack directive (see Options below).
+
+MULTIPLE INPUT BUFFERS
+ Some scanners (such as those which support "include" files)
+ require reading from several input streams. As flex
+ scanners do a large amount of buffering, one cannot control
+ where the next input will be read from by simply writing a
+ YY_INPUT which is sensitive to the scanning context.
+ YY_INPUT is only called when the scanner reaches the end of
+ its buffer, which may be a long time after scanning a state-
+ ment such as an "include" which requires switching the input
+ source.
+
+ To negotiate these sorts of problems, flex provides a
+ mechanism for creating and switching between multiple input
+ buffers. An input buffer is created by using:
+
+ YY_BUFFER_STATE yy_create_buffer( FILE *file, int size )
+
+ which takes a FILE pointer and a size and creates a buffer
+
+
+
+Version 2.5 Last change: April 1995 23
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ associated with the given file and large enough to hold size
+ characters (when in doubt, use YY_BUF_SIZE for the size).
+ It returns a YY_BUFFER_STATE handle, which may then be
+ passed to other routines (see below). The YY_BUFFER_STATE
+ type is a pointer to an opaque struct yy_buffer_state struc-
+ ture, so you may safely initialize YY_BUFFER_STATE variables
+ to ((YY_BUFFER_STATE) 0) if you wish, and also refer to the
+ opaque structure in order to correctly declare input buffers
+ in source files other than that of your scanner. Note that
+ the FILE pointer in the call to yy_create_buffer is only
+ used as the value of yyin seen by YY_INPUT; if you redefine
+ YY_INPUT so it no longer uses yyin, then you can safely pass
+ a nil FILE pointer to yy_create_buffer. You select a partic-
+ ular buffer to scan from using:
+
+ void yy_switch_to_buffer( YY_BUFFER_STATE new_buffer )
+
+ switches the scanner's input buffer so subsequent tokens
+ will come from new_buffer. Note that yy_switch_to_buffer()
+ may be used by yywrap() to set things up for continued scan-
+ ning, instead of opening a new file and pointing yyin at it.
+ Note also that switching input sources via either
+ yy_switch_to_buffer() or yywrap() does not change the start
+ condition.
+
+ void yy_delete_buffer( YY_BUFFER_STATE buffer )
+
+ is used to reclaim the storage associated with a buffer. (
+ buffer can be nil, in which case the routine does nothing.)
+ You can also clear the current contents of a buffer using:
+
+ void yy_flush_buffer( YY_BUFFER_STATE buffer )
+
+ This function discards the buffer's contents, so the next
+ time the scanner attempts to match a token from the buffer,
+ it will first fill the buffer anew using YY_INPUT.
+
+ yy_new_buffer() is an alias for yy_create_buffer(), provided
+ for compatibility with the C++ use of new and delete for
+ creating and destroying dynamic objects.
+
+ Finally, the YY_CURRENT_BUFFER macro returns a
+ YY_BUFFER_STATE handle to the current buffer.
+
+ Here is an example of using these features for writing a
+ scanner which expands include files (the <<EOF>> feature is
+ discussed below):
+
+ /* the "incl" state is used for picking up the name
+ * of an include file
+ */
+ %x incl
+
+
+
+Version 2.5 Last change: April 1995 24
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ %{
+ #define MAX_INCLUDE_DEPTH 10
+ YY_BUFFER_STATE include_stack[MAX_INCLUDE_DEPTH];
+ int include_stack_ptr = 0;
+ %}
+
+ %%
+ include BEGIN(incl);
+
+ [a-z]+ ECHO;
+ [^a-z\n]*\n? ECHO;
+
+ <incl>[ \t]* /* eat the whitespace */
+ <incl>[^ \t\n]+ { /* got the include file name */
+ if ( include_stack_ptr >= MAX_INCLUDE_DEPTH )
+ {
+ fprintf( stderr, "Includes nested too deeply" );
+ exit( 1 );
+ }
+
+ include_stack[include_stack_ptr++] =
+ YY_CURRENT_BUFFER;
+
+ yyin = fopen( yytext, "r" );
+
+ if ( ! yyin )
+ error( ... );
+
+ yy_switch_to_buffer(
+ yy_create_buffer( yyin, YY_BUF_SIZE ) );
+
+ BEGIN(INITIAL);
+ }
+
+ <<EOF>> {
+ if ( --include_stack_ptr < 0 )
+ {
+ yyterminate();
+ }
+
+ else
+ {
+ yy_delete_buffer( YY_CURRENT_BUFFER );
+ yy_switch_to_buffer(
+ include_stack[include_stack_ptr] );
+ }
+ }
+
+ Three routines are available for setting up input buffers
+ for scanning in-memory strings instead of files. All of
+ them create a new input buffer for scanning the string, and
+ return a corresponding YY_BUFFER_STATE handle (which you
+
+
+
+Version 2.5 Last change: April 1995 25
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ should delete with yy_delete_buffer() when done with it).
+ They also switch to the new buffer using
+ yy_switch_to_buffer(), so the next call to yylex() will
+ start scanning the string.
+
+ yy_scan_string(const char *str)
+ scans a NUL-terminated string.
+
+ yy_scan_bytes(const char *bytes, int len)
+ scans len bytes (including possibly NUL's) starting at
+ location bytes.
+
+ Note that both of these functions create and scan a copy of
+ the string or bytes. (This may be desirable, since yylex()
+ modifies the contents of the buffer it is scanning.) You
+ can avoid the copy by using:
+
+ yy_scan_buffer(char *base, yy_size_t size)
+ which scans in place the buffer starting at base, con-
+ sisting of size bytes, the last two bytes of which must
+ be YY_END_OF_BUFFER_CHAR (ASCII NUL). These last two
+ bytes are not scanned; thus, scanning consists of
+ base[0] through base[size-2], inclusive.
+
+ If you fail to set up base in this manner (i.e., forget
+ the final two YY_END_OF_BUFFER_CHAR bytes), then
+ yy_scan_buffer() returns a nil pointer instead of
+ creating a new input buffer.
+
+ The type yy_size_t is an integral type to which you can
+ cast an integer expression reflecting the size of the
+ buffer.
+
+END-OF-FILE RULES
+ The special rule "<<EOF>>" indicates actions which are to be
+ taken when an end-of-file is encountered and yywrap()
+ returns non-zero (i.e., indicates no further files to pro-
+ cess). The action must finish by doing one of four things:
+
+ - assigning yyin to a new input file (in previous ver-
+ sions of flex, after doing the assignment you had to
+ call the special action YY_NEW_FILE; this is no longer
+ necessary);
+
+ - executing a return statement;
+
+ - executing the special yyterminate() action;
+
+ - or, switching to a new buffer using
+ yy_switch_to_buffer() as shown in the example above.
+
+
+
+
+
+Version 2.5 Last change: April 1995 26
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ <<EOF>> rules may not be used with other patterns; they may
+ only be qualified with a list of start conditions. If an
+ unqualified <<EOF>> rule is given, it applies to all start
+ conditions which do not already have <<EOF>> actions. To
+ specify an <<EOF>> rule for only the initial start condi-
+ tion, use
+
+ <INITIAL><<EOF>>
+
+
+ These rules are useful for catching things like unclosed
+ comments. An example:
+
+ %x quote
+ %%
+
+ ...other rules for dealing with quotes...
+
+ <quote><<EOF>> {
+ error( "unterminated quote" );
+ yyterminate();
+ }
+ <<EOF>> {
+ if ( *++filelist )
+ yyin = fopen( *filelist, "r" );
+ else
+ yyterminate();
+ }
+
+
+MISCELLANEOUS MACROS
+ The macro YY_USER_ACTION can be defined to provide an action
+ which is always executed prior to the matched rule's action.
+ For example, it could be #define'd to call a routine to con-
+ vert yytext to lower-case. When YY_USER_ACTION is invoked,
+ the variable yy_act gives the number of the matched rule
+ (rules are numbered starting with 1). Suppose you want to
+ profile how often each of your rules is matched. The fol-
+ lowing would do the trick:
+
+ #define YY_USER_ACTION ++ctr[yy_act]
+
+ where ctr is an array to hold the counts for the different
+ rules. Note that the macro YY_NUM_RULES gives the total
+ number of rules (including the default rule, even if you use
+ -s), so a correct declaration for ctr is:
+
+ int ctr[YY_NUM_RULES];
+
+
+ The macro YY_USER_INIT may be defined to provide an action
+ which is always executed before the first scan (and before
+
+
+
+Version 2.5 Last change: April 1995 27
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ the scanner's internal initializations are done). For exam-
+ ple, it could be used to call a routine to read in a data
+ table or open a logging file.
+
+ The macro yy_set_interactive(is_interactive) can be used to
+ control whether the current buffer is considered interac-
+ tive. An interactive buffer is processed more slowly, but
+ must be used when the scanner's input source is indeed
+ interactive to avoid problems due to waiting to fill buffers
+ (see the discussion of the -I flag below). A non-zero value
+ in the macro invocation marks the buffer as interactive, a
+ zero value as non-interactive. Note that use of this macro
+ overrides %option always-interactive or %option never-
+ interactive (see Options below). yy_set_interactive() must
+ be invoked prior to beginning to scan the buffer that is (or
+ is not) to be considered interactive.
+
+ The macro yy_set_bol(at_bol) can be used to control whether
+ the current buffer's scanning context for the next token
+ match is done as though at the beginning of a line. A non-
+ zero macro argument makes rules anchored with
+
+ The macro YY_AT_BOL() returns true if the next token scanned
+ from the current buffer will have '^' rules active, false
+ otherwise.
+
+ In the generated scanner, the actions are all gathered in
+ one large switch statement and separated using YY_BREAK,
+ which may be redefined. By default, it is simply a "break",
+ to separate each rule's action from the following rule's.
+ Redefining YY_BREAK allows, for example, C++ users to
+ #define YY_BREAK to do nothing (while being very careful
+ that every rule ends with a "break" or a "return"!) to avoid
+ suffering from unreachable statement warnings where because
+ a rule's action ends with "return", the YY_BREAK is inacces-
+ sible.
+
+VALUES AVAILABLE TO THE USER
+ This section summarizes the various values available to the
+ user in the rule actions.
+
+ - char *yytext holds the text of the current token. It
+ may be modified but not lengthened (you cannot append
+ characters to the end).
+
+ If the special directive %array appears in the first
+ section of the scanner description, then yytext is
+ instead declared char yytext[YYLMAX], where YYLMAX is a
+ macro definition that you can redefine in the first
+ section if you don't like the default value (generally
+ 8KB). Using %array results in somewhat slower
+ scanners, but the value of yytext becomes immune to
+
+
+
+Version 2.5 Last change: April 1995 28
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ calls to input() and unput(), which potentially destroy
+ its value when yytext is a character pointer. The
+ opposite of %array is %pointer, which is the default.
+
+ You cannot use %array when generating C++ scanner
+ classes (the -+ flag).
+
+ - int yyleng holds the length of the current token.
+
+ - FILE *yyin is the file which by default flex reads
+ from. It may be redefined but doing so only makes
+ sense before scanning begins or after an EOF has been
+ encountered. Changing it in the midst of scanning will
+ have unexpected results since flex buffers its input;
+ use yyrestart() instead. Once scanning terminates
+ because an end-of-file has been seen, you can assign
+ yyin at the new input file and then call the scanner
+ again to continue scanning.
+
+ - void yyrestart( FILE *new_file ) may be called to point
+ yyin at the new input file. The switch-over to the new
+ file is immediate (any previously buffered-up input is
+ lost). Note that calling yyrestart() with yyin as an
+ argument thus throws away the current input buffer and
+ continues scanning the same input file.
+
+ - FILE *yyout is the file to which ECHO actions are done.
+ It can be reassigned by the user.
+
+ - YY_CURRENT_BUFFER returns a YY_BUFFER_STATE handle to
+ the current buffer.
+
+ - YY_START returns an integer value corresponding to the
+ current start condition. You can subsequently use this
+ value with BEGIN to return to that start condition.
+
+INTERFACING WITH YACC
+ One of the main uses of flex is as a companion to the yacc
+ parser-generator. yacc parsers expect to call a routine
+ named yylex() to find the next input token. The routine is
+ supposed to return the type of the next token as well as
+ putting any associated value in the global yylval. To use
+ flex with yacc, one specifies the -d option to yacc to
+ instruct it to generate the file y.tab.h containing defini-
+ tions of all the %tokens appearing in the yacc input. This
+ file is then included in the flex scanner. For example, if
+ one of the tokens is "TOK_NUMBER", part of the scanner might
+ look like:
+
+ %{
+ #include "y.tab.h"
+ %}
+
+
+
+Version 2.5 Last change: April 1995 29
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ %%
+
+ [0-9]+ yylval = atoi( yytext ); return TOK_NUMBER;
+
+
+OPTIONS
+ flex has the following options:
+
+ -b Generate backing-up information to lex.backup. This is
+ a list of scanner states which require backing up and
+ the input characters on which they do so. By adding
+ rules one can remove backing-up states. If all
+ backing-up states are eliminated and -Cf or -CF is
+ used, the generated scanner will run faster (see the -p
+ flag). Only users who wish to squeeze every last cycle
+ out of their scanners need worry about this option.
+ (See the section on Performance Considerations below.)
+
+ -c is a do-nothing, deprecated option included for POSIX
+ compliance.
+
+ -d makes the generated scanner run in debug mode. When-
+ ever a pattern is recognized and the global
+ yy_flex_debug is non-zero (which is the default), the
+ scanner will write to stderr a line of the form:
+
+ --accepting rule at line 53 ("the matched text")
+
+ The line number refers to the location of the rule in
+ the file defining the scanner (i.e., the file that was
+ fed to flex). Messages are also generated when the
+ scanner backs up, accepts the default rule, reaches the
+ end of its input buffer (or encounters a NUL; at this
+ point, the two look the same as far as the scanner's
+ concerned), or reaches an end-of-file.
+
+ -f specifies fast scanner. No table compression is done
+ and stdio is bypassed. The result is large but fast.
+ This option is equivalent to -Cfr (see below).
+
+ -h generates a "help" summary of flex's options to stdout
+ and then exits. -? and --help are synonyms for -h.
+
+ -i instructs flex to generate a case-insensitive scanner.
+ The case of letters given in the flex input patterns
+ will be ignored, and tokens in the input will be
+ matched regardless of case. The matched text given in
+ yytext will have the preserved case (i.e., it will not
+ be folded).
+
+ -l turns on maximum compatibility with the original AT&T
+ lex implementation. Note that this does not mean full
+
+
+
+Version 2.5 Last change: April 1995 30
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ compatibility. Use of this option costs a considerable
+ amount of performance, and it cannot be used with the
+ -+, -f, -F, -Cf, or -CF options. For details on the
+ compatibilities it provides, see the section "Incompa-
+ tibilities With Lex And POSIX" below. This option also
+ results in the name YY_FLEX_LEX_COMPAT being #define'd
+ in the generated scanner.
+
+ -n is another do-nothing, deprecated option included only
+ for POSIX compliance.
+
+ -p generates a performance report to stderr. The report
+ consists of comments regarding features of the flex
+ input file which will cause a serious loss of perfor-
+ mance in the resulting scanner. If you give the flag
+ twice, you will also get comments regarding features
+ that lead to minor performance losses.
+
+ Note that the use of REJECT, %option yylineno, and
+ variable trailing context (see the Deficiencies / Bugs
+ section below) entails a substantial performance
+ penalty; use of yymore(), the ^ operator, and the -I
+ flag entail minor performance penalties.
+
+ -s causes the default rule (that unmatched scanner input
+ is echoed to stdout) to be suppressed. If the scanner
+ encounters input that does not match any of its rules,
+ it aborts with an error. This option is useful for
+ finding holes in a scanner's rule set.
+
+ -t instructs flex to write the scanner it generates to
+ standard output instead of lex.yy.c.
+
+ -v specifies that flex should write to stderr a summary of
+ statistics regarding the scanner it generates. Most of
+ the statistics are meaningless to the casual flex user,
+ but the first line identifies the version of flex (same
+ as reported by -V), and the next line the flags used
+ when generating the scanner, including those that are
+ on by default.
+
+ -w suppresses warning messages.
+
+ -B instructs flex to generate a batch scanner, the oppo-
+ site of interactive scanners generated by -I (see
+ below). In general, you use -B when you are certain
+ that your scanner will never be used interactively, and
+ you want to squeeze a little more performance out of
+ it. If your goal is instead to squeeze out a lot more
+ performance, you should be using the -Cf or -CF
+ options (discussed below), which turn on -B automati-
+ cally anyway.
+
+
+
+Version 2.5 Last change: April 1995 31
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ -F specifies that the fast scanner table representation
+ should be used (and stdio bypassed). This representa-
+ tion is about as fast as the full table representation
+ (-f), and for some sets of patterns will be consider-
+ ably smaller (and for others, larger). In general, if
+ the pattern set contains both "keywords" and a catch-
+ all, "identifier" rule, such as in the set:
+
+ "case" return TOK_CASE;
+ "switch" return TOK_SWITCH;
+ ...
+ "default" return TOK_DEFAULT;
+ [a-z]+ return TOK_ID;
+
+ then you're better off using the full table representa-
+ tion. If only the "identifier" rule is present and you
+ then use a hash table or some such to detect the key-
+ words, you're better off using -F.
+
+ This option is equivalent to -CFr (see below). It can-
+ not be used with -+.
+
+ -I instructs flex to generate an interactive scanner. An
+ interactive scanner is one that only looks ahead to
+ decide what token has been matched if it absolutely
+ must. It turns out that always looking one extra char-
+ acter ahead, even if the scanner has already seen
+ enough text to disambiguate the current token, is a bit
+ faster than only looking ahead when necessary. But
+ scanners that always look ahead give dreadful interac-
+ tive performance; for example, when a user types a new-
+ line, it is not recognized as a newline token until
+ they enter another token, which often means typing in
+ another whole line.
+
+ Flex scanners default to interactive unless you use the
+ -Cf or -CF table-compression options (see below).
+ That's because if you're looking for high-performance
+ you should be using one of these options, so if you
+ didn't, flex assumes you'd rather trade off a bit of
+ run-time performance for intuitive interactive
+ behavior. Note also that you cannot use -I in conjunc-
+ tion with -Cf or -CF. Thus, this option is not really
+ needed; it is on by default for all those cases in
+ which it is allowed.
+
+ You can force a scanner to not be interactive by using
+ -B (see above).
+
+ -L instructs flex not to generate #line directives.
+ Without this option, flex peppers the generated scanner
+ with #line directives so error messages in the actions
+
+
+
+Version 2.5 Last change: April 1995 32
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ will be correctly located with respect to either the
+ original flex input file (if the errors are due to code
+ in the input file), or lex.yy.c (if the errors are
+ flex's fault -- you should report these sorts of errors
+ to the email address given below).
+
+ -T makes flex run in trace mode. It will generate a lot
+ of messages to stderr concerning the form of the input
+ and the resultant non-deterministic and deterministic
+ finite automata. This option is mostly for use in
+ maintaining flex.
+
+ -V prints the version number to stdout and exits. --ver-
+ sion is a synonym for -V.
+
+ -7 instructs flex to generate a 7-bit scanner, i.e., one
+ which can only recognized 7-bit characters in its
+ input. The advantage of using -7 is that the scanner's
+ tables can be up to half the size of those generated
+ using the -8 option (see below). The disadvantage is
+ that such scanners often hang or crash if their input
+ contains an 8-bit character.
+
+ Note, however, that unless you generate your scanner
+ using the -Cf or -CF table compression options, use of
+ -7 will save only a small amount of table space, and
+ make your scanner considerably less portable. Flex's
+ default behavior is to generate an 8-bit scanner unless
+ you use the -Cf or -CF, in which case flex defaults to
+ generating 7-bit scanners unless your site was always
+ configured to generate 8-bit scanners (as will often be
+ the case with non-USA sites). You can tell whether
+ flex generated a 7-bit or an 8-bit scanner by inspect-
+ ing the flag summary in the -v output as described
+ above.
+
+ Note that if you use -Cfe or -CFe (those table compres-
+ sion options, but also using equivalence classes as
+ discussed see below), flex still defaults to generating
+ an 8-bit scanner, since usually with these compression
+ options full 8-bit tables are not much more expensive
+ than 7-bit tables.
+
+ -8 instructs flex to generate an 8-bit scanner, i.e., one
+ which can recognize 8-bit characters. This flag is
+ only needed for scanners generated using -Cf or -CF, as
+ otherwise flex defaults to generating an 8-bit scanner
+ anyway.
+
+ See the discussion of -7 above for flex's default
+ behavior and the tradeoffs between 7-bit and 8-bit
+ scanners.
+
+
+
+Version 2.5 Last change: April 1995 33
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ -+ specifies that you want flex to generate a C++ scanner
+ class. See the section on Generating C++ Scanners
+ below for details.
+
+ -C[aefFmr]
+ controls the degree of table compression and, more gen-
+ erally, trade-offs between small scanners and fast
+ scanners.
+
+ -Ca ("align") instructs flex to trade off larger tables
+ in the generated scanner for faster performance because
+ the elements of the tables are better aligned for
+ memory access and computation. On some RISC architec-
+ tures, fetching and manipulating longwords is more
+ efficient than with smaller-sized units such as short-
+ words. This option can double the size of the tables
+ used by your scanner.
+
+ -Ce directs flex to construct equivalence classes,
+ i.e., sets of characters which have identical lexical
+ properties (for example, if the only appearance of
+ digits in the flex input is in the character class
+ "[0-9]" then the digits '0', '1', ..., '9' will all be
+ put in the same equivalence class). Equivalence
+ classes usually give dramatic reductions in the final
+ table/object file sizes (typically a factor of 2-5) and
+ are pretty cheap performance-wise (one array look-up
+ per character scanned).
+
+ -Cf specifies that the full scanner tables should be
+ generated - flex should not compress the tables by tak-
+ ing advantages of similar transition functions for dif-
+ ferent states.
+
+ -CF specifies that the alternate fast scanner represen-
+ tation (described above under the -F flag) should be
+ used. This option cannot be used with -+.
+
+ -Cm directs flex to construct meta-equivalence classes,
+ which are sets of equivalence classes (or characters,
+ if equivalence classes are not being used) that are
+ commonly used together. Meta-equivalence classes are
+ often a big win when using compressed tables, but they
+ have a moderate performance impact (one or two "if"
+ tests and one array look-up per character scanned).
+
+ -Cr causes the generated scanner to bypass use of the
+ standard I/O library (stdio) for input. Instead of
+ calling fread() or getc(), the scanner will use the
+ read() system call, resulting in a performance gain
+ which varies from system to system, but in general is
+ probably negligible unless you are also using -Cf or
+
+
+
+Version 2.5 Last change: April 1995 34
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ -CF. Using -Cr can cause strange behavior if, for exam-
+ ple, you read from yyin using stdio prior to calling
+ the scanner (because the scanner will miss whatever
+ text your previous reads left in the stdio input
+ buffer).
+
+ -Cr has no effect if you define YY_INPUT (see The Gen-
+ erated Scanner above).
+
+ A lone -C specifies that the scanner tables should be
+ compressed but neither equivalence classes nor meta-
+ equivalence classes should be used.
+
+ The options -Cf or -CF and -Cm do not make sense
+ together - there is no opportunity for meta-equivalence
+ classes if the table is not being compressed. Other-
+ wise the options may be freely mixed, and are cumula-
+ tive.
+
+ The default setting is -Cem, which specifies that flex
+ should generate equivalence classes and meta-
+ equivalence classes. This setting provides the highest
+ degree of table compression. You can trade off
+ faster-executing scanners at the cost of larger tables
+ with the following generally being true:
+
+ slowest & smallest
+ -Cem
+ -Cm
+ -Ce
+ -C
+ -C{f,F}e
+ -C{f,F}
+ -C{f,F}a
+ fastest & largest
+
+ Note that scanners with the smallest tables are usually
+ generated and compiled the quickest, so during develop-
+ ment you will usually want to use the default, maximal
+ compression.
+
+ -Cfe is often a good compromise between speed and size
+ for production scanners.
+
+ -ooutput
+ directs flex to write the scanner to the file output
+ instead of lex.yy.c. If you combine -o with the -t
+ option, then the scanner is written to stdout but its
+ #line directives (see the -L option above) refer to the
+ file output.
+
+ -Pprefix
+
+
+
+Version 2.5 Last change: April 1995 35
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ changes the default yy prefix used by flex for all
+ globally-visible variable and function names to instead
+ be prefix. For example, -Pfoo changes the name of
+ yytext to footext. It also changes the name of the
+ default output file from lex.yy.c to lex.foo.c. Here
+ are all of the names affected:
+
+ yy_create_buffer
+ yy_delete_buffer
+ yy_flex_debug
+ yy_init_buffer
+ yy_flush_buffer
+ yy_load_buffer_state
+ yy_switch_to_buffer
+ yyin
+ yyleng
+ yylex
+ yylineno
+ yyout
+ yyrestart
+ yytext
+ yywrap
+
+ (If you are using a C++ scanner, then only yywrap and
+ yyFlexLexer are affected.) Within your scanner itself,
+ you can still refer to the global variables and func-
+ tions using either version of their name; but exter-
+ nally, they have the modified name.
+
+ This option lets you easily link together multiple flex
+ programs into the same executable. Note, though, that
+ using this option also renames yywrap(), so you now
+ must either provide your own (appropriately-named) ver-
+ sion of the routine for your scanner, or use %option
+ noyywrap, as linking with -lfl no longer provides one
+ for you by default.
+
+ -Sskeleton_file
+ overrides the default skeleton file from which flex
+ constructs its scanners. You'll never need this option
+ unless you are doing flex maintenance or development.
+
+ flex also provides a mechanism for controlling options
+ within the scanner specification itself, rather than from
+ the flex command-line. This is done by including %option
+ directives in the first section of the scanner specifica-
+ tion. You can specify multiple options with a single
+ %option directive, and multiple directives in the first sec-
+ tion of your flex input file.
+
+ Most options are given simply as names, optionally preceded
+ by the word "no" (with no intervening whitespace) to negate
+
+
+
+Version 2.5 Last change: April 1995 36
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ their meaning. A number are equivalent to flex flags or
+ their negation:
+
+ 7bit -7 option
+ 8bit -8 option
+ align -Ca option
+ backup -b option
+ batch -B option
+ c++ -+ option
+
+ caseful or
+ case-sensitive opposite of -i (default)
+
+ case-insensitive or
+ caseless -i option
+
+ debug -d option
+ default opposite of -s option
+ ecs -Ce option
+ fast -F option
+ full -f option
+ interactive -I option
+ lex-compat -l option
+ meta-ecs -Cm option
+ perf-report -p option
+ read -Cr option
+ stdout -t option
+ verbose -v option
+ warn opposite of -w option
+ (use "%option nowarn" for -w)
+
+ array equivalent to "%array"
+ pointer equivalent to "%pointer" (default)
+
+ Some %option's provide features otherwise not available:
+
+ always-interactive
+ instructs flex to generate a scanner which always con-
+ siders its input "interactive". Normally, on each new
+ input file the scanner calls isatty() in an attempt to
+ determine whether the scanner's input source is
+ interactive and thus should be read a character at a
+ time. When this option is used, however, then no such
+ call is made.
+
+ main directs flex to provide a default main() program for
+ the scanner, which simply calls yylex(). This option
+ implies noyywrap (see below).
+
+ never-interactive
+ instructs flex to generate a scanner which never con-
+ siders its input "interactive" (again, no call made to
+
+
+
+Version 2.5 Last change: April 1995 37
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ isatty()). This is the opposite of always-interactive.
+
+ stack
+ enables the use of start condition stacks (see Start
+ Conditions above).
+
+ stdinit
+ if set (i.e., %option stdinit) initializes yyin and
+ yyout to stdin and stdout, instead of the default of
+ nil. Some existing lex programs depend on this
+ behavior, even though it is not compliant with ANSI C,
+ which does not require stdin and stdout to be compile-
+ time constant.
+
+ yylineno
+ directs flex to generate a scanner that maintains the
+ number of the current line read from its input in the
+ global variable yylineno. This option is implied by
+ %option lex-compat.
+
+ yywrap
+ if unset (i.e., %option noyywrap), makes the scanner
+ not call yywrap() upon an end-of-file, but simply
+ assume that there are no more files to scan (until the
+ user points yyin at a new file and calls yylex()
+ again).
+
+ flex scans your rule actions to determine whether you use
+ the REJECT or yymore() features. The reject and yymore
+ options are available to override its decision as to whether
+ you use the options, either by setting them (e.g., %option
+ reject) to indicate the feature is indeed used, or unsetting
+ them to indicate it actually is not used (e.g., %option
+ noyymore).
+
+ Three options take string-delimited values, offset with '=':
+
+ %option outfile="ABC"
+
+ is equivalent to -oABC, and
+
+ %option prefix="XYZ"
+
+ is equivalent to -PXYZ. Finally,
+
+ %option yyclass="foo"
+
+ only applies when generating a C++ scanner ( -+ option). It
+ informs flex that you have derived foo as a subclass of
+ yyFlexLexer, so flex will place your actions in the member
+ function foo::yylex() instead of yyFlexLexer::yylex(). It
+ also generates a yyFlexLexer::yylex() member function that
+
+
+
+Version 2.5 Last change: April 1995 38
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ emits a run-time error (by invoking
+ yyFlexLexer::LexerError()) if called. See Generating C++
+ Scanners, below, for additional information.
+
+ A number of options are available for lint purists who want
+ to suppress the appearance of unneeded routines in the gen-
+ erated scanner. Each of the following, if unset (e.g.,
+ %option nounput ), results in the corresponding routine not
+ appearing in the generated scanner:
+
+ input, unput
+ yy_push_state, yy_pop_state, yy_top_state
+ yy_scan_buffer, yy_scan_bytes, yy_scan_string
+
+ (though yy_push_state() and friends won't appear anyway
+ unless you use %option stack).
+
+PERFORMANCE CONSIDERATIONS
+ The main design goal of flex is that it generate high-
+ performance scanners. It has been optimized for dealing
+ well with large sets of rules. Aside from the effects on
+ scanner speed of the table compression -C options outlined
+ above, there are a number of options/actions which degrade
+ performance. These are, from most expensive to least:
+
+ REJECT
+ %option yylineno
+ arbitrary trailing context
+
+ pattern sets that require backing up
+ %array
+ %option interactive
+ %option always-interactive
+
+ '^' beginning-of-line operator
+ yymore()
+
+ with the first three all being quite expensive and the last
+ two being quite cheap. Note also that unput() is imple-
+ mented as a routine call that potentially does quite a bit
+ of work, while yyless() is a quite-cheap macro; so if just
+ putting back some excess text you scanned, use yyless().
+
+ REJECT should be avoided at all costs when performance is
+ important. It is a particularly expensive option.
+
+ Getting rid of backing up is messy and often may be an enor-
+ mous amount of work for a complicated scanner. In princi-
+ pal, one begins by using the -b flag to generate a
+ lex.backup file. For example, on the input
+
+ %%
+
+
+
+Version 2.5 Last change: April 1995 39
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ foo return TOK_KEYWORD;
+ foobar return TOK_KEYWORD;
+
+ the file looks like:
+
+ State #6 is non-accepting -
+ associated rule line numbers:
+ 2 3
+ out-transitions: [ o ]
+ jam-transitions: EOF [ \001-n p-\177 ]
+
+ State #8 is non-accepting -
+ associated rule line numbers:
+ 3
+ out-transitions: [ a ]
+ jam-transitions: EOF [ \001-` b-\177 ]
+
+ State #9 is non-accepting -
+ associated rule line numbers:
+ 3
+ out-transitions: [ r ]
+ jam-transitions: EOF [ \001-q s-\177 ]
+
+ Compressed tables always back up.
+
+ The first few lines tell us that there's a scanner state in
+ which it can make a transition on an 'o' but not on any
+ other character, and that in that state the currently
+ scanned text does not match any rule. The state occurs when
+ trying to match the rules found at lines 2 and 3 in the
+ input file. If the scanner is in that state and then reads
+ something other than an 'o', it will have to back up to find
+ a rule which is matched. With a bit of headscratching one
+ can see that this must be the state it's in when it has seen
+ "fo". When this has happened, if anything other than
+ another 'o' is seen, the scanner will have to back up to
+ simply match the 'f' (by the default rule).
+
+ The comment regarding State #8 indicates there's a problem
+ when "foob" has been scanned. Indeed, on any character
+ other than an 'a', the scanner will have to back up to
+ accept "foo". Similarly, the comment for State #9 concerns
+ when "fooba" has been scanned and an 'r' does not follow.
+
+ The final comment reminds us that there's no point going to
+ all the trouble of removing backing up from the rules unless
+ we're using -Cf or -CF, since there's no performance gain
+ doing so with compressed scanners.
+
+ The way to remove the backing up is to add "error" rules:
+
+ %%
+
+
+
+Version 2.5 Last change: April 1995 40
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ foo return TOK_KEYWORD;
+ foobar return TOK_KEYWORD;
+
+ fooba |
+ foob |
+ fo {
+ /* false alarm, not really a keyword */
+ return TOK_ID;
+ }
+
+
+ Eliminating backing up among a list of keywords can also be
+ done using a "catch-all" rule:
+
+ %%
+ foo return TOK_KEYWORD;
+ foobar return TOK_KEYWORD;
+
+ [a-z]+ return TOK_ID;
+
+ This is usually the best solution when appropriate.
+
+ Backing up messages tend to cascade. With a complicated set
+ of rules it's not uncommon to get hundreds of messages. If
+ one can decipher them, though, it often only takes a dozen
+ or so rules to eliminate the backing up (though it's easy to
+ make a mistake and have an error rule accidentally match a
+ valid token. A possible future flex feature will be to
+ automatically add rules to eliminate backing up).
+
+ It's important to keep in mind that you gain the benefits of
+ eliminating backing up only if you eliminate every instance
+ of backing up. Leaving just one means you gain nothing.
+
+ Variable trailing context (where both the leading and trail-
+ ing parts do not have a fixed length) entails almost the
+ same performance loss as REJECT (i.e., substantial). So
+ when possible a rule like:
+
+ %%
+ mouse|rat/(cat|dog) run();
+
+ is better written:
+
+ %%
+ mouse/cat|dog run();
+ rat/cat|dog run();
+
+ or as
+
+ %%
+ mouse|rat/cat run();
+
+
+
+Version 2.5 Last change: April 1995 41
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ mouse|rat/dog run();
+
+ Note that here the special '|' action does not provide any
+ savings, and can even make things worse (see Deficiencies /
+ Bugs below).
+
+ Another area where the user can increase a scanner's perfor-
+ mance (and one that's easier to implement) arises from the
+ fact that the longer the tokens matched, the faster the
+ scanner will run. This is because with long tokens the pro-
+ cessing of most input characters takes place in the (short)
+ inner scanning loop, and does not often have to go through
+ the additional work of setting up the scanning environment
+ (e.g., yytext) for the action. Recall the scanner for C
+ comments:
+
+ %x comment
+ %%
+ int line_num = 1;
+
+ "/*" BEGIN(comment);
+
+ <comment>[^*\n]*
+ <comment>"*"+[^*/\n]*
+ <comment>\n ++line_num;
+ <comment>"*"+"/" BEGIN(INITIAL);
+
+ This could be sped up by writing it as:
+
+ %x comment
+ %%
+ int line_num = 1;
+
+ "/*" BEGIN(comment);
+
+ <comment>[^*\n]*
+ <comment>[^*\n]*\n ++line_num;
+ <comment>"*"+[^*/\n]*
+ <comment>"*"+[^*/\n]*\n ++line_num;
+ <comment>"*"+"/" BEGIN(INITIAL);
+
+ Now instead of each newline requiring the processing of
+ another action, recognizing the newlines is "distributed"
+ over the other rules to keep the matched text as long as
+ possible. Note that adding rules does not slow down the
+ scanner! The speed of the scanner is independent of the
+ number of rules or (modulo the considerations given at the
+ beginning of this section) how complicated the rules are
+ with regard to operators such as '*' and '|'.
+
+ A final example in speeding up a scanner: suppose you want
+ to scan through a file containing identifiers and keywords,
+
+
+
+Version 2.5 Last change: April 1995 42
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ one per line and with no other extraneous characters, and
+ recognize all the keywords. A natural first approach is:
+
+ %%
+ asm |
+ auto |
+ break |
+ ... etc ...
+ volatile |
+ while /* it's a keyword */
+
+ .|\n /* it's not a keyword */
+
+ To eliminate the back-tracking, introduce a catch-all rule:
+
+ %%
+ asm |
+ auto |
+ break |
+ ... etc ...
+ volatile |
+ while /* it's a keyword */
+
+ [a-z]+ |
+ .|\n /* it's not a keyword */
+
+ Now, if it's guaranteed that there's exactly one word per
+ line, then we can reduce the total number of matches by a
+ half by merging in the recognition of newlines with that of
+ the other tokens:
+
+ %%
+ asm\n |
+ auto\n |
+ break\n |
+ ... etc ...
+ volatile\n |
+ while\n /* it's a keyword */
+
+ [a-z]+\n |
+ .|\n /* it's not a keyword */
+
+ One has to be careful here, as we have now reintroduced
+ backing up into the scanner. In particular, while we know
+ that there will never be any characters in the input stream
+ other than letters or newlines, flex can't figure this out,
+ and it will plan for possibly needing to back up when it has
+ scanned a token like "auto" and then the next character is
+ something other than a newline or a letter. Previously it
+ would then just match the "auto" rule and be done, but now
+ it has no "auto" rule, only a "auto\n" rule. To eliminate
+ the possibility of backing up, we could either duplicate all
+
+
+
+Version 2.5 Last change: April 1995 43
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ rules but without final newlines, or, since we never expect
+ to encounter such an input and therefore don't how it's
+ classified, we can introduce one more catch-all rule, this
+ one which doesn't include a newline:
+
+ %%
+ asm\n |
+ auto\n |
+ break\n |
+ ... etc ...
+ volatile\n |
+ while\n /* it's a keyword */
+
+ [a-z]+\n |
+ [a-z]+ |
+ .|\n /* it's not a keyword */
+
+ Compiled with -Cf, this is about as fast as one can get a
+ flex scanner to go for this particular problem.
+
+ A final note: flex is slow when matching NUL's, particularly
+ when a token contains multiple NUL's. It's best to write
+ rules which match short amounts of text if it's anticipated
+ that the text will often include NUL's.
+
+ Another final note regarding performance: as mentioned above
+ in the section How the Input is Matched, dynamically resiz-
+ ing yytext to accommodate huge tokens is a slow process
+ because it presently requires that the (huge) token be res-
+ canned from the beginning. Thus if performance is vital,
+ you should attempt to match "large" quantities of text but
+ not "huge" quantities, where the cutoff between the two is
+ at about 8K characters/token.
+
+GENERATING C++ SCANNERS
+ flex provides two different ways to generate scanners for
+ use with C++. The first way is to simply compile a scanner
+ generated by flex using a C++ compiler instead of a C com-
+ piler. You should not encounter any compilations errors
+ (please report any you find to the email address given in
+ the Author section below). You can then use C++ code in
+ your rule actions instead of C code. Note that the default
+ input source for your scanner remains yyin, and default
+ echoing is still done to yyout. Both of these remain FILE *
+ variables and not C++ streams.
+
+ You can also use flex to generate a C++ scanner class, using
+ the -+ option (or, equivalently, %option c++), which is
+ automatically specified if the name of the flex executable
+ ends in a '+', such as flex++. When using this option, flex
+ defaults to generating the scanner to the file lex.yy.cc
+ instead of lex.yy.c. The generated scanner includes the
+
+
+
+Version 2.5 Last change: April 1995 44
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ header file FlexLexer.h, which defines the interface to two
+ C++ classes.
+
+ The first class, FlexLexer, provides an abstract base class
+ defining the general scanner class interface. It provides
+ the following member functions:
+
+ const char* YYText()
+ returns the text of the most recently matched token,
+ the equivalent of yytext.
+
+ int YYLeng()
+ returns the length of the most recently matched token,
+ the equivalent of yyleng.
+
+ int lineno() const
+ returns the current input line number (see %option
+ yylineno), or 1 if %option yylineno was not used.
+
+ void set_debug( int flag )
+ sets the debugging flag for the scanner, equivalent to
+ assigning to yy_flex_debug (see the Options section
+ above). Note that you must build the scanner using
+ %option debug to include debugging information in it.
+
+ int debug() const
+ returns the current setting of the debugging flag.
+
+ Also provided are member functions equivalent to
+ yy_switch_to_buffer(), yy_create_buffer() (though the first
+ argument is an istream* object pointer and not a FILE*),
+ yy_flush_buffer(), yy_delete_buffer(), and yyrestart()
+ (again, the first argument is a istream* object pointer).
+
+ The second class defined in FlexLexer.h is yyFlexLexer,
+ which is derived from FlexLexer. It defines the following
+ additional member functions:
+
+ yyFlexLexer( istream* arg_yyin = 0, ostream* arg_yyout = 0 )
+ constructs a yyFlexLexer object using the given streams
+ for input and output. If not specified, the streams
+ default to cin and cout, respectively.
+
+ virtual int yylex()
+ performs the same role is yylex() does for ordinary
+ flex scanners: it scans the input stream, consuming
+ tokens, until a rule's action returns a value. If you
+ derive a subclass S from yyFlexLexer and want to access
+ the member functions and variables of S inside yylex(),
+ then you need to use %option yyclass="S" to inform flex
+ that you will be using that subclass instead of yyFlex-
+ Lexer. In this case, rather than generating
+
+
+
+Version 2.5 Last change: April 1995 45
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ yyFlexLexer::yylex(), flex generates S::yylex() (and
+ also generates a dummy yyFlexLexer::yylex() that calls
+ yyFlexLexer::LexerError() if called).
+
+ virtual void switch_streams(istream* new_in = 0,
+ ostream* new_out = 0) reassigns yyin to new_in (if
+ non-nil) and yyout to new_out (ditto), deleting the
+ previous input buffer if yyin is reassigned.
+
+ int yylex( istream* new_in, ostream* new_out = 0 )
+ first switches the input streams via switch_streams(
+ new_in, new_out ) and then returns the value of
+ yylex().
+
+ In addition, yyFlexLexer defines the following protected
+ virtual functions which you can redefine in derived classes
+ to tailor the scanner:
+
+ virtual int LexerInput( char* buf, int max_size )
+ reads up to max_size characters into buf and returns
+ the number of characters read. To indicate end-of-
+ input, return 0 characters. Note that "interactive"
+ scanners (see the -B and -I flags) define the macro
+ YY_INTERACTIVE. If you redefine LexerInput() and need
+ to take different actions depending on whether or not
+ the scanner might be scanning an interactive input
+ source, you can test for the presence of this name via
+ #ifdef.
+
+ virtual void LexerOutput( const char* buf, int size )
+ writes out size characters from the buffer buf, which,
+ while NUL-terminated, may also contain "internal" NUL's
+ if the scanner's rules can match text with NUL's in
+ them.
+
+ virtual void LexerError( const char* msg )
+ reports a fatal error message. The default version of
+ this function writes the message to the stream cerr and
+ exits.
+
+ Note that a yyFlexLexer object contains its entire scanning
+ state. Thus you can use such objects to create reentrant
+ scanners. You can instantiate multiple instances of the
+ same yyFlexLexer class, and you can also combine multiple
+ C++ scanner classes together in the same program using the
+ -P option discussed above.
+
+ Finally, note that the %array feature is not available to
+ C++ scanner classes; you must use %pointer (the default).
+
+ Here is an example of a simple C++ scanner:
+
+
+
+
+Version 2.5 Last change: April 1995 46
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ // An example of using the flex C++ scanner class.
+
+ %{
+ int mylineno = 0;
+ %}
+
+ string \"[^\n"]+\"
+
+ ws [ \t]+
+
+ alpha [A-Za-z]
+ dig [0-9]
+ name ({alpha}|{dig}|\$)({alpha}|{dig}|[_.\-/$])*
+ num1 [-+]?{dig}+\.?([eE][-+]?{dig}+)?
+ num2 [-+]?{dig}*\.{dig}+([eE][-+]?{dig}+)?
+ number {num1}|{num2}
+
+ %%
+
+ {ws} /* skip blanks and tabs */
+
+ "/*" {
+ int c;
+
+ while((c = yyinput()) != 0)
+ {
+ if(c == '\n')
+ ++mylineno;
+
+ else if(c == '*')
+ {
+ if((c = yyinput()) == '/')
+ break;
+ else
+ unput(c);
+ }
+ }
+ }
+
+ {number} cout << "number " << YYText() << '\n';
+
+ \n mylineno++;
+
+ {name} cout << "name " << YYText() << '\n';
+
+ {string} cout << "string " << YYText() << '\n';
+
+ %%
+
+ int main( int /* argc */, char** /* argv */ )
+ {
+ FlexLexer* lexer = new yyFlexLexer;
+
+
+
+Version 2.5 Last change: April 1995 47
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ while(lexer->yylex() != 0)
+ ;
+ return 0;
+ }
+ If you want to create multiple (different) lexer classes,
+ you use the -P flag (or the prefix= option) to rename each
+ yyFlexLexer to some other xxFlexLexer. You then can include
+ <FlexLexer.h> in your other sources once per lexer class,
+ first renaming yyFlexLexer as follows:
+
+ #undef yyFlexLexer
+ #define yyFlexLexer xxFlexLexer
+ #include <FlexLexer.h>
+
+ #undef yyFlexLexer
+ #define yyFlexLexer zzFlexLexer
+ #include <FlexLexer.h>
+
+ if, for example, you used %option prefix="xx" for one of
+ your scanners and %option prefix="zz" for the other.
+
+ IMPORTANT: the present form of the scanning class is experi-
+ mental and may change considerably between major releases.
+
+INCOMPATIBILITIES WITH LEX AND POSIX
+ flex is a rewrite of the AT&T Unix lex tool (the two imple-
+ mentations do not share any code, though), with some exten-
+ sions and incompatibilities, both of which are of concern to
+ those who wish to write scanners acceptable to either imple-
+ mentation. Flex is fully compliant with the POSIX lex
+ specification, except that when using %pointer (the
+ default), a call to unput() destroys the contents of yytext,
+ which is counter to the POSIX specification.
+
+ In this section we discuss all of the known areas of incom-
+ patibility between flex, AT&T lex, and the POSIX specifica-
+ tion.
+
+ flex's -l option turns on maximum compatibility with the
+ original AT&T lex implementation, at the cost of a major
+ loss in the generated scanner's performance. We note below
+ which incompatibilities can be overcome using the -l option.
+
+ flex is fully compatible with lex with the following excep-
+ tions:
+
+ - The undocumented lex scanner internal variable yylineno
+ is not supported unless -l or %option yylineno is used.
+
+ yylineno should be maintained on a per-buffer basis,
+ rather than a per-scanner (single global variable)
+ basis.
+
+
+
+Version 2.5 Last change: April 1995 48
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ yylineno is not part of the POSIX specification.
+
+ - The input() routine is not redefinable, though it may
+ be called to read characters following whatever has
+ been matched by a rule. If input() encounters an end-
+ of-file the normal yywrap() processing is done. A
+ ``real'' end-of-file is returned by input() as EOF.
+
+ Input is instead controlled by defining the YY_INPUT
+ macro.
+
+ The flex restriction that input() cannot be redefined
+ is in accordance with the POSIX specification, which
+ simply does not specify any way of controlling the
+ scanner's input other than by making an initial assign-
+ ment to yyin.
+
+ - The unput() routine is not redefinable. This restric-
+ tion is in accordance with POSIX.
+
+ - flex scanners are not as reentrant as lex scanners. In
+ particular, if you have an interactive scanner and an
+ interrupt handler which long-jumps out of the scanner,
+ and the scanner is subsequently called again, you may
+ get the following message:
+
+ fatal flex scanner internal error--end of buffer missed
+
+ To reenter the scanner, first use
+
+ yyrestart( yyin );
+
+ Note that this call will throw away any buffered input;
+ usually this isn't a problem with an interactive
+ scanner.
+
+ Also note that flex C++ scanner classes are reentrant,
+ so if using C++ is an option for you, you should use
+ them instead. See "Generating C++ Scanners" above for
+ details.
+
+ - output() is not supported. Output from the ECHO macro
+ is done to the file-pointer yyout (default stdout).
+
+ output() is not part of the POSIX specification.
+
+ - lex does not support exclusive start conditions (%x),
+ though they are in the POSIX specification.
+
+ - When definitions are expanded, flex encloses them in
+ parentheses. With lex, the following:
+
+
+
+
+Version 2.5 Last change: April 1995 49
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ NAME [A-Z][A-Z0-9]*
+ %%
+ foo{NAME}? printf( "Found it\n" );
+ %%
+
+ will not match the string "foo" because when the macro
+ is expanded the rule is equivalent to "foo[A-Z][A-Z0-
+ 9]*?" and the precedence is such that the '?' is asso-
+ ciated with "[A-Z0-9]*". With flex, the rule will be
+ expanded to "foo([A-Z][A-Z0-9]*)?" and so the string
+ "foo" will match.
+
+ Note that if the definition begins with ^ or ends with
+ $ then it is not expanded with parentheses, to allow
+ these operators to appear in definitions without losing
+ their special meanings. But the <s>, /, and <<EOF>>
+ operators cannot be used in a flex definition.
+
+ Using -l results in the lex behavior of no parentheses
+ around the definition.
+
+ The POSIX specification is that the definition be
+ enclosed in parentheses.
+
+ - Some implementations of lex allow a rule's action to
+ begin on a separate line, if the rule's pattern has
+ trailing whitespace:
+
+ %%
+ foo|bar<space here>
+ { foobar_action(); }
+
+ flex does not support this feature.
+
+ - The lex %r (generate a Ratfor scanner) option is not
+ supported. It is not part of the POSIX specification.
+
+ - After a call to unput(), yytext is undefined until the
+ next token is matched, unless the scanner was built
+ using %array. This is not the case with lex or the
+ POSIX specification. The -l option does away with this
+ incompatibility.
+
+ - The precedence of the {} (numeric range) operator is
+ different. lex interprets "abc{1,3}" as "match one,
+ two, or three occurrences of 'abc'", whereas flex
+ interprets it as "match 'ab' followed by one, two, or
+ three occurrences of 'c'". The latter is in agreement
+ with the POSIX specification.
+
+ - The precedence of the ^ operator is different. lex
+ interprets "^foo|bar" as "match either 'foo' at the
+
+
+
+Version 2.5 Last change: April 1995 50
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ beginning of a line, or 'bar' anywhere", whereas flex
+ interprets it as "match either 'foo' or 'bar' if they
+ come at the beginning of a line". The latter is in
+ agreement with the POSIX specification.
+
+ - The special table-size declarations such as %a sup-
+ ported by lex are not required by flex scanners; flex
+ ignores them.
+
+ - The name FLEX_SCANNER is #define'd so scanners may be
+ written for use with either flex or lex. Scanners also
+ include YY_FLEX_MAJOR_VERSION and YY_FLEX_MINOR_VERSION
+ indicating which version of flex generated the scanner
+ (for example, for the 2.5 release, these defines would
+ be 2 and 5 respectively).
+
+ The following flex features are not included in lex or the
+ POSIX specification:
+
+ C++ scanners
+ %option
+ start condition scopes
+ start condition stacks
+ interactive/non-interactive scanners
+ yy_scan_string() and friends
+ yyterminate()
+ yy_set_interactive()
+ yy_set_bol()
+ YY_AT_BOL()
+ <<EOF>>
+ <*>
+ YY_DECL
+ YY_START
+ YY_USER_ACTION
+ YY_USER_INIT
+ #line directives
+ %{}'s around actions
+ multiple actions on a line
+
+ plus almost all of the flex flags. The last feature in the
+ list refers to the fact that with flex you can put multiple
+ actions on the same line, separated with semi-colons, while
+ with lex, the following
+
+ foo handle_foo(); ++num_foos_seen;
+
+ is (rather surprisingly) truncated to
+
+ foo handle_foo();
+
+ flex does not truncate the action. Actions that are not
+ enclosed in braces are simply terminated at the end of the
+
+
+
+Version 2.5 Last change: April 1995 51
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ line.
+
+DIAGNOSTICS
+ warning, rule cannot be matched indicates that the given
+ rule cannot be matched because it follows other rules that
+ will always match the same text as it. For example, in the
+ following "foo" cannot be matched because it comes after an
+ identifier "catch-all" rule:
+
+ [a-z]+ got_identifier();
+ foo got_foo();
+
+ Using REJECT in a scanner suppresses this warning.
+
+ warning, -s option given but default rule can be matched
+ means that it is possible (perhaps only in a particular
+ start condition) that the default rule (match any single
+ character) is the only one that will match a particular
+ input. Since -s was given, presumably this is not intended.
+
+ reject_used_but_not_detected undefined or
+ yymore_used_but_not_detected undefined - These errors can
+ occur at compile time. They indicate that the scanner uses
+ REJECT or yymore() but that flex failed to notice the fact,
+ meaning that flex scanned the first two sections looking for
+ occurrences of these actions and failed to find any, but
+ somehow you snuck some in (via a #include file, for exam-
+ ple). Use %option reject or %option yymore to indicate to
+ flex that you really do use these features.
+
+ flex scanner jammed - a scanner compiled with -s has encoun-
+ tered an input string which wasn't matched by any of its
+ rules. This error can also occur due to internal problems.
+
+ token too large, exceeds YYLMAX - your scanner uses %array
+ and one of its rules matched a string longer than the YYLMAX
+ constant (8K bytes by default). You can increase the value
+ by #define'ing YYLMAX in the definitions section of your
+ flex input.
+
+ scanner requires -8 flag to use the character 'x' - Your
+ scanner specification includes recognizing the 8-bit charac-
+ ter 'x' and you did not specify the -8 flag, and your
+ scanner defaulted to 7-bit because you used the -Cf or -CF
+ table compression options. See the discussion of the -7
+ flag for details.
+
+ flex scanner push-back overflow - you used unput() to push
+ back so much text that the scanner's buffer could not hold
+ both the pushed-back text and the current token in yytext.
+ Ideally the scanner should dynamically resize the buffer in
+ this case, but at present it does not.
+
+
+
+Version 2.5 Last change: April 1995 52
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ input buffer overflow, can't enlarge buffer because scanner
+ uses REJECT - the scanner was working on matching an
+ extremely large token and needed to expand the input buffer.
+ This doesn't work with scanners that use REJECT.
+
+ fatal flex scanner internal error--end of buffer missed -
+ This can occur in an scanner which is reentered after a
+ long-jump has jumped out (or over) the scanner's activation
+ frame. Before reentering the scanner, use:
+
+ yyrestart( yyin );
+
+ or, as noted above, switch to using the C++ scanner class.
+
+ too many start conditions in <> you listed more start condi-
+ tions in a <> construct than exist (so you must have listed
+ at least one of them twice).
+
+FILES
+ -lfl library with which scanners must be linked.
+
+ lex.yy.c
+ generated scanner (called lexyy.c on some systems).
+
+ lex.yy.cc
+ generated C++ scanner class, when using -+.
+
+ <FlexLexer.h>
+ header file defining the C++ scanner base class, Flex-
+ Lexer, and its derived class, yyFlexLexer.
+
+ flex.skl
+ skeleton scanner. This file is only used when building
+ flex, not when flex executes.
+
+ lex.backup
+ backing-up information for -b flag (called lex.bck on
+ some systems).
+
+DEFICIENCIES / BUGS
+ Some trailing context patterns cannot be properly matched
+ and generate warning messages ("dangerous trailing con-
+ text"). These are patterns where the ending of the first
+ part of the rule matches the beginning of the second part,
+ such as "zx*/xy*", where the 'x*' matches the 'x' at the
+ beginning of the trailing context. (Note that the POSIX
+ draft states that the text matched by such patterns is unde-
+ fined.)
+
+ For some trailing context rules, parts which are actually
+ fixed-length are not recognized as such, leading to the
+ abovementioned performance loss. In particular, parts using
+
+
+
+Version 2.5 Last change: April 1995 53
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ '|' or {n} (such as "foo{3}") are always considered
+ variable-length.
+
+ Combining trailing context with the special '|' action can
+ result in fixed trailing context being turned into the more
+ expensive variable trailing context. For example, in the
+ following:
+
+ %%
+ abc |
+ xyz/def
+
+
+ Use of unput() invalidates yytext and yyleng, unless the
+ %array directive or the -l option has been used.
+
+ Pattern-matching of NUL's is substantially slower than
+ matching other characters.
+
+ Dynamic resizing of the input buffer is slow, as it entails
+ rescanning all the text matched so far by the current (gen-
+ erally huge) token.
+
+ Due to both buffering of input and read-ahead, you cannot
+ intermix calls to <stdio.h> routines, such as, for example,
+ getchar(), with flex rules and expect it to work. Call
+ input() instead.
+
+ The total table entries listed by the -v flag excludes the
+ number of table entries needed to determine what rule has
+ been matched. The number of entries is equal to the number
+ of DFA states if the scanner does not use REJECT, and some-
+ what greater than the number of states if it does.
+
+ REJECT cannot be used with the -f or -F options.
+
+ The flex internal algorithms need documentation.
+
+SEE ALSO
+ lex(1), yacc(1), sed(1), awk(1).
+
+ John Levine, Tony Mason, and Doug Brown, Lex & Yacc,
+ O'Reilly and Associates. Be sure to get the 2nd edition.
+
+ M. E. Lesk and E. Schmidt, LEX - Lexical Analyzer Generator
+
+ Alfred Aho, Ravi Sethi and Jeffrey Ullman, Compilers: Prin-
+ ciples, Techniques and Tools, Addison-Wesley (1986).
+ Describes the pattern-matching techniques used by flex
+ (deterministic finite automata).
+
+
+
+
+
+Version 2.5 Last change: April 1995 54
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+AUTHOR
+ Vern Paxson, with the help of many ideas and much inspira-
+ tion from Van Jacobson. Original version by Jef Poskanzer.
+ The fast table representation is a partial implementation of
+ a design done by Van Jacobson. The implementation was done
+ by Kevin Gong and Vern Paxson.
+
+ Thanks to the many flex beta-testers, feedbackers, and con-
+ tributors, especially Francois Pinard, Casey Leedom, Robert
+ Abramovitz, Stan Adermann, Terry Allen, David Barker-
+ Plummer, John Basrai, Neal Becker, Nelson H.F. Beebe,
+ benson@odi.com, Karl Berry, Peter A. Bigot, Simon Blanchard,
+ Keith Bostic, Frederic Brehm, Ian Brockbank, Kin Cho, Nick
+ Christopher, Brian Clapper, J.T. Conklin, Jason Coughlin,
+ Bill Cox, Nick Cropper, Dave Curtis, Scott David Daniels,
+ Chris G. Demetriou, Theo Deraadt, Mike Donahue, Chuck
+ Doucette, Tom Epperly, Leo Eskin, Chris Faylor, Chris
+ Flatters, Jon Forrest, Jeffrey Friedl, Joe Gayda, Kaveh R.
+ Ghazi, Wolfgang Glunz, Eric Goldman, Christopher M. Gould,
+ Ulrich Grepel, Peer Griebel, Jan Hajic, Charles Hemphill,
+ NORO Hideo, Jarkko Hietaniemi, Scott Hofmann, Jeff Honig,
+ Dana Hudes, Eric Hughes, John Interrante, Ceriel Jacobs,
+ Michal Jaegermann, Sakari Jalovaara, Jeffrey R. Jones, Henry
+ Juengst, Klaus Kaempf, Jonathan I. Kamens, Terrence O Kane,
+ Amir Katz, ken@ken.hilco.com, Kevin B. Kenny, Steve Kirsch,
+ Winfried Koenig, Marq Kole, Ronald Lamprecht, Greg Lee,
+ Rohan Lenard, Craig Leres, John Levine, Steve Liddle, David
+ Loffredo, Mike Long, Mohamed el Lozy, Brian Madsen, Malte,
+ Joe Marshall, Bengt Martensson, Chris Metcalf, Luke Mewburn,
+ Jim Meyering, R. Alexander Milowski, Erik Naggum, G.T.
+ Nicol, Landon Noll, James Nordby, Marc Nozell, Richard
+ Ohnemus, Karsten Pahnke, Sven Panne, Roland Pesch, Walter
+ Pelissero, Gaumond Pierre, Esmond Pitt, Jef Poskanzer, Joe
+ Rahmeh, Jarmo Raiha, Frederic Raimbault, Pat Rankin, Rick
+ Richardson, Kevin Rodgers, Kai Uwe Rommel, Jim Roskind,
+ Alberto Santini, Andreas Scherer, Darrell Schiebel, Raf
+ Schietekat, Doug Schmidt, Philippe Schnoebelen, Andreas
+ Schwab, Larry Schwimmer, Alex Siegel, Eckehard Stolz, Jan-
+ Erik Strvmquist, Mike Stump, Paul Stuart, Dave Tallman, Ian
+ Lance Taylor, Chris Thewalt, Richard M. Timoney, Jodi Tsai,
+ Paul Tuinenga, Gary Weik, Frank Whaley, Gerhard Wilhelms,
+ Kent Williams, Ken Yap, Ron Zellar, Nathan Zelle, David
+ Zuhn, and those whose names have slipped my marginal mail-
+ archiving skills but whose contributions are appreciated all
+ the same.
+
+ Thanks to Keith Bostic, Jon Forrest, Noah Friedman, John
+ Gilmore, Craig Leres, John Levine, Bob Mulcahy, G.T. Nicol,
+ Francois Pinard, Rich Salz, and Richard Stallman for help
+ with various distribution headaches.
+
+
+
+
+
+Version 2.5 Last change: April 1995 55
+
+
+
+
+
+
+FLEX(1) USER COMMANDS FLEX(1)
+
+
+
+ Thanks to Esmond Pitt and Earle Horton for 8-bit character
+ support; to Benson Margulies and Fred Burke for C++ support;
+ to Kent Williams and Tom Epperly for C++ class support; to
+ Ove Ewerlid for support of NUL's; and to Eric Hughes for
+ support of multiple buffers.
+
+ This work was primarily done when I was with the Real Time
+ Systems Group at the Lawrence Berkeley Laboratory in Berke-
+ ley, CA. Many thanks to all there for the support I
+ received.
+
+ Send comments to vern@ee.lbl.gov.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Version 2.5 Last change: April 1995 56
+
+
+
projects / unix-history / commitdiff