lex (1) and flex(1)

1. lex(1) and flex(1)

2. Lex public interface • FILE *yyin; /* set before calling yylex() */ • intyylex(); /* call once per token */ • char yytext[]; /* chars matched by yylex() */ • intyywrap(); /* end-of-file handler */

3. .l file format header %% body %% helper functions

4. Lex header • C code inside %{ … %} • prototypes for helper functions • #include’s that #define integer token categories • Macro definitions, e.g. letter [a-zA-Z] digit [0-9] ident {letter}({letter}|{digit})* • Warning: macros are fraught with peril

5. Lex body • Regular expressions with semantic actions “ “ { /* discard */ } {ident} { return IDENT; } “*” { return ASTERISK; } “.” { return PERIOD; } • Match the longest r.e. possible • Break ties with whichever appears first • If it fails to match: copy unmatched to stdout

6. Lex helper functions • Follows rules of ordinary C code • Compute lexical attributes • Do stuff the regular expressions can’t do • Write a yywrap() to switch files on EOF

7. Lex regular expressions • \c escapes for most operators • “s” match C string as-is (superescape) • r{m,n} match r between m and n times • r/s match r when s follows • ^r match r when at beginning of line • r$ match r when at end of line

8. struct token struct token { int category; char *text; intlinenumber; int column; char *filename; union literal value; }

9. “string removal tool” %% “zap me”

10. whitespace trimmer %% [ \t]+ putchar(‘ ‘); [ \t]+ /* drop entirely */

11. string replacement %% username printf(“%s”, getlogin() );

12. Line/word counter int lines=0, chars=0; %% \n ++lines; ++chars; . ++chars; %% main() { yylex(); printf(“lines: %d chars: %d\n”, lines, chars); }

13. Example: C reals • Is it: [0-9]*.[0-9]* • Is it: ([0-9]+.[0-9]* | [0-9]*.[0-9]+)