1 / 37

Lexical Analysis

Lexical Analysis. Why separate lexical and syntax analyses? simpler design efficiency portability. Tokens, Patterns, Lexemes. Tokens Terminal symbols in the grammar Patterns Description of a class of tokens Lexemes Words in the the source program. Languages .

adlai
Download Presentation

Lexical Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Lexical Analysis • Why separate lexical and syntax analyses? • simpler design • efficiency • portability by Neng-Fa Zhou

  2. Tokens, Patterns, Lexemes • Tokens • Terminal symbols in the grammar • Patterns • Description of a class of tokens • Lexemes • Words in the the source program by Neng-Fa Zhou

  3. Languages • Fixed and finite alphabet (vocabulary) • Finite length sentences • Possibly infinite number of sentences • Examples • Natural numbers {1,2,3,...10,11,...} • Strings over {a,b} anban • Terms on parts of a string • prefix, suffix, substring, proper .... by Neng-Fa Zhou

  4. Operations on Languages by Neng-Fa Zhou

  5. Examples L = {A,B,...,Z,a,b,...,z} D = {0,1,...,9} L  D : the set of letters and digits LD : a letter followed by a digit L4 : four-letter strings L* : all strings of letters, including e L(L  D)* : strings of letters and digits beginning with a letter D+ : strings of one or more digits by Neng-Fa Zhou

  6. Regular Expression(RE) • e is a RE • a symbol in S is a RE • Let r and s be REs. • (r) | (s) : or • (r)(s) : concatenation • (r)* : zero or more instances • (r)+ : one or more instances • (r)? : zero or one instance by Neng-Fa Zhou

  7. Examples Precedence of Operators all left associative high S = {a,b} 1. a|b 2. (a|b)(a|b) 3. a* 4. (a|b)* 5. a| a*b r* r+ r? rs low r|s by Neng-Fa Zhou

  8. Algebraic Properties of RE by Neng-Fa Zhou

  9. Regular Definitions d1 r1 d2 r2 di is a RE over S {d1,d2,...,di-1} .... dn rn not recursive by Neng-Fa Zhou

  10. Example-1 %{ int num_lines = 0, num_chars = 0; %} %% \n ++num_lines; ++num_chars; . ++num_chars; %% main() { yylex(); printf( "# of lines = %d, # of chars = %d\n", num_lines, num_chars ); } yywrap(){return 0;} by Neng-Fa Zhou

  11. Example-2 D [0-9] INT {D}{D}* %% {INT}("."{INT}((e|E)("+"|-)?{INT})?)? {printf("valid %s\n",yytext);} . {printf("unrecognized %s\n",yytext);} %% int main(int argc, char *argv[]){ ++argv, --argc; if (argc>0) yyin = fopen(argv[0],"r"); else yyin = stdin; yylex(); } yywrap(){return 0;} by Neng-Fa Zhou

  12. java.util.regex import java.util.regex.*; class Number { public static void main(String[] args){ String regExNum = "\\d+(\\.\\d+((e|E)(\\+|-)?\\d+)?)?"; if (Pattern.matches(regExNum,args[0])) System.out.println("valid"); else System.out.println("invalid"); } } by Neng-Fa Zhou

  13. String Pattern Matching in Perl print "Input a string :"; $_ = <STDIN>; chomp($_); if (/^[0-9]+(\.[0-9]+((e|E)(\+|-)?[0-9]+)?)?$/){ print "valid\n"; } else { print "invalid\n"; } by Neng-Fa Zhou

  14. Finite Automata • Nondeterministic finite automaton (NFA) NFA = (S,T,s0,F) • S: a set of states • T: a transition mapping • s0: the start state • F: final states or accepting states by Neng-Fa Zhou

  15. Example by Neng-Fa Zhou

  16. Deterministic Finite Automata (DFA) T: a transition function There is only one arc going out from each node on each symbol. by Neng-Fa Zhou

  17. Simulating a DFA s = s0; c = nextchar; while (c != eof) { s = move(s,c); c = nextchar; } if (s is in F) return "yes"; else return "no"; by Neng-Fa Zhou

  18. From RE to NFA • e • a in S • s|t by Neng-Fa Zhou

  19. From RE to NFA (cont.) • st • s* by Neng-Fa Zhou

  20. Example (a|b)*a by Neng-Fa Zhou

  21. Building Lexical Analyzer RE NFA DFA Algorithm 3.23 (Thompson's construction) Algorithm 3.32 (Subset construction) Emulator by Neng-Fa Zhou

  22. Conversion of an NFA into a DFA • Intuition • move(s,a) is a function in a DFA • move(s,a) is a mapping in a NFA NFA DFA A state reachable from s0 in the DFA on an input string corresponds to a set of states in NFA that are reachable on the same string. by Neng-Fa Zhou

  23. Computation of e-Closure e-Closure(T): Set of NFA states reachable from some NFA state s in T by e-transition alone. by Neng-Fa Zhou

  24. From an NFA to a DFA(The subset construction) by Neng-Fa Zhou

  25. Example NFA DFA by Neng-Fa Zhou

  26. Algorithm 3.39 P = {F, S-F}; do begin P0=P; for each group G in P do begin partition G into subgroups such that two states s and t of G are in the same subgroup iff for all input symbols a, s and t have transitions on a to states in the same group; replace G in P by the set of all subgroups formed; end if (P == P0) return;; end; by Neng-Fa Zhou

  27. Example a b AC B AC B B D D B E E B AC by Neng-Fa Zhou

  28. Construct a DFA Directly from a Regular Expression by Neng-Fa Zhou

  29. Implementation Issues • Input buffering • Read in characters one by one • Unable to look ahead • Inefficient • Read in a whole string and store it in memory • Requires a big buffer • Buffer pairs by Neng-Fa Zhou

  30. Buffer Pairs by Neng-Fa Zhou

  31. Use Sentinels by Neng-Fa Zhou

  32. Lexical Analyzer by Neng-Fa Zhou

  33. Lex • A tool for automatically generating lexical analyzers by Neng-Fa Zhou

  34. Lex Specifications declarations %% translation rules %% auxiliary procedures p1 {action1} p2 {action2} ... pn {actionn} by Neng-Fa Zhou

  35. Lex Regular Expressions by Neng-Fa Zhou

  36. yylex() yylex(){ switch (pattern_match()){ case 1: {action1} case 2: {action2} ... case n: {actionn} } } by Neng-Fa Zhou

  37. Example DIGIT [0-9] ID [a-z][a-z0-9]* %% {DIGIT}+ {printf("An integer:%s(%d)\n",yytext,atoi(yytext));} {DIGIT}+"."{DIGIT}* {printf("A float: %s (%g)\n",yytext,atof(yytext));} if|then|begin|end|procedure|function {printf("A keyword: %s\n",yytext);} {ID} {printf("An identifier %s\n",yytext);} "+"|"-"|"*"|"/" {printf("An operator %s\n",yytext);} "{"[^}\n]*"}" {/* eat up one-line comments */} [ \t\n]+ {/* eat up white space */} . {printf("Unrecognized character: %s\n", yytext);} %% int main(int argc, char *argv[]){ ++argv, --argc; if (argc>0) yyin = fopen(argv[0],"r"); else yyin = stdin; yylex(); } by Neng-Fa Zhou

More Related