Lexical Analysis

Lexical Analysis

Review • What is a compiler? • What are the Phases of a Compiler? Lexical Analysis

The Goals of Chapter 3 • See how to construct a lexical analyzer • Lexeme description • Code to recognize lexemes • Lexical Analyzer Generator • Lex and Flex • Regular Expressions • To Nondeterministic finite autamota • To Finite Automata • Then to Code Lexical Analysis

What is a Lexical Analyzer supposed to do? • Read the input characters of the source program, • Group them into lexemes, • Check with the symbol table regarding this lexeme • Produce as output a sequence of tokens • This stream is sent to the parser Lexical Analysis

What is a Lexical Analyzer supposed to do? • It may also • strip out comments and whitespace. • Keep track of line numbers in the source file (for error messages) Lexical Analysis

Terms • Token • A pair (token name, [attribute value]) • Ex: (integer, 32) • Pattern • A description of the form the lexemes will take • Ex: regular expressions • Lexeme • An instance matched by the pattern. Lexical Analysis

What are the tokens? • Code: • printf(“Total= %d\n”, score); • Tokens: • (id, printf) • openParen • literal “Total= %d\n” • comma • (id, score) • closeParen • semicolon Lexical Analysis

Token Attributes • When more than one lexeme can match a pattern, the lexical analyzer must provide the next phase additional information. • Number – needs a value • Id – needs the identifier name, or probably a symbol table pointer Lexical Analysis

Specification of Tokens • Regular expressions are an important notation for specifying lexeme patterns. • We are going to • first study the formal notation for a regular expression • Then we will show how to build a lexical analyzer that uses regular expressions. • Then we will see how they are used in a lexical analyzer. Lexical Analysis

Buffering • In principle, the analyzer goes through the source string a character at a time; • In practice, it must be able to access substrings of the source. • Hence the source is normally read into a buffer • The scanner needs two subscripts to note places in the buffer • lexeme start & current position Lexical Analysis

Strings and Alphabets • Def: Alphabet – a finite set of symbols • Typically letters, digits, and punctuation • Example: • {0,1} is a binary alphabet. Lexical Analysis

Strings and Alphabets • Def: String – (over an alphabet) is a finite sequence of symbols drawn from that alphabet. • Sometimes called a “word” • The empty string e Lexical Analysis

Strings and Alphabets • Def: Language – any countable set of strings over some fixed alphabet. • Notes: • Abstract languages like , and {} fit this definition, but so also do C programs • This definition also does not require any meaning – that will come later. Lexical Analysis

Finite State Automata • The compiler writer defines tokens in the language by means of regular expressions. • Informally a regular expression is a compact notation that indicates what characters may go together into lexemes belonging to that particular token and how they go together. • We will see regular expressions later Lexical Analysis

The lexical analyzer is best implemented as a finite state machine or a finite state automaton. • Informally a Finite-State Automaton is a system that has a finite set of states with rules for making transitions between states. • The goal now is to explain these two things in detail and bridge the gap from the first to the second. Lexical Analysis

State Diagrams and State Tables • Def:State Diagram -- is a directed graph where the vertices in the graph represent states, and the edges indicate transitions between the states. • Let’s consider a vending machine that sells candy bars for 25 cents, and it takes nickels, dimes, and quarters. Lexical Analysis

Figure 2.1 -- pg.. 21 Lexical Analysis

Def:State Table -- is a table with states down the left side, inputs across the top, and row/column values indicate the current state/input and what state to go to. Lexical Analysis

Formal Definition • Def: FSA -- A FSA, M consists of • a finite set of input symbols S (the input alphabet) • a finite set of states Q • A starting state q0 (which is an element of Q) • A set of accepting states F (a subset of Q) (these are sometimes called final states) • A state-transition function N: (Q x S) -> Q • M = (S, Q, q0, F, N) Lexical Analysis

Example 1: • Given Machine Construct Table: Lexical Analysis

Example 1 (cont.): • State 1 is the Starting State. This is shown by the arrow in the Machine, and the fact that it is the first state in the table. • States 3 and 4 are the accepting states. This is shown by the double circles in the machine, and the fact that they are underlined in the table. Lexical Analysis

Example 2: • Given Table, Construct Machine: Lexical Analysis

Example 2 (cont.): • This machine shows that it is entirely possible to have unreachable states in an automaton. • These states can be removed without affecting the automaton. Lexical Analysis

Acceptance • We use FSA's for recognizing tokens • A character string is recognized (or accepted) by FSA M if, when the last character has been read, M is in one of the accepting states. • If we pass through an accepting state, but end in a non-accepting state, the string is NOT accepted. Lexical Analysis

Def:language -- A language is any set of strings. • Def:a language over an alphabet S is any set of strings made up only of the characters from S • Def:L(M) -- the language accepted by M is the set of all strings over S that are accepted by M Lexical Analysis

if we have an FSA for every token, then the language accepted by that FSA is the set of all lexemes embraced by that token. • Def:equivalent -- M1 == M2 iff L(M1) = L(M2). Lexical Analysis

A FSA can be easily programmed if the state table is stored in memory as a two-dimensional array. table : array[1..nstates,1..ninputs] of byte; • Given an input string w, the code would look something like this: state := 1; for i:=1 to length(w) do begin col:= char_to_col(w[i]); state:= table[state, col] end; Lexical Analysis

Nondeterministic Finite-State Automata • So far, the behavior of our FSAs has always been predictable. But there is another type of FSA in which the state transitions are not predictable. • In these machines, the state transition function is of the form: N: Q x (S U {e}) -> P(Q) • Note: some authors use a Greek lambda, l or L Lexical Analysis

This means two things: • There can be transitions without input. (That is why the e is in the domain) • Input can transition to a number of states. (That is the significance of the power set in the codomain) Lexical Analysis

Since this makes the behavior unpredictable, we call it a nondeterministic FSA • So now we have DFAs and NFAs (or NDFAs) • a string is accepted if there is at least 1 path from the start state to an accepting state. Lexical Analysis

Example: Given Machine Trace the input = baabab Lexical Analysis

e-Transitions • a spontaneous transition without input. • Example: Trace input aaba Lexical Analysis

Equivalence • For every non-deterministic machine M we can construct an equivalent deterministic machine M' • Therefore, why study N-FSA? • 1.Theory. • 2.Tokens -> Reg.Expr. -> N-FSA -> D-FSA Lexical Analysis

The Subset Construction • Constructing an equivalent DFA from a given NFA hinges on the fact that transitions between state sets are deterministic even if the transitions between states are not. • Acceptance states of the subset machine are those subsets that contain at least 1 accepting state. Lexical Analysis

Generic brute force construction is impractical. • As the number of states in M increases, the number of states in M' increases drastically (n vs. 2n). If we have a NFA with 20 states |P(Q)| is something over a million. • This also leads to the creation of many unreachable states. (which can be omitted) • The trick is to only create subset states as you need them. Lexical Analysis

Example: • Given: NFA • Build DFA out of NFA Lexical Analysis

Why do we care? • Lexical Analysis and Syntactic Analysis are typically run off of tables. • These tables are large and laborious to build. • Therefore, we use a program to build the tables. Lexical Analysis

But there are two major problems: • How do we represent a token for the table generating program? • How does the program convert this into the corresponding FSA? • Tokens are described using regular expressions. Lexical Analysis

Regular Expressions • Informally a regular expression of an alphabet S is a combination of characters from S and certain operators indicating concatenation, selection, or repetition. • b* -- 0 or more b's (Kleene Star) • b+ -- 1 or more b's • | -- a|b -- choice Lexical Analysis

Regular Expressions • Def:Regular Expression: • any character in S is an RE • e is an RE • if R and S are RE's so are RS, R|S, R*, R+, S*, S+. • Only expressions formed by these rules are regular. Lexical Analysis

Regular Expressions • REs can be used to describe only a limited variety of languages, but they are powerful enough to be used to define tokens. • One limitation -- many languages put length limitations on their tokens, RE's have no means of enforcing such limitations. Lexical Analysis

Extensions to REs • One or more Instances + • Zero or One Instance ? • Character Classes • Digit -> [0-9] • Digits -> Digit+ • Number -> Digits (. Digits)? (E [+-]? Digits)? Lexical Analysis

Lexical Analysis