Natural Language Processing

Natural Language Processing Lecture 8—9/24/2013 Jim Martin

Today • Review • Word classes • Part of speech tagging • HMMs • Basic HMM model • Decoding • Viterbi Speech and Language Processing - Jurafsky and Martin

Word Classes:Parts of Speech • 8 (ish) traditional parts of speech • Noun, verb, adjective, preposition, adverb, article, interjection, pronoun, conjunction, etc • Called: parts-of-speech, lexical categories, word classes, morphological classes, lexical tags... • Lots of debate within linguistics about the number, nature, and universality of these • We’ll completely ignore this debate. Speech and Language Processing - Jurafsky and Martin

POS Tagging • The process of assigning a part-of-speech or lexical class marker to each word in a collection. WORD tag the DET koala N put V the DET keys N on P the DET table N Speech and Language Processing - Jurafsky and Martin

Penn TreeBank POS Tagset Speech and Language Processing - Jurafsky and Martin

POS Tagging • Words often have more than one part of speech: back • The back door = JJ • On my back = NN • Win the voters back = RB • Promised to back the bill = VB • The POS tagging problem is to determine the POS tag for a particular instance of a word in context • Usually a sentence Speech and Language Processing - Jurafsky and Martin

POS Tagging • Note this is distinct from the task of identifying which sense of a word is being used given a particular part of speech. That’s called word sense disambiguation. We’ll get to that later. • “… backedthe car into a pole” • “… backedthe wrong candidate” Speech and Language Processing - Jurafsky and Martin

How Hard is POS Tagging? Measuring Ambiguity Speech and Language Processing - Jurafsky and Martin

Two Methods for POS Tagging • Rule-based tagging • (ENGTWOL; Section 5.4) • Stochastic • Probabilistic sequence models • HMM (Hidden Markov Model) tagging • MEMMs (Maximum Entropy Markov Models) Speech and Language Processing - Jurafsky and Martin

POS Tagging as Sequence Classification • We are given a sentence (an “observation” or “sequence of observations”) • Secretariat is expected to race tomorrow • What is the best sequence of tags that corresponds to this sequence of observations? • Probabilistic view • Consider all possible sequences of tags • Out of this universe of sequences, choose the tag sequence which is most probable given the observation sequence of n words w1…wn. Speech and Language Processing - Jurafsky and Martin

Getting to HMMs • We want, out of all sequences of n tags t1…tn the single tag sequence such that P(t1…tn|w1…wn) is highest. • Hat ^ means “our estimate of the best one” • Argmaxx f(x) means “the x such that f(x) is maximized” Speech and Language Processing - Jurafsky and Martin

Getting to HMMs • This equation should give us the best tag sequence • But how to make it operational? How to compute this value? • Intuition of Bayesian inference: • Use Bayes rule to transform this equation into a set of probabilities that are easier to compute (and give the right answer) Speech and Language Processing - Jurafsky and Martin

Using Bayes Rule Know this. Speech and Language Processing - Jurafsky and Martin

Likelihood and Prior Speech and Language Processing - Jurafsky and Martin

Two Kinds of Probabilities • Tag transition probabilities p(ti|ti-1) • Determiners likely to precede adjs and nouns • That/DT flight/NN • The/DT yellow/JJ hat/NN • So we expect P(NN|DT) and P(JJ|DT) to be high • Compute P(NN|DT) by counting in a labeled corpus: Speech and Language Processing - Jurafsky and Martin

Two Kinds of Probabilities • Word likelihood probabilities p(wi|ti) • VBZ (3sg Pres Verb) likely to be “is” • Compute P(is|VBZ) by counting in a labeled corpus: Speech and Language Processing - Jurafsky and Martin

Example: The Verb “race” • Secretariat/NNP is/VBZ expected/VBN to/TO race/VB tomorrow/NR • People/NNS continue/VB to/TO inquire/VB the/DT reason/NN for/IN the/DTrace/NN for/IN outer/JJ space/NN • How do we pick the right tag? Speech and Language Processing - Jurafsky and Martin

Disambiguating “race” Speech and Language Processing - Jurafsky and Martin

Break • Quiz readings • Chapters 1 to 6 • Chapter 2 • Chapter 3 • Skip 3.4.1, 3.10, 3.12 • Chapter 4 • Skip 4.7, 4.8-4.11 • Chapter 5 • Skip 5.5.4, 5.6, 5.8-5.10 • Chapter 6 • Skip 6.6-6.9 Speech and Language Processing - Jurafsky and Martin

Hidden Markov Models • What we’ve just described is called a Hidden Markov Model (HMM) • This is a kind of generative model. • There is a hidden underlying generator of observable events • The hidden generator can be modeled as a network of states and transitions • We want to infer the underlying state sequence given the observed event sequence Speech and Language Processing - Jurafsky and Martin

Hidden Markov Models • States Q = q1, q2…qN; • Observations O= o1, o2…oN; • Each observation is a symbol from a vocabulary V = {v1,v2,…vV} • Transition probabilities • Transition probability matrix A = {aij} • Observation likelihoods • Output probability matrix B={bi(k)} • Special initial probability vector  Speech and Language Processing - Jurafsky and Martin

HMMs for Ice Cream • You are a climatologist in the year 2799 studying global warming • You can’t find any records of the weather in Baltimore for summer of 2007 • But you find Jason Eisner’s diary which lists how many ice-creams Jason ate every day that summer • Your job: figure out how hot it was each day Speech and Language Processing - Jurafsky and Martin

Eisner Task • Given • Ice Cream Observation Sequence: 1,2,3,2,2,2,3… • Produce: • Hidden Weather Sequence: H,C,H,H,H,C, C… Speech and Language Processing - Jurafsky and Martin

HMM for Ice Cream Speech and Language Processing - Jurafsky and Martin

Ice Cream HMM HHH HHC HCH HCC CCC CCH CHC CHH ArgmaxP(sequence | 1 3 1) • Let’s just do 131 as the sequence • How many underlying state (hot/cold) sequences are there? • How do you pick the right one? Speech and Language Processing - Jurafsky and Martin

Ice Cream HMM Cold as the initial state P(Cold|Start) .2 .5 .4 .4 .3 .5 Observing a 1 on a cold day P(1 | Cold) Hot as the next state P(Hot | Cold) Observing a 3 on a hot day P(3 | Hot) Cold as the next state P(Cold|Hot) .0024 Observing a 1 on a cold day P(1 | Cold) Let’s just do 1 sequence: CHC Speech and Language Processing - Jurafsky and Martin

POS Transition Probabilities Speech and Language Processing - Jurafsky and Martin

Observation Likelihoods Speech and Language Processing - Jurafsky and Martin

Question 3020 If there are 30 or so tags in the Penn set And the average sentence is around 20 words... How many tag sequences do we have to enumerate to argmax over in the worst case scenario? Speech and Language Processing - Jurafsky and Martin

Decoding • Ok, now we have a complete model that can give us what we need. Recall that we need to get • We could just enumerate all paths given the input and use the model to assign probabilities to each. • Not a good idea. • Luckily dynamic programming (last seen in Ch. 3 with minimum edit distance) helps us here Speech and Language Processing - Jurafsky and Martin

Intuition • Consider a state sequence (tag sequence) that ends at state j with a particular tag T. • The probability of that tag sequence can be broken into two parts • The probability of the BEST tag sequence up through j-1 • Multiplied by the transition probability from the tag at the end of the j-1 sequence to T. • And the observation probability of the word given tag T. Speech and Language Processing - Jurafsky and Martin

The Viterbi Algorithm Speech and Language Processing - Jurafsky and Martin

Viterbi Example Speech and Language Processing - Jurafsky and Martin

Viterbi Summary • Create an array • With columns corresponding to inputs • Rows corresponding to possible states • Sweep through the array in one pass filling the columns left to right using our transition probs and observations probs • Dynamic programming key is that we need only store the MAX prob path to each cell, (not all paths). Speech and Language Processing - Jurafsky and Martin

Evaluation • So once you have you POS tagger running how do you evaluate it? • Overall error rate with respect to a gold-standard test set. • Error rates on particular tags • Error rates on particular words • Tag confusions... Speech and Language Processing - Jurafsky and Martin

Error Analysis • Look at a confusion matrix • See what errors are causing problems • Noun (NN) vs ProperNoun (NNP) vs Adj (JJ) • Preterite (VBD) vs Participle (VBN) vs Adjective (JJ) Speech and Language Processing - Jurafsky and Martin

Evaluation • The result is compared with a manually coded “Gold Standard” • Typically accuracy reaches 96-97% • This may be compared with result for a baseline tagger (one that uses no context). • Important: 100% is impossible even for human annotators. Speech and Language Processing - Jurafsky and Martin

Summary • Parts of speech • Tagsets • Part of speech tagging • HMM Tagging • Markov Chains • Hidden Markov Models • Viterbi decoding Speech and Language Processing - Jurafsky and Martin

Next Time • Three tasks for HMMs • Decoding • Viterbi algorithm • Assigning probabilities to inputs • Forward algorithm • Finding parameters for a model • EM Speech and Language Processing - Jurafsky and Martin

Natural Language Processing