Reza Bosaghzadeh & Nathan Schneider LS2 ~ 1 December 2008

1. Unsupervised Approaches to Sequence Tagging, Morphology Induction, and Lexical Resource Acquisition Reza Bosaghzadeh & Nathan Schneider LS2 ~ 1 December 2008

2. Unsupervised Methods • Sequence Labeling (Part-of-Speech Tagging) • Morphology Induction • Lexical Resource Acquisition . pronoun verb preposition det noun adverb She ran to the station quickly un-supervise-dlearn-ing

3. Part-of-speech (POS) tagging • Prototype-driven model Cluster based on a few examples (Haghighi and Klein 2006) • Contrastive Estimation Discount positive examples at cost of implicit negative example (Smith and Eisner 2005)

4. Target Label Prototypes Prototype-driven taggingHaghighi & Klein (2006) Annotated Data Unlabeled Data Prototype List + slide courtesy Haghighi & Klein

5. Size Restrict Terms Location Features Prototype-driven taggingHaghighi& Klein (2006) Information Extraction: Classified Ads Newly remodeled2 Bdrms/1 Bath, spacious upper unit, located in Hilltop Mall area. Walking distance to shopping, public transportation, schools and park.Paid water and garbage.No dogs allowed. Newly remodeled 2 Bdrms/1 Bath, spacious upper unit, located in Hilltop Mall area. Walking distance to shopping, public transportation, schools and park. Paid water and garbage. No dogs allowed. Prototype List slide courtesy Haghighi & Klein

6. PUNC NN VBN CC JJ CD IN DET NNS IN NNP RB Prototype-driven taggingHaghighi & Klein (2006) English POS Newly remodeled 2Bdrms/1Bath,spacious upper unit,locatedin Hilltop Mallarea.Walkingdistance toshopping, public transportation,schoolsandpark.Paid water andgarbage. Nodogs allowed. Newly remodeled 2 Bdrms/1 Bath, spacious upper unit, located in Hilltop Mall area. Walking distance to shopping, public transportation, schools and park. Paid water and garbage. No dogs allowed. Prototype List slide courtesy Haghighi & Klein

7. Prototype-driven taggingHaghighi & Klein (2006) • 3 prototypes per tag • Automatically extracted by frequency (Haghighi and Klein 2006)

8. Prototype-driven taggingHaghighi & Klein (2006) • Trigram tagger, same features as (Smith & Eisner 2005) • Word type, suffixes up to length 3, contains-hyphen, contains-digit, initial capitalization • Tie each word to its most similar prototype, using context-based similarity technique (Schütze 1993) • SVD dimensionality reduction • Cosine similarity between context vectors (Haghighi and Klein 2006)

9. Prototype-driven taggingHaghighi & Klein (2006) Pros • Fairly easy to choose a few tag prototypes • Number of prototypes is flexible • Doesn’t require tagging dictionary Cons • Still need a tag set • Chosen prototypes may not work so well if infrequent or highly ambiguous

10. Contrastive EstimationSmith & Eisner (2005) • Already discussed in class • Key idea: exploits implicit negative evidence • Mutating training examples often gives ungrammatical (negative) sentences • During training, shift probability mass from generated negative examples to given positive examples

11. Unsupervised POS taggingThe State of the Art Best supervised result (CRF): 99.5%!

12. Unsupervised Methods • Sequence Labeling (Part-of-Speech Tagging) • Morphology Induction • Lexical Resource Acquisition . pronoun verb preposition det noun adverb She ran to the station quickly un-supervise-dlearn-ing

13. Unsupervised Approaches to Morphology • Morphology refers to the internal structure of words • A morpheme is a minimal meaningful linguistic unit • Morpheme segmentation is the process of dividing words into their component morphemes unsupervised learning .

14. Unsupervised Approaches to Morphology • Morphology refers to the internal structure of words • A morpheme is a minimal meaningful linguistic unit • Morpheme segmentation is the process of dividing words into their component morphemes un-supervise-dlearn-ing • Word segmentation is the process of finding word boundaries in a stream of speech or text unsupervisedlearningofnaturallanguage

15. Unsupervised Approaches to Morphology • Morphology refers to the internal structure of words • A morpheme is a minimal meaningful linguistic unit • Morpheme segmentation is the process of dividing words into their component morphemes un-supervise-dlearn-ing • Word segmentation is the process of finding word boundaries in a stream of speech or text unsupervised learning of natural language

16. ParaMor: Morphological paradigmsMonson et al. (2007, 2008) • Learns inflectional paradigms from raw text • Requires only the vocabulary of a corpus • Looks at word counts of substrings, and proposes (stem, suffix) pairings based on type frequency • 3-stage algorithm • Stage 1: Candidate paradigms based on frequencies • Stages 2-3: Refinement of paradigm set via merging and filtering • Paradigms can be used for morpheme segmentation or stemming

17. ParaMor: Monson et al. (2007, 2008) • A sampling of Spanish verb conjugations

18. ParaMor: Morphological paradigmsMonson et al. (2007, 2008) • A proposed paradigm with stems {habl, bail} and suffixes {-ar, -o, -amos, -an}

19. ParaMor: Morphological paradigmsMonson et al. (2007, 2008) • Same paradigm from previous slide, but with stems {habl, bail, compr}

20. ParaMor: Morphological paradigmsMonson et al. (2007, 2008) • From just this list, other paradigm analyses (which happen to be incorrect) are possible

21. ParaMor: Morphological paradigmsMonson et al. (2007, 2008) • Another possibility: stems {hab, bai}, suffixes {-lar, -lo, -lamos, -lan}

22. ParaMor: Morphological paradigmsMonson et al. (2007, 2008) • Spurious segmentations—this paradigm doesn’t generalize to comprar(or most verbs)

23. ParaMor: Morphological paradigmsMonson et al. (2007, 2008) • What if not all conjugations were in the corpus?

24. ParaMor: Morphological paradigmsMonson et al. (2007, 2008) • We have two similar paradigms that we want to merge

25. ParaMor: Morphological paradigmsMonson et al. (2007, 2008) • This amounts to smoothing, or “hallucinating” out-of-vocabulary items

26. ParaMor: Morphological paradigmsMonson et al. (2007, 2008) • Heuristic-based, deterministic algorithm can learn inflectional paradigms from raw text • Paradigms can be used straightforwardly to predict segmentations • Combining the outputs of ParaMor and Morfessor (another system) won the segmentation task at MorphoChallenge 2008 for every language: English, Arabic, Turkish, German, and Finnish • Currently, ParaMor assumes suffix-based morphology

27. Bayesian word segmentation Goldwater et al. (2006; in submission) • Word segmentation results – comparison • See Narges & Andreas’s presentation for more on this model Goldwater Unigram DP Goldwater Bigram HDP

28. Multilingual morpheme segmentation Snyder & Barzilay (2008) • Considers parallel phrases and tries to find morpheme correspondences • Stray morphemes don’t correspond across languages • Abstract morphemes cross languages: (ar,er), (o, e), (amos,ons), (an,ent), (habl, parl)

29. Morphology Papers: Inputs & Outputs

30. Unsupervised Methods • Sequence Labeling (Part-of-Speech Tagging) • Morphology Induction • Lexical Resource Acquisition . pronoun verb preposition det noun adverb She ran to the station quickly un-supervise-dlearn-ing

31. Lexical Resource Acquisition • Bilingual Lexicons from Monolingual Corpora (Haghighi et al. 2008) • Narrative Event Chains (Chambers and Jurafsky 2008) • A Statistical Verb Lexicon for Semantic Role Labeling (Grenager and Manning 2006)

32. Bilingual lexicons from monolingual corporaHaghighi et al. (2008) Source Words s Target Words t Source Text Target Text Matching m nombre estado state world slide courtesy Haghighi et al. política name mundo nation

33. Bilingual Lexicons from Monolingual CorporaHaghighi et al. (2008) Data Representation Orthographic Features Orthographic Features estado state #st #es 1.0 1.0 sta tat 1.0 1.0 te# do# 1.0 1.0 Source Text Target Text Context Features Context Features mundo world 20.0 17.0 politica politics 5.0 10.0 Used a variant of CCA (Canonical Correlation Analysis) Trained with ViterbiEM to find best matching sociedad society 6.0 10.0 slide courtesy Haghighi et al.

34. Bilingual Lexicons from Monolingual CorporaHaghighi et al. (2008) Feature Experiments • MCCA: Orthographic and context features Precision 4k EN-ES Wikipedia Articles (Haghighi et al. 2008)

35. Narrative eventsChambers & Jurafsky (2008) • Given a corpus, identifies related events that constitute a “narrative” and (when possible) predict their typical temporal ordering • E.g.: criminal prosecution narrative, with verbs: arrest, accuse, plead, testify, acquit/convict • Key insight: related events tend to share a participant in a document • The common participant may fill different syntactic/semantic roles with respect to verbs: arrest.object, accuse.object, plead.subject

36. Narrative eventsChambers & Jurafsky (2008) • A temporal classifier can reconstruct pairwise canonical event orderings, producing a directed graph for each narrative

37. Statistical verb lexiconGrenager& Manning (2006) • From dependency parses, a generative model predicts semantic roles corresponding to each verb’s arguments, as well as their syntactic realizations • PropBank-style: arg0, arg1, etc. per verb (do not necessarily correspond across verbs) • Learned syntactic patterns of the form: (subj=give.arg0, verb=give, np#1=give.arg2, np#2=give.arg1) or (subj=give.arg0, verb=give, np#2=give.arg1, pp_to=give.arg2) • Used for semantic role labeling

38. “Semanticity”: Our proposed scale of semantic richness • text < POS < syntax/morphology/alignments < coreference/semantic roles/temporal ordering < translations/narrative event sequences • We score each model’s inputs and outputs on this scale, and call the input-to-output increase “semantic gain” • Haghighi et al.’s bilingual lexicon induction wins in this respect, going from raw text to lexical translations

39. Robustness to language variation • About half of the papers we examined had English-only evaluations • We considered which techniques were most adaptable to other (esp. resource-poor) languages. Two main factors: • Reliance on existing tools/resourcesfor preprocessing (parsers, coreference resolvers, …) • Any linguistic specificityin the model (e.g. suffix-based morphology)

40. Summary We examined three areas of unsupervised NLP: • Sequence tagging: How can we predict POS (or topic) tags for words in sequence? • Morphology: How are words put together from morphemes (and how can we break them apart)? • Lexical resources: How can we identify lexical translations, semantic roles and argument frames, or narrative event sequences from text? In eight recent papers we found a variety of approaches, including heuristic algorithms, Bayesian methods, and EM-style techniques.

41. Target Label Prototypes Questions? Thanks to Noah and Kevin for their feedback on the paper; Andreas and Narges for their collaboration on the presentations; and all of you for giving us your attention! subj=give.arg0verb=givenp#1=give.arg2np#2=give.arg1 • un-supervise-dlearn-ing