Prototype-Driven Learning for Sequence Models

1. Prototype-Driven Learning for Sequence Models Aria Haghighi and Dan Klein University of California Berkeley Slides prepared by Andrew Carlson for the Semi-supervised NL Learning Reading Group

2. Motivation: Learn models with least effort • Supervised learning requires many labeled examples • Unsupervised learning requires a carefully designed model – does not necessarily minimize total effort • Prototype-driven learning can require less total effort

3. Prototype-driven learning • Specify prototypical examples for each target label • Example: for POS tagging, list the target tags and a few examples of each tag

4. English POS Prototypes

5. Arguments for prototype-driven learning • Minimum one would have to provide to a human annotator • Pedagogical use • Natural language exhibits proform and prototype effects

6. General approach • Link any given word to similar prototypes using distributional similarity

7. General approach • Link any given word to similar prototypes using distributional similarity • Encode these prototype links as features in a log-linear generative model, trained to fit unlabeled data

8. General approach • Link any given word to similar prototypes using distributional similarity • Encode these prototype links as features in a log-linear generative model, trained to fit unlabeled data Example: reported may be linked to said, which is a prototype for the POS tag VBD.

9. English POS task (Penn treebank)

10. Chinese POS task (Penn treebank)

11. Classified Ad Segmentation (Grenager et al. 2005) Task: Segment classified advertisements into topical sections

12. Classified Ad Segmentation (Grenager et al. 2005) Task: Segment classified advertisements into topical sections Typical of unsupervised learning on a new domain: Grenager et al. altered their HMM to prefer diagonal transitions, then modified the transition structure to explicitly model boundary tokens.

13. Approach • For each document x, we would like to predict a sequence of labels y • Build a generative model and choose parameters θ to maximize the log-likelihood of the observed data D:

14. Markov Random Fields • Use MRF model family • Undirected equivalent to HMMs

15. Markov Random Fields • Use MRF model family • Undirected equivalent to HMMs Edge/transition clique potential Joint probability Node/emission clique potential Normalizer

16. Markov Random Fields • Use MRF model family • Undirected equivalent to HMMs Edge/transition clique potential Joint probability Node/emission clique potential Normalizer is a potential over a clique with form

17. English POS trigram tagger

18. Using distributional similarity and prototypes • Add a node feature PROTO = z for all prototypes z similar to the word at that node • For POS tagging, similarity is based on positional context vectors • For the classified ad task, position is ignored and a wider window is used

19. Prototype list for classified ads

20. Parameter estimation • See the paper • Gradient-based method (L-BFGS), forward-backward, Viterbi

21. English POS tagging • Used the WSJ portion of the Penn treebank • Used two sizes– 48K tokens, 193K tokens

22. Baseline features • BASE features • Node features: exact word, character suffixes, init-caps, contains-hyphen, contains-digit • Edge features: tag trigrams

23. Building the prototype list • Automatically extracted the prototype list • For each label, selected the top three occurring words that were not given another label more often

24. Building the prototype list • Automatically extracted the prototype list • For each label, selected the top three occurring words that were not given another label more often Yes. This does use labeled data! The authors did it this way to give repeatable results, and to avoid excessive tuning.

25. English POS Prototypes

26. Use the prototypes • Restricting the prototype words to have their respective labels improved performance, but did not help similar non-prototype words. • Solution: add PROTO features to similar words

27. English POS trigram tagger

28. English POS results

29. English POS transition mass true estimated

30. Chinese POS results • Reduces the error rate from BASE by 35%, but not as good as English results • Reasons: task is harder, and had less data for distributional similarity

31. Classified ad segmentation • For distributional similarity, used context vectors but ignored distance and direction • Added special BOUNDARY state to handle tokens that indicate transitions • Special model tweaking– deviance from “least effort” motivation

32. Classified ad segmentationresults • BASE scored 46.4% accuracy • BASE+PROTO+SIM scores 71.5% accuracy • BASE+PROTO+SIM+BOUND scores 72.4% • Grenager et al. reported supervised accuracy of 74.4%

33. Common classified ad confusions

34. Classified ad transition mass estimate from all features labeled estimate from BASE features

35. Conclusions • Prototype-driven learning provides a compact and declarative way to specify a target labeling scheme. • Distributional similarity features seem to work well in linking words to prototypes. • Bridges gap between unsupervised sequence-free distributional clustering approaches and supervised sequence model learning.