1 / 36

Learning PCFGs: Estimating Parameters, Learning Grammar Rules

Learning PCFGs: Estimating Parameters, Learning Grammar Rules. Many slides are taken or adapted from slides by Dan Klein. Treebanks. An example tree from the Penn Treebank. The Penn Treebank. 1 million tokens In 50,000 sentences, each labeled with A POS tag for each token

avital
Download Presentation

Learning PCFGs: Estimating Parameters, Learning Grammar Rules

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Learning PCFGs: Estimating Parameters,Learning Grammar Rules Many slides are taken or adapted from slides by Dan Klein

  2. Treebanks An example tree from the Penn Treebank

  3. The Penn Treebank • 1 million tokens • In 50,000 sentences, each labeled with • A POS tag for each token • Labeled constituents • “Extra” information • Phrase annotations like “TMP” • “empty” constituents for wh-movement traces, empty subjects for raising constructions

  4. Supervised PCFG Learning • Preprocess the treebank • Remove all “extra” information (empties, extra annotations) • Convert to Chomsky Normal Form • Possibly prune some punctuation, lower-case all words, compute word “shapes”, and other processing to combat sparsity. • Count the occurrence of each nonterminal c(N) and each observed production rule c(N->NL NR) and c(N->w) • Set the probability for each rule to the MLE: P(N->NL NR) = c(N->NL NR) / c(N) P(N->w) = c(N->w) / c(N) Easy, peasy, lemon-squeezy.

  5. Complications • Smoothing • Especially for lexicalized grammars, many test productions will never be observed during training • We don’t necessarily want to assign these productions zero probability • Instead, define backoff distributions, e.g.: Pfinal(VPtransmogrified -> Vtransmogrified PPinto) = αP(VPtransmogrified -> Vtransmogrified PPinto) + (1-α) P(VP -> V PPinto)

  6. Problems with Supervised PCFG Learning • Coming up with labeled data is hard! • Time-consuming • Expensive • Hard to adapt to new domains, tasks, languages • Corpus availability drives research (instead of tasks driving the research) • Penn Treebank took many person-years to manually annotate it.

  7. Unsupervised Learning of PCFGS: Feasible?

  8. Unsupervised Learning • Systems take raw data and automatically detect data • Why? • More data is available • Kids learn (some aspects of) language with no supervision • Insights into machine learning and clustering

  9. Grammar Induction and Learnability • Some have argued that learning syntax from positive data alone is impossible • Gold, 1967: non-identifiability in the limit • Chomsky, 1980: poverty of the stimulus • Surprising result: it’s possible to get entirely unsupervised parsing to work (reasonably) well.

  10. Learnability • Learnability: formal conditions under which a class of languages can be learned • Setup: • Class of languages Λ • Algorithm H (the learner) • H sees a sequence X of strings x1 … xn • H maps sequences X to languages L in Λ • Question is: for what classes Λ do learners H exist?

  11. Learnability [Gold, 1967] • Criterion: Identification in the limit • A presentation of L is an infinite sequence of x’s from L in which each x occurs at least once • A learner H identifies L in the limit if, for any presentation of L, from some point n onwards, H always outputs L • A class Λ is identifiable in the limit if there is some single H which correctly identifies in the limit every L in Λ. • Example: L = {{a},{a,b}} is identifiable in the limit. • Theorem (Gold, 67): Any Λwhich contains all finite languages and at least one infinite language (ie is superfinite) is unlearnable in this sense.

  12. Learnability [Gold, 1967] • Proof sketch • Assume Λ is superfinite, H identifies Λ in the limit • There exists a chain L1⊂ L2⊂ … L∞ • Construct the following misleading sequence • Present strings from L1 until H outputs L1 • Present strings from L2 until H outputs L2 • … • This is a presentation of L∞ but H never outputs L∞

  13. Learnability [Horning, 1969] • Problem, IIL requires that H succeeds on all examples, even the weird ones • Another criterion: measure one identification • Assume a distribution PL(x) for each L • Assume PL(x) puts non-zero probability on all and only the x in L • Assume an infinite presentation of x drawn i.i.d. from PL(x) • H measure-one identifies L if the probability of [drawing a sequence X from which H can identify L] is 1. • Theorem (Horning, 69): PCFGs can be identified in this sense. • Note: there can be misleading sequences, but they have to be (infinitely) unlikely

  14. Learnability [Horning, 1969] • Proof sketch • Assume Λ is a recursively enumerable set of recursive languages (e.g., the set of all PCFGs) • Assume an ordering on all strings x1 < x2 < … • Define: two sequences A and B agree through n iff for all x<xn, x is in A  x is in B. • Define the error set E(L,n,m): • All sequences such that the first m elements do not agree with L through n • These are the sequences which contain early strings outside of L (can’t happen), or which fail to contain all of the early strings in L (happens less as m increases) • Claim: P(E(L,n,m)) goes to 0 as m goes to ∞ • Let dL(n) be the smallest m such that P(E) < 2–n • Let d(n) be the largest dL(n) in first n languages • Learner: after d(n), pick first L that agrees with evidence through n • This can only fail for sequences X if X keeps showing up in E(L, n, d(n)), which happens infinitely often with probability zero.

  15. Learnability • Gold’s results say little about real learners (the requirements are too strong) • Horning’s algorithm is completely impractical • It needs astronomical amounts of data • Even measure-one identification doesn’t say anything about tree structures • It only talks about learning grammatical sets • Strong generative vs. weak generative capacity

  16. Unsupervised POS Tagging • Some (discouraging) experiments [Merialdo 94] • Setup: • You know the set of allowable tags for each word (but not frequency of each tag) • Learn a supervised model on k training sentences • Learn P(w|t), P(ti|ti-1,ti-2) on these sentences • On n>k, reestimate with EM

  17. Merialdo: Results

  18. Grammar Induction Unsupervised Learning of Grammars and Parameters

  19. Right-branching Baseline • In English (but not necessarily in other languages), trees tend to be right-branching: • A simple, English-specific baseline is to choose the right chain structure for each sentence.

  20. Distributional Clustering

  21. Nearest Neighbors

  22. Learn PCFGs with EM [Lari and Young, 1990] • Setup: • Full binary grammar with n nonterminals {X1, …, Xn} (that is, at the beginning, the grammar has all possible rules) • Parse uniformly/randomly at first • Re-estimate rule expecations off of parses • Repeat • Their conclusion: it doesn’t really work

  23. EM for PCFGs: Details • Start with a “full” grammar, with all possible binary rules for our nonterminals N1 … Nk. Designate one of them as the start symbol, say N1 • Assign some starting distribution to the rules, such as • Random • Uniform • Some “smart” initialization techniques (see assigned reading) • E-step: Take an unannotated sentence S, and compute, for all nonterminals N, NL, NR, and all terminals w: E(N | S), E(N->NL NR, N is used| S), E(N->w, N is used| S) • M-step: Reset rule probabilities to the MLE: P(N->NL NR) = E(N->NL NR|S) / E(N | S) P(N->w) = E(N->w | S) / E(N | S) • Repeat 3 and 4 until rule probabilities stabilize, or “converge”

  24. Definitions This is the sum of P(T, S | G) over all possible trees T for w1m where the root is N1.

  25. E-Step • We can define the expectations we want in terms of π, α, β quantities:

  26. Inside Probabilities Base case: Induction: Nj Nl Nr wp wd wd+1 wq

  27. Outside Probabilities Base case: Induction:

  28. Problem: Model Symmetries

  29. Distributional Syntax?

  30. Problem: Identifying Constituents

  31. A nested distributional model • We’d like a model that • Ties spans to linear contexts (like distributional clustering) • Considers only proper tree structures (like PCFGs) • Has no symmetries to break (like a dependency model)

  32. Constituent Context Model (CCM)

  33. Results: Constituency

  34. Results: Dependencies

  35. Results: Combined Models

  36. Multilingual Results

More Related