Statistical Machine Translation

1. Statistical Machine Translation Translation without Understanding Colin Cherry

2. Who is this guy? • One of Dr. Lin’s PhD students • Did my Masters degree at U of A • Research Area: Machine Translation • Home town: Halifax, Nova Scotia • Please ask questions!

3. Machine Translation • Translation is easy for (bilingual) people • Process: • Read the text in English • Understand it • Write it down in French

4. Machine Translation • Translation is easy for (bilingual) people • Process: • Read the text in English • Understand it • Write it down in French • Hard for computers • The human process is invisible, intangible

5. One approach: Babelfish • A rule-based approach to machine translation • A 30-year-old feat in Software Eng. • Programming knowledge in by hand is difficult and expensive

6. Alternate Approach: Statistics • What if we had a model for P(F|E) ? • We could use Bayes rule:

7. Why Bayes rule at all? • Why not model P(E|F) directly? • P(F|E)P(E) decomposition allows us to be sloppy • P(E) worries about good English • P(F|E) worries about French that matches English • The two can be trained independently

8. Crime Scene Analogy • F is a crime scene. E is a person who may have committed the crime • P(E|F) - look at the scene - who did it? • P(E) - who had a motive? (Profiler) • P(F|E) - could they have done it? (CSI - transportation, access to weapons, alabi) • Some people might have great motives, but no means - you need both!

9. On voit Jon à la télévision Table borrowed from Jason Eisner

10. Where will we get P(F|E)? Machine Learning Magic Books in English Same books, in French P(F|E) model We call collections stored in two languages parallel corpora or parallel texts Want to update your system? Just add more text!

11. Our Inspiration: • The Canadian Parliamentary Debates! • Stored electronically in both French and English and available over the Internet

12. Problem: • How are we going to generalize from examples of translations? • I’ll spend the rest of this lecture telling you: • What makes a useful P(F|E) • How to obtain the statistics needed for P(F|E) from parallel texts

13. Strategy: Generative Story • When modeling P(X|Y): • Assume you start with Y • Decompose the creation of X from Y into some number of operations • Track statistics of individual operations • For a new example X,Y: P(X|Y) can be calculated based on the probability of the operations needed to get X from Y

14. The quick fox jumps over the lazy dog Le renard rapide saut par - dessus le chien parasseux What if…?

15. New Information • Call this new info a word alignment (A) • With A, we can make a good story The quick fox jumps over the lazy dog Le renard rapide saut par - dessus le chien parasseux

16. P(F,A|E) Story null The quick fox jumps over the lazy dog

17. P(F,A|E) Story null The quick fox jumps over the lazy dog f1 f2 f3 … f10

18. P(F,A|E) Story null The quick fox jumps over the lazy dog f1 f2 f3 … f10

19. P(F,A|E) Story null The quick fox jumps over the lazy dog Le renard rapide saut par - dessus le chien parasseux

20. P(F,A|E) Story null The quick fox jumps over the lazy dog Le renard rapide saut par - dessus le chien parasseux

21. null The quick fox jumps over the lazy dog null The quick fox jumps over the lazy dog null The quick fox jumps over the lazy dog null The quick fox jumps over the lazy dog null The quick fox jumps over the lazy dog null The quick fox jumps over the lazy dog Le renard rapide saut par - dessus le chien parasseux Le renard rapide saut par - dessus le chien parasseux Le renard rapide saut par - dessus le chien parasseux Le renard rapide saut par - dessus le chien parasseux Le renard rapide saut par - dessus le chien parasseux Le renard rapide saut par - dessus le chien parasseux Getting Pt(f|e) • We need numbers for Pt(f|e) • Example: Pt(le|the) • Count lines in a large collection of aligned text

22. Where do we get the lines? • That sure looked like a lot of monkeys… • Remember POS tagging w/ HMMs: • You didn’t need a tagged corpus to train a tagger • We’ll get alignments out of unaligned text by treating the alignment as a hidden variable • Generalization of ideas in HMM training: called EM

23. Where’s “heaven” in Vietnamese? English: In the beginning God created the heavens and the earth. Vietnamese: Ban dâu Dúc Chúa Tròi dung nên tròi dât. English: God called the expanse heaven. Vietnamese: Dúc Chúa Tròi dat tên khoang không la tròi. English: … you are this day like the stars of heaven in number. Vietnamese: … các nguoi dông nhu sao trên tròi. Example borrowed from Jason Eisner

24. Where’s “heaven” in Vietnamese? English: In the beginning God created the heavens and the earth. Vietnamese: Ban dâu Dúc Chúa Tròi dung nên tròi dât. English: God called the expanse heaven. Vietnamese: Dúc Chúa Tròi dat tên khoang không la tròi. English: … you are this day like the stars of heaven in number. Vietnamese: … các nguoi dông nhu sao trên tròi. Example borrowed from Jason Eisner

25. EM: Estimation Maximization • Assume a probability distribution (weights) over hidden events • Take counts of events based on this distribution • Use counts to estimate new parameters • Use parameters to re-weight examples. • Rinse and repeat

26. 0.65 0.25 0.05 null I like milk null I like milk null I like milk Je aime le lait Je aime le lait Je aime le lait 0.01 0.01 0.01 null I like milk null I like milk null I like milk Je aime le lait Je aime le lait Je aime le lait 0.01 0.001 null I like milk null I like milk Je aime le lait Je aime le lait Alignment Hypotheses

27. Weighted Alignments • What we’ll do is: • Consider every possible alignment • Give each alignment a weight - indicating how good it is • Count weighted alignments as normal

28. Good grief! We forgot about P(F|E)! • No worries, a little more stats gets us what we need:

29. Big Example: Corpus 1 fast car voiture rapide 2 fast rapide

30. Possible Alignments 1a 1b 2 fast car fast car fast voiture rapide voiture rapide rapide

31. Parameters 1a 1b 2 fast car fast car fast voiture rapide voiture rapide rapide

32. Weight Calculations 1a 1b 2 fast car fast car fast voiture rapide voiture rapide rapide

33. Count Lines 1a 1b 2 fast car fast car fast 1/2 1/2 1 voiture rapide voiture rapide rapide

34. Count Lines 1a 1b 2 fast car fast car fast 1/2 1/2 1 voiture rapide voiture rapide rapide

35. Count Lines 1a 1b 2 fast car fast car fast 1/2 1/2 1 voiture rapide voiture rapide rapide Normalize

36. Parameters 1a 1b 2 fast car fast car fast voiture rapide voiture rapide rapide

37. Weight Calculations 1a 1b 2 fast car fast car fast voiture rapide voiture rapide rapide

38. Count Lines 1a 1b 2 fast car fast car fast 1/4 3/4 1 voiture rapide voiture rapide rapide

39. Count Lines 1a 1b 2 fast car fast car fast 1/4 3/4 1 voiture rapide voiture rapide rapide

40. Count Lines 1a 1b 2 fast car fast car fast 1/4 3/4 1 voiture rapide voiture rapide rapide Normalize

41. After many iterations: 1a 1b 2 fast car fast car fast ~0 ~1 1 voiture rapide voiture rapide rapide

42. Seems too easy? • What if you have no 1-word sentence? • Words in shorter sentences will get more weight - fewer possible alignments • Weight is additive throughout the corpus: if a word e shows up frequently with some other word f, P(f|e) will go up

43. Some things I skipped • Enumerating all possible alignments: • Very easy with this model: The independence assumptions save us • Model could be a lot better: • Word positions • Multiple f’s generated by the same e • Can actually use an HMM!

44. The Final Product • Now we have a model for P(F|E) • Test it by aligning a corpus! • IE: Find argmaxAP(A|F,E) • Use it for translation: • Combine with favorite model for P(E) • Search space of English sentences for one that maximizes P(E)P(F|E) for a given F

45. Questions? ?