1 / 78

The AVENUE Project: Bootstrapping MT Prototypes for Languages with Limited Resources

The AVENUE Project: Bootstrapping MT Prototypes for Languages with Limited Resources. Faculty: Alon Lavie , Jaime Carbonell, Lori Levin, Ralf Brown Students: Erik Peterson, Christian Monson, Ariadna Font-Llitjos, Alison Alvarez. Progression of MT. Started with rule-based systems

bena
Download Presentation

The AVENUE Project: Bootstrapping MT Prototypes for Languages with Limited Resources

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The AVENUE Project:Bootstrapping MT Prototypes for Languages with Limited Resources Faculty: Alon Lavie, Jaime Carbonell, Lori Levin, Ralf Brown Students: Erik Peterson, Christian Monson, Ariadna Font-Llitjos, Alison Alvarez

  2. Progression of MT • Started with rule-based systems • Very large expert human effort to construct language-specific resources (grammars, lexicons) • High-quality MT extremely expensive  only for handful of language pairs • Along came EBMT and then SMT… • Replaced human effort with extremely large volumes of parallel text data • Less expensive, but still only feasible for a small number of language pairs • We “traded” human labor with data • Where does this take us in 5-10 years? • Large parallel corpora for maybe 25-50 language pairs • What about all the other languages? • Is all this data (with very shallow representation of language structure) really necessary? • Can we build MT approaches that learn deeper levels of language structure and how they map from one language to another? AVENUE Project

  3. Why Machine Translation for Languages with Limited Resources? • We are in the age of information explosion • The internet+web+Google anyone can get the information they want anytime… • But what about the text in all those other languages? • How do they read all this English stuff? • How do we read all the stuff that they put online? • MT for these languages would Enable: • Better government access to native indigenous and minority communities • Better minority and native community participation in information-rich activities (health care, education, government) without giving up their languages. • Civilian and military applications (disaster relief) • Language preservation AVENUE Project

  4. The Roadmap to Learning-based MT • Automatic acquisition of necessary language resources and knowledge using machine learning methodologies: • Learning morphology (analysis/generation) • Rapid acquisition of broad coverage word-to-word and phrase-to-phrase translation lexicons • Learning of syntactic structural mappings • Tree-to-tree structure transformations [Knight et al], [Eisner], [Melamed] require parse trees for both languages • Learning syntactic transfer rules with resources (grammar, parses) for just one of the two languages • Automatic rule refinement and/or post-editing • Effective integration of acquired knowledge with statistical/distributional information AVENUE Project

  5. CMU’s AVENUE Approach • Elicitation: use bilingual native informants to produce a small high-quality word-aligned bilingual corpus of translated phrases and sentences • Building Elicitation corpora from feature structures • Feature Detection and Navigation • Transfer-rule Learning: apply ML-based methods to automatically acquire syntactic transfer rules for translation between the two languages • Learn from major language to minor language • Translate from minor language to major language • XFER + Decoder: • XFER engine produces a lattice of all possible transferred structures at all levels • Decoder searches and selects the best scoring combination • Rule Refinement: refine the acquired rules via a process of interaction with bilingual informants • Morphology Learning • Word and Phrase bilingual lexicon acquisition AVENUE Project

  6. Word-aligned elicited data English Language Model Learning Module Run Time Transfer System Word-to-Word Translation Probabilities Transfer Rules {PP,4894};;Score:0.0470PP::PP [NP POSTP] -> [PREP NP]((X2::Y1)(X1::Y2)) Decoder Lattice Translation Lexicon AVENUE Architecture AVENUE Project

  7. Learning Transfer-Rules for Languages with Limited Resources • Rationale: • Bilingual native informant(s) can translate and align a small pre-designed elicitation corpus, using elicitation tool • Elicitation corpus designed to be typologically and structurally comprehensive and compositional • Transfer-rule engine and new learning approach support acquisition of generalized transfer-rules from the data AVENUE Project

  8. Type information Part-of-speech/constituent information Alignments x-side constraints y-side constraints xy-constraints, e.g. ((Y1 AGR) = (X1 AGR)) Transfer Rule Formalism ;SL: the old man, TL: ha-ish ha-zaqen NP::NP [DET ADJ N] -> [DET N DET ADJ] ( (X1::Y1) (X1::Y3) (X2::Y4) (X3::Y2) ((X1 AGR) = *3-SING) ((X1 DEF = *DEF) ((X3 AGR) = *3-SING) ((X3 COUNT) = +) ((Y1 DEF) = *DEF) ((Y3 DEF) = *DEF) ((Y2 AGR) = *3-SING) ((Y2 GENDER) = (Y4 GENDER)) ) AVENUE Project

  9. Rule Learning - Overview • Goal: Acquire Syntactic Transfer Rules • Use available knowledge from the source side (grammatical structure) • Three steps: • Flat Seed Generation: first guesses at transfer rules; flat syntactic structure • Compositionality:use previously learned rules to add hierarchical structure • Constraint Learning: refine rules by learning appropriate feature constraints AVENUE Project

  10. Flat Seed Rule Generation AVENUE Project

  11. Compositionality AVENUE Project

  12. Constraint Learning AVENUE Project

  13. AVENUE Prototypes • General XFER framework under development for past three years • Prototype systems so far: • German-to-English, Spanish-to-English • Hindi-to-English, Hebrew-to-English • In progress or planned: • Mapudungun-to-Spanish • Quechua-to-Spanish • Arabic-to-English • Native-Brazilian languages to Brazilian Portuguese AVENUE Project

  14. Morphology Learning • Unsupervised learning of morphemes and their function from raw monolingual data • Segmentation of words into morphemes • Identification of morphological paradigms (inflections and derivations) • Learning association between morphemes and their function in the language AVENUE Project

  15. Ø.ed.ing.ly.s 4 clear open … d.ded.ding.ds 19 ad boar defen … Ø.ed.ing.s 106 clear defend open present … Ø.ed.ing.ly 6 clear open present Total d.ded.ding 27 ai boar defen … Ø.ed.ing 201 aid clear defend deliver … Ø.ed.ly 11 clear direct present quiet … ed.ly 12 bodi clear correct quiet … • Morphology Learning • AVENUE Approach: • Organize the raw data in the • form of a network of paradigm • candidate schemes • Search the network for a • collection of schemes that • represent true morphology • paradigms of the language • Learn mappings between the • schemes and features/functions • using minimal pairs of elicited • data • Construct analyzer based on the • collection of schemes and the • acquired function mappings AVENUE Project

  16. a.as.i.o.os.sandra.tanier.ter.tro.trol 1 cas a.as.o.os 43 african cas jurídic l ... a.as.o 59 cas citad jurídic l ... a.as.os 50 afectad cas jurídic l ... a.o.os 105 impuest indonesi italian jurídic ... as.o.os 54 cas implicad jurídic l ... a.as 199 huelg incluid industri inundad ... a.o 214 id indi indonesi inmediat ... as.o 85 intern jurídic just l ... a.os 134 impedid impuest indonesi inundad ... as.os 68 cas implicad inundad jurídic ... o.os 268 human implicad indici indocumentad ... a.tro 2 cas cen tro 16 catas ce cen cua ... a 1237 huelg ib id iglesi ... as 404 huelg huelguist incluid industri ... o 1139 hub hug human huyend ... os 534 humorístic human hígad impedid ... • Figure : Hierarchical scheme lattice automatically derived from a Spanish newswire corpus of 40,011 words and 6,975 unique types. AVENUE Project

  17. Automated Rule Refinement • Rationale: • Bilingual informants can identify translation errors and pinpoint the errors • A sophisticated trace of the translation path can identify likely sources for the error and do “Blame Assignment” • Rule Refinement operators can be developed to modify the underlying translation grammar (and lexicon) based on characteristics of the error source: • Add or delete feature constraints from a rule • Bifurcate a rule into two rules (general and specific) • Add or correct lexical entries AVENUE Project

  18. New Research Directions • Automatic Transfer Rule Learning: • In the “large-data” scenario: from large volumes of uncontrolled parallel text automatically word-aligned • In the absence of morphology or POS annotated lexica • Learning mappings for non-compositional structures • Effective models for rule scoring for • Decoding: using scores at runtime • Pruning the large collections of learned rules • Learning Unification Constraints – VSL • Integrated Xfer Engine and Decoder • Improved models for scoring tree-to-tree mappings, integration with LM and other knowledge sources in the course of the search AVENUE Project

  19. Missing Science • Monolingual learning tasks: • Learning morphology: morphemes and their meaning • Learning syntactic and semantic structures: grammar induction • Bilingual Learning Tasks: • Automatic acquisition of word and phrase translation lexicons • Learning structural mappings (syntactic, semantic, non-compositional) • Models that effectively combine learned symbolic knowledge with statistical information: new “decoders” AVENUE Project

  20. AVENUE Project

  21. English-Chinese Example AVENUE Project

  22. English-Hindi Example AVENUE Project

  23. Spanish-Mapudungun Example AVENUE Project

  24. English-Arabic Example AVENUE Project

  25. Value constraints Agreement constraints Transfer Rule Formalism (II) ;SL: the old man, TL: ha-ish ha-zaqen NP::NP [DET ADJ N] -> [DET N DET ADJ] ( (X1::Y1) (X1::Y3) (X2::Y4) (X3::Y2) ((X1 AGR) = *3-SING) ((X1 DEF = *DEF) ((X3 AGR) = *3-SING) ((X3 COUNT) = +) ((Y1 DEF) = *DEF) ((Y3 DEF) = *DEF) ((Y2 AGR) = *3-SING) ((Y2 GENDER) = (Y4 GENDER)) ) AVENUE Project

  26. AVENUE Partners AVENUE Project

  27. The Transfer Engine AVENUE Project

  28. Seeded VSL: Some Open Issues • Three types of constraints: • X-side constrain applicability of rule • Y-side assist in generation • X-Y transfer features from SL to TL • Which of the three types improves translation performance? • Use rules without features to populate lattice, decoder will select the best translation… • Learn only X-Y constraints, based on list of universal projecting features • Other notions of version-spaces of feature constraints: • Current feature learning is specific to rules that have identical transfer components • Important issue during transfer is to disambiguate among rules that have same SL side but different TL side – can we learn effective constraints for this? AVENUE Project

  29. Examples of Learned Rules (Hindi-to-English) AVENUE Project

  30. XFER MT for Hebrew-to-English • Two month intensive effort to apply our XFER approach to the development of a Hebrew-to-English MT system • Challenges: • No large parallel corpus • Limited coverage translation lexicon • Rich Morphology: incomplete analyzer available • Accomplished: • Collected available resources, establish methodology for processing Hebrew input • Translated and aligned Elicitation Corpus • Learned XFER rules • Developed (small) manual XFER grammar as a point of comparison • System debugging and development • Evaluated performance on unseen test data using automatic evaluation metrics AVENUE Project

  31. Hebrew Input בשורה הבאה Preprocessing Morphology Transfer Rules English Language Model {NP1,3} NP1::NP1 [NP1 "H" ADJ] -> [ADJ NP1] ((X3::Y1) (X1::Y2) ((X1 def) = +) ((X1 status) =c absolute) ((X1 num) = (X3 num)) ((X1 gen) = (X3 gen)) (X0 = X1)) Transfer Engine Translation Lexicon Decoder N::N |: ["$WR"] -> ["BULL"] ((X1::Y1) ((X0 NUM) = s) ((Y0 lex) = "BULL")) N::N |: ["$WRH"] -> ["LINE"] ((X1::Y1) ((X0 NUM) = s) ((Y0 lex) = "LINE")) Translation Output Lattice (0 1 "IN" @PREP) (1 1 "THE" @DET) (2 2 "LINE" @N) (1 2 "THE LINE" @NP) (0 2 "IN LINE" @PP) (0 4 "IN THE NEXT LINE" @PP) English Output in the next line AVENUE Project

  32. Morphology Example • Input word: B$WRH 0 1 2 3 4 |--------B$WRH--------| |-----B-----|$WR|--H--| |--B--|-H--|--$WRH---| AVENUE Project

  33. Morphology Example Y0: ((SPANSTART 0) Y1: ((SPANSTART 0) Y2: ((SPANSTART 1) (SPANEND 4) (SPANEND 2) (SPANEND 3) (LEX B$WRH) (LEX B) (LEX $WR) (POS N) (POS PREP)) (POS N) (GEN F) (GEN M) (NUM S) (NUM S) (STATUS ABSOLUTE)) (STATUS ABSOLUTE)) Y3: ((SPANSTART 3) Y4: ((SPANSTART 0) Y5: ((SPANSTART 1) (SPANEND 4) (SPANEND 1) (SPANEND 2) (LEX $LH) (LEX B) (LEX H) (POS POSS)) (POS PREP)) (POS DET)) Y6: ((SPANSTART 2) Y7: ((SPANSTART 0) (SPANEND 4) (SPANEND 4) (LEX $WRH) (LEX B$WRH) (POS N) (POS LEX)) (GEN F) (NUM S) (STATUS ABSOLUTE)) AVENUE Project

  34. Sample Output (dev-data) maxwell anurpung comes from ghana for israel four years ago and since worked in cleaning in hotels in eilat a few weeks ago announced if management club hotel that for him to leave israel according to the government instructions and immigration police in a letter in broken english which spread among the foreign workers thanks to them hotel for their hard work and announced that will purchase for hm flight tickets for their countries from their money AVENUE Project

  35. Test set of 62 sentences from Haaretz newspaper, 2 reference translations Evaluation Results AVENUE Project

  36. Future Directions • Continued work on automatic rule learning (especially Seeded Version Space Learning) • Use Hebrew and Hindi systems as test platforms for experimenting with advanced learning research • Rule Refinement via interaction with bilingual speakers • Developing a well-founded model for assigning scores (probabilities) to transfer rules • Redesigning and improving decoder to better fit the specific characteristics of the XFER model • Improved leveraging from manual grammar resources • MEMT with improved • Combination of output from different translation engines with different confidence scores • strong decoding capabilities AVENUE Project

  37. Flat Seed Generation Create a transfer rule that is specific to the sentence pair, but abstracted to the POS level. No syntactic structure. AVENUE Project

  38. Compositionality - Overview • Traverse the c-structure of the English sentence, add compositional structure for translatable chunks • Adjust constituent sequences, alignments • Remove unnecessary constraints, i.e. those that are contained in the lower-level rule AVENUE Project

  39. Seeded Version Space Learning: Overview • Goal: add appropriate feature constraints to the acquired rules • Methodology: • Preserve general structural transfer • Learn specific feature constraints from example set • Seed rules are grouped into clusters of similar transfer structure (type, constituent sequences, alignments) • Each cluster forms a version space: a partially ordered hypothesis space with a specific and a general boundary • The seed rules in a group form the specific boundary of a version space • The general boundary is the (implicit) transfer rule with the same type, constituent sequences, and alignments, but no feature constraints AVENUE Project

  40. Seeded Version Space Learning: Generalization • The partial order of the version space: Definition: A transfer rule tr1 is strictly more general than another transfer rule tr2 if all f-structures that are satisfied by tr2 are also satisfied by tr1. • Generalize rules by merging them: • Deletion of constraint • Raising two value constraints to an agreement constraint, e.g. ((x1 num) = *pl), ((x3 num) = *pl)  ((x1 num) = (x3 num)) AVENUE Project

  41. Seeded Version Space Learning • NP v det n NP VP … • Group seed rules into version spaces as above. • Make use of partial order of rules in version space. Partial order is defined • via the f-structures satisfying the constraints. • Generalize in the space by repeated merging of rules: • Deletion of constraint • Moving value constraints to agreement constraints, e.g. • ((x1 num) = *pl), ((x3 num) = *pl)  • ((x1 num) = (x3 num) • 4. Check translation power of generalized rules against sentence pairs AVENUE Project

  42. Seeded Version Space Learning:The Search • The Seeded Version Space algorithm itself is the repeated generalization of rules by merging • A merge is successful if the set of sentences that can correctly be translated with the merged rule is a superset of the union of sets that can be translated with the unmerged rules, i.e. check power of rule • Merge until no more successful merges AVENUE Project

  43. Conclusions • Transfer rules (both manual and learned) offer significant contributions that can complement existing data-driven approaches • Also in medium and large data settings? • Initial steps to development of a statistically grounded transfer-based MT system with: • Rules that are scored based on a well-founded probability model • Strong and effective decoding that incorporates the most advanced techniques used in SMT decoding • Working from the “opposite” end of research on incorporating models of syntax into “standard” SMT systems [Knight et al] • Our direction makes sense in the limited data scenario AVENUE Project

  44. AVENUE Architecture Run-Time Module Learning Module SL Input SL Parser Morphology Pre-proc Elicitation Process Transfer Rule Learning Transfer Rules Transfer Engine TL Output TL Generator Decoder User AVENUE Project

  45. Learning Transfer-Rules for Languages with Limited Resources • Rationale: • Large bilingual corpora not available • Bilingual native informant(s) can translate and align a small pre-designed elicitation corpus, using elicitation tool • Elicitation corpus designed to be typologically comprehensive and compositional • Transfer-rule engine and new learning approach support acquisition of generalized transfer-rules from the data AVENUE Project

  46. The Elicitation Corpus • Translated, aligned by bilingual informant • Corpus consists of linguistically diverse constructions • Based on elicitation and documentation work of field linguists (e.g. Comrie 1977, Bouquiaux 1992) • Organized compositionally: elicit simple structures first, then use them as building blocks • Goal: minimize size, maximize linguistic coverage AVENUE Project

  47. The Transfer Engine AVENUE Project

  48. Type information Part-of-speech/constituent information Alignments x-side constraints y-side constraints xy-constraints, e.g. ((Y1 AGR) = (X1 AGR)) Transfer Rule Formalism ;SL: the man, TL: der Mann NP::NP [DET N] -> [DET N] ( (X1::Y1) (X2::Y2) ((X1 AGR) = *3-SING) ((X1 DEF = *DEF) ((X2 AGR) = *3-SING) ((X2 COUNT) = +) ((Y1 AGR) = *3-SING) ((Y1 DEF) = *DEF) ((Y2 AGR) = *3-SING) ((Y2 GENDER) = (Y1 GENDER)) ) AVENUE Project

  49. Value constraints Agreement constraints Transfer Rule Formalism (II) ;SL: the man, TL: der Mann NP::NP [DET N] -> [DET N] ( (X1::Y1) (X2::Y2) ((X1 AGR) = *3-SING) ((X1 DEF = *DEF) ((X2 AGR) = *3-SING) ((X2 COUNT) = +) ((Y1 AGR) = *3-SING) ((Y1 DEF) = *DEF) ((Y2 AGR) = *3-SING) ((Y2 GENDER) = (Y1 GENDER)) ) AVENUE Project

  50. Rule Learning - Overview • Goal: Acquire Syntactic Transfer Rules • Use available knowledge from the source side (grammatical structure) • Three steps: • Flat Seed Generation: first guesses at transfer rules; flat syntactic structure • Compositionality:use previously learned rules to add hierarchical structure • Seeded Version Space Learning: refine rules by generalizing with validation (learn appropriate feature constraints) AVENUE Project

More Related