1 / 17

Evaluating Machine Translation Metrics for Student-Written Translations

This study by Michaud and McCoy explores various criteria for judging translation quality, including fluency and fidelity. The researchers analyze different translations to assess accuracy and effectiveness. They also compare teacher scores with automated metrics.

olwen
Download Presentation

Evaluating Machine Translation Metrics for Student-Written Translations

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Applying Machine Translation Metrics to Student-Written Translations Lisa N. MichaudComputer Science DepartmentMerrimack CollegeNorth Andover, Massachusetts, USA Patricia Ann McCoyLanguage DepartmentUniversidad de las AmericasPueblaPuebla, Mexico

  2. Criteria for judging translations fluency(is it well-formed?) fidelity (does it convey original meaning?) (Hovy et al., 2002) Michaud and McCoy

  3. Multiplicity of translations • In each one of these jobs the professor could have agreed to work 6 hours a day and therefore would not be surpassing the working day hour limit. • In each one of these jobs the teacher could have agreed to work 6 hours per day and therefore he wouldn't be bound by the limits of the working day. • In each of these examples the teaching could have been arranged so that he/she works six hours a day and would not be affected by any workday limitations. • In both of these jobs the professor could have agreed to work six hours daily and therefore he wouldn't be affecting his work shift limit. Michaud and McCoy

  4. Multiplicity of translations • In each one of these jobs the professor could have agreed to work 6 hours a day and therefore would not be surpassing the working day hour limit. • In each one of these jobs the teacher could have agreed to work 6 hours per day and therefore he wouldn't be bound by the limits of the working day. • In each of these examples the teaching could have been arranged so that he/she works six hours a day and would not be affected by any workday limitations. • In both of these jobs the professor could have agreed to work six hours daily and therefore he wouldn't be affecting his work shift limit. Michaud and McCoy

  5. BLEU Hypothesis Multiple References Michaud and McCoy

  6. TERp Hypothesis INSERTION PHRASALEQUIVALENCE or SYNONYM SAMESTEM SUBSTITUTION SHIFT Single Reference Michaud and McCoy

  7. TERp alignment and tags Michaud and McCoy

  8. Student translation corpus Michaud and McCoy

  9. Does TERp agree with an expert? Instructor Scores vs Inverted TERp 650 sentences (22%) Pearson Correlation r = 0.232236 Michaud and McCoy

  10. Score distribution Michaud and McCoy

  11. Instructor rubric (original) 10 Excellent 9 Good 8Satisfactory 0-7Deficient Michaud and McCoy

  12. Evaluating TERp tags (pilot) Michaud and McCoy

  13. Future work Michaud and McCoy

  14. Instructor rubric (revised) 100 Excellent 90 Good 80 Satisfactory 0-70 Deficient Michaud and McCoy

  15. Modifying the TERp Score Hypothesis INSERTION PHRASALEQUIVALENCE or SYNONYM SAMESTEM SUBSTITUTION SHIFT Single Reference Michaud and McCoy

  16. Recognizing false cognates Hypothesis cynical Source cínico SUBSTITUTION brazen Single Reference Michaud and McCoy

  17. Extracting mistranslation pairs SPANISHDICTIONARY ENGLISHDICTIONARY cynical cynicalbrazen cínico zona zone Michaud and McCoy

More Related