Silke Gutermuth & Silvia Hansen-Schirra University of Mainz Germany

1. Post-editing machine translation – a usability test for professional translation settings Silke Gutermuth & Silvia Hansen-Schirra University ofMainz Germany

2. Post-editing? • “term used for the correction of machine translation output by human linguists/editors” (Veale & Way 1997) • “taking raw machine translated output and then editing it to produce a 'translation' which is suitable for the needs of the client” (one student explaining post-editing to another) • “is the process of improving a machine-generated translation with a minimum of manual labour” (TAUS Report 2010)

3. Degrees of Post-editing • lightorfastpost-editing • essential correctionsonly • time factor: quick • full post-editing • morecorrections=> higherquality • time factor: slow • (O‘Brien et al. 2009)

4. Background • Motivation: evaluationofmachinetranslation (MT), post-editingof MT, eye-enhanced CAT workbenches(e.g. O‘Brien 2011, Doherty et al. 2010, Carl & Jakobsen 2010, Hyrskykari 2006) • Project: in cooperationwithCopenhagen Business School (http://www.cbs.dk/Forskning/Institutter-centre/Institutter/CRITT/Menu/Forskningsprojekter) • Experiment: • English-German • translation vs. post-editing vs. editing • 6 sourcetexts (ST) with different complexitylevels(Hvelplund 2011) • 12 professional translators, 12 semi-professional translators • eye-tracking (Tobii TX 300), key-logging (Translog), retrospectiveinterviews, questionnaires

5. Translators‘ self-estimation

6. Translators‘ self-estimation

7. Translators‘ evaluationof MT quality

8. Translators‘ evaluationof MT quality

9. Translators‘ evaluationof MT quality Professional translators: conscious, subjective rating of machine translated output is extremely negative. Can eye-tracking tell a different story dealing with objective and measurable facts?

10. Processing time

11. Processing time edited texts quite often suffer from a distortion of meaning => source text needed for good quality translation => post-editing

12. Processing of ST vs. TT Translation

13. Processing of ST vs. TT Translation vs. Post-editing

14. Processing metricsTranslation vs. Post-editing Translation: correlationbetweenincreasing ST complexityand TT processingmetrics Post-editing: no significant influence of ST complexity on TT processing metrics

15. Processing of ST Translationvs. Post-editing => post-editing more efficient WHY?

16. Fixation Duration ofclauses Average fixationduration (in milliseconds ) per clause

17. Fixation Duration ofclauses Average fixationduration (in milliseconds ) per clause Goodqualityof MT for non-finite clauses ST: to end thesuffering TT-P: um das Leiden zu beenden ST: Althoughemphasizingthat TT-P: Obwohl betont wird, dass ST: toprotestagainst TT-P: um gegen … zu protestieren ST: in thewakeoffightingflaring TT-P: im Zuge des Kampfes gegen ein erneutesupagain in Dafur Aufflammen in Darfur

18. PreliminaryConclusions • Efficient post-editing is possible under the following conditions: • good machine translation quality • post-editors who are language experts, i.e. they need • knowledge of the conventions of the source and target language • knowledge of the text type and register

19. What‘snext? • Analysis ofothercontrastivedifferencesandgaps • Analysis ofambiguitiesandprocessingproblems • Comparisonofcomplexitylevels • Analysis ofmonitoringprocessesduring TT production (withTranslog) • Comparisonof professionals vs. semi-professionals • Correlationsbetweenprocess data and quality of participants’ outputs • Comparisonwithothertranslationpairs

20. Bibliography • Carl, Michael andJakobsen, ArntLykke (2010): RelatingProduction Units andAlignment Units in Translation Activity Data, In Proceedingsof International Workshop on Natural Language Processing andCognitive Science (NLPCS), Madeira, Portugal. • Doherty, Stephen, O'Brien, Sharon and Carl, Michael (2010): Eye tracking as an MT evaluation technique. Machine Translation, 24, 1, pp1-13. • Hvelplund, Kristian Tangsgaard (2011): Allocation of cognitive resources in translation an eye-tracking and key-logging study. PhD thesis, Department of International Language Studies and Computational Linguistics, Copenhagen Business School. • Hyrskykari , Aulikki (2006): Eyes in Attentive Interfaces: Experiences from Creating iDict, a Gaze-Aware Reading Aid. Dissertation, Tampere University Press. • O'Brien, Sharon and Roturier,Johann and De Almeida, Giselle (2009): Post-Editing MT Output Views from the researcher, trainer, publisher and practitioner. http://www.mt-archive.info/MTS-2009-O’Brien-ppt.pdf • O'Brien, Sharon (2011): Towards Predicting Post-Editing Productivity. Machine Translation, 25, 3, pp197-215. • Postediting in Practice. A TAUS Report, March 2010 p.6 • Veale, T. and Way, A. (1997). Gaijin: A Bootstrapping Approach to Example-BasedMachine Translation. RecentAdvances in Natural Language International Conference, 239-244.

21. Silke Gutermuth & Silvia Hansen-Schirra gutermsi@uni-mainz.de & hansenss@uni-mainz.de http://www.staff.uni-mainz.de/hansenss/ Contact