1 / 21

The value of Post Editing - IBM Case Study

Frank X. Rojas, Jian Ming Xu, Santi Pont Nesta, Álex Martínez Corrià, Salim Roukos, Helena Chapman, Saroj K. Vohra June 2011. The value of Post Editing - IBM Case Study. IBM Case Study – MT Post Editing. Introduction MT Innovation Process Overview Findings Conclusion / Recommendations.

krisalyn
Download Presentation

The value of Post Editing - IBM Case Study

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Frank X. Rojas, Jian Ming Xu, Santi Pont Nesta, Álex Martínez Corrià, Salim Roukos, Helena Chapman, Saroj K. Vohra June 2011 The value of Post Editing - IBM Case Study

  2. IBM Case Study – MT Post Editing • Introduction • MT Innovation • Process Overview • Findings • Conclusion / Recommendations

  3. IBM World Wide Translation Operations Marketing Material Machine Translation Legal/Safety/ Contracts Multimedia Overall End to End Product Integrated Publications Process Information Management Francization Centralized DTP Cultural Consultancy Web • 24 Centers World Wide • ~115 Translation Suppliers • Process ~2.8 B Words • Translate ~0.4 B Words • ~60 language pairs One Stop Shop for all Translation Services

  4. IBM Professional Translation Services Consistent Quality Standards Global Brand Identity Professional Quality Standards 2 • Future: • Ability to reduce cost using conventional methods reaching limits • Business pressure for additional cost elimination • Looking to MT Technology as next wave to reach business goals TraditionalTechnology ProcessMgmt CAT Editor Memory Assets 1 Unit Cost >50% Reduction Professional Memory 72%  85% Re-Use 3 Human Skill

  5. n.Fluent customized with WWTO translation memories RTTS introduced in 2006 as platform for speech and text translation, developed by IBM Research RTTS licensed to IBM partners - MT portal- Generic crowdsourcing - Text translation services June 2008 eSupport“Translate This Page”switch to n.Fluent Historical Perspective 2010 MT piloting Pilot: SPA, ITA, FRE, GER ------------------------------------- New E2E process Partnership: WWTO/n.Fluent 8.6 M words Initial n.Fluent/WWTO Spanish MT pilot ------------------------------------- Improve efficiency of professional translators 2012 2011 2011 MT Training Pilot: GER, BPR, JPN, CHS ------------------------------------- MT payment profiles ready 16.0 M words target 2010 2009 2008 2007 2006 eSupport (www)“Translate This Page”JPN pilot /rule engine

  6. MT Critical Success Metrics • Necessary and sufficient condition to measure success • 5.0 M words sampled • Minimum of 3 languages • Net Contribution to ROI by MT Engine:10% of payable words should be MT • No more than 5% adverse impact to Overall Quality Index • No more than 5% impact to Customer Satisfaction • Lack of industry metrics and guidance. • Active research on MT technology... no guidance on operational impacts • A business vacuum existed on how to integrate MT services • No operational process had been defined for MT services

  7. Recent Digital Innovations with Biggest Impact in the Business World* • IBM’s Watson Q&A computer • Google’s autonomous car • Technologies to understand and produce natural human speech • Instantaneous, high-quality machine translation • Smartphones / App phones in the developing world *Andrew McAfee is a principal research scientist in the MIT Sloan School of Business

  8. IT HELP DESK BLEU 0.5 0.45 0.4 Quality 0.35 0.3 0.25 0.2 0.15 0.1 0.05 Base 29k 180k 350k 0 Words Real-Time Translation Server (RTTS) & n.Fluent • Real Time Translation Server (RTTS) • IBMs MT Engine • RTTS provides machine translation for n.Fluent &other applications • APIs allow other applications to access these translation services. • Customization tools – Domains, chat-specific models, … • Commercially licensed to IBM partners • Language Pairs to/from English: • n.Fluent • IBMs MT translation application • Providing machine translation services for: • Text, web pages, and documents (Word, Excel, …) • Instant Messaging chats (via IM plug-in) • Mobile translation application (BlackBerry and others) • Enabled with LEARNING via crowdsourcing (internal 450K IBMers) • Deployed for eSupport self serving tech support (external) Français العربية Español Deutsch 日本語 한국어 中文 Русский English Italiano Português

  9. n.Fluent customized with WWTO translation memories RTTS introduced in 2006 as platform for speech and text translation, developed by IBM Research RTTS licensed to IBM partners - MT portal- Generic crowdsourcing - Text translation services June 2008 eSupport“Translate This Page”switch to n.Fluent Historical Perspective 2010 MT piloting Pilot: SPA, ITA, FRE, GER ------------------------------------- New E2E process Partnership: WWTO/n.Fluent 8.6 M words Initial n.Fluent/WWTO Spanish MT pilot ------------------------------------- Improve efficiency of professional translators 2012 2011 2011 MT Training Pilot: GER, BPR, JPN, CHS ------------------------------------- MT payment profiles ready 16.0 M words target 2010 2009 2008 2007 2006 eSupport (www)“Translate This Page”JPN pilot /rule engine

  10. Shipment English New / Changed 100%Exact Match TM MT MT Post Editing End to End Workflow TM Pre-Process Editing Session TMMatchAnalysis CAT Translation • Show best choice vs vs • Select best choice(Post Edit rules) • Commit language • Upfront & on-going MT tuning via IBM TM professional translations • Professional translation = Best context • Matching methods • Traditional TM – breaks down content @ segment level • Machine TM – breaks down segments @ block level using MT models • – reconstructs segments preserving formats/mark-up tags • MT service level integration TESTING QUALITY MT Pre-Process MTModel & Trans. = Localization Kit (NLV Folder)

  11. New / Changed New / Changed 100%Exact Match 100%Exact Match TM MT MT Pre-processing ALL segment“no match segments” Domain specificparallel training corpus TM • Initial MT corpus • done before start of project Build dynamic,domain specificMT model MTinitial corpus Localization kit General parallel training corpus MT Translation ofno match segments

  12. TM Editing Environment TM Environment Xxx xxx xx xxx xxx xxx. The application unprotects files before exporting them. Yy yyy yyy Translator options • Ignore fuzzy and MT • Post edit MT • Post edit fuzzy Translation Memory 0 - The application unprotects files before exporting them. 1[m] – La aplicación desprotege archivos antes de exportarlos. 2[f 85%] - La aplicación protege los archivos antes de exportarlos Two Seconds Rule: Translators are trained on several strategies to make a quick choice [Ctrl + 1] MT TM TM Environment Xxx xxx xx xxx xxx xxx. La aplicación desprotege los archivos antes de exportarlos. Yy yyy yyy Typed

  13. Productivity Measurements accept match [~0 time] edit match [X time] reject match [manual translation] • Start segment • Choose action • End segment • MT productivity evaluation log (MTeval Log) • N events • Words | Time | Existing Proposal | Used Proposal | ... • Examine productivity per payment category • SUM(Words) / SUM(Time) • Use of IBM Business Analytic Tool (SPSS) • Trim events that fall into 5% (slowest) and 95% (fastest) percentile Each event EM : ExactRM : ReplaceFM : FuzzyMT : MachineNP : No Proposal A) = “best” Existing ProposalB) = “alternative” Existing ProposalC) = reject all Existing Proposal, 100% human labor

  14. Single Shipment EXAMPLE • Total # events : 2,309 (377+1,932) • Total words: 24,150 Total time: 27,362 • 3,911 w/ MT match 11,377 w/ MT match • 20,239 w/o MT match 15,985 w/o MT match • MT impact to productivity • MT : 0.44 words/sec [1777 words / 4071 sec] • NP • 0.21 w/ MT match • 0.32 w/o MT match  Baseline (placebo) • MT Leverage : 71.8% [1777 / (1777+697)] rate(MT) / rate(NP): 1.37 i.e. Translator can complete 37% more words in the same time. Key metrics

  15. 8.00 7.00 6.00 5.00 FM Productivity ratio 4.00 FM-MT MT 3.00 2.00 1.00 0.00 FRE GER ITA SPA MT Impact on Fuzzy Match : 4Q10 Findings • When FM & MT matches exist simultaneously • Productivity: rate(MT) / rate(NP): • Case : Translator edits FM • FM-MT Combined case • Case: Translator edits MT • Overall • Machine matches not as good as professional (fuzzy) matches • No statistical impact to fuzzy productivity to include MT matches. • SPA highest sample case 28.6% 4.4% FM-MT Pick Rate: 57.6% 46.9% ** Findings subject to change with additional sampling.

  16. MT Key Metrics: 4Q10 Findings • 8.6 M words sampled in real time translation service. • SPA : Qualified MT engine 4Q10 • ITA : Qualified MT engine 4Q10 • FRA : Qualified MT engine 1Q11 • While rate(MT) / rate(NP) is high, the findings were not statistically significant in 4Q. • GER : Insufficient productivity from MT engine ** Findings subject to change with additional sampling.

  17. Overall Savings Assessment • Overall savings % • Word savings due to MT efficiency • Convert time savings  MT payment factor % • MT payment factor X [MT % words + NP % words] • Results in less payable words. • MT productivity savings drives a overall savings • These are not the same due to MT % distribution. • Supply chain has to consider cost of MT services ** Findings subject to change with additional sampling.

  18. Pay for MT Words Translated not MT Matches • We pay for final results (MT payable words) not MT matches • MT matches considered “opinion” until chosen by a human • Too many opinions & opinions by immature MT models are less efficient. • Actual MT payable words have value beyond the specific project • Post Edited words are reused in future and unknown MT context • Engine has to deliver consistent MT payable words • Minimum needed to quality an MT engine for compensation • High MT productivity [rate(MT) / rate(NP)] • High MT leverage [% of MT matches used] • Compensation to be based on MT payment factor

  19. Variance across Languages • There is no single maturity path when modeling MT engines across many languages. • IBM Pilot: each trained MT engine is a unique asset. • Some languages require more modeling/tuning than others. • Language pairs that service “Loose -> Structured” languages are struggling • German requires more effort than Spanish • Are there limitations to statistical MT engines? • New thinking may need to be explored? • Each MT engine will have separate MT payment factors.

  20. Perspective of MT Post Edit Pilots DomainSpecific HIGHER All IBMexternal/internalPubs / UI Memory Assets Quality / Reliability WWTO“human” New external(2011 Pilots) internal IBM n.Fluent“machine” internal IBM LOWER General MT Post Editing has impacts across entire Translation Service Hierarchy

  21. MT Post Editing Project – Key Lessons • Professional (Human) memories are the best assets and deliver the highest quality. • Professional memories are a key asset for MT success. • All Memory assets need to be protected and managed. • Flow of memories between Professional and Machine must be properly balanced. • Dynamic modeling offers significant advantage over static modeling. • Continuous business analytics is needed to optimize machine assets. • A single cost model per language is needed, independent of MT services/engines. • An aggressive yet cautious approach is warranted to go forward. MT Post Editing does improve productivity and efficiency of a localization supply chain.

More Related