1 / 38

The MSR ESL Assistant: Detecting and correcting non-native errors in English

The MSR ESL Assistant: Detecting and correcting non-native errors in English. Michael Gamon, Chris Brockett, William B. Dolan, Jianfeng Gao, Dmitriy Belenko (Microsoft Research), Alexandre Klementiev (University of Illinois at Urbana Champaign), Claudia Leacock (Butler Hill Group).

Download Presentation

The MSR ESL Assistant: Detecting and correcting non-native errors in English

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. The MSR ESL Assistant: Detecting and correcting non-native errors in English Michael Gamon, Chris Brockett, William B. Dolan, Jianfeng Gao, Dmitriy Belenko (Microsoft Research), Alexandre Klementiev (University of Illinois at Urbana Champaign), Claudia Leacock (Butler Hill Group)

  2. Making NLP useful

  3. Overview • Motivation • Part I: The system • Error statistics • Different solutions for different errors • Machine learned classifiers for preposition and determiner errors • Adding a language model and web-based examples • Part II: Evaluation on native and non-native data • Part III: Usage and interactions

  4. Motivation: The Story of the Disappearing and Reappearing Slide • 750M people use English as a second or foreign language (vs. 375M as first language) • 74% of use of English is between non-native speakers • As many as 300M people study English in China

  5. Error statistics • Previous studies: • Articles and prepositions account for 20% - 50% of ESL errors • Prepositions are difficult for learners with various L1 backgrounds

  6. Error statistics • NICT Japanese Learners of English corpus: • 26.6% of errors are determiner related • 10% of errors are preposition related • CLEC Chinese Learners’ Corpus: • 10% of errors determiner and number related • 2% preposition related, 5% collocation errors (which often involve prepositional collocations)

  7. Most frequent errors made by East Asian non-native speakers • Preposition presence and choice: Finally, the pollution on the world is serious. • Definite and indefinite determiner presence and choice: We should think whether we have abilityto do it well. • Noun pluralization: So other works couldn't be done in adequate times. • Gerund/infinitive confusion: So, money is also important in improve people's spirit. • Auxiliary verb presence and choice: The fire will break out, it can do harmfulto people. • Over-regularized verb inflection: It was builded in 1995. • Adjective/noun confusion: There was a wonderful women volleyball match between Chinese team and Cuba team. • Word order (adjective sequences and nominal compounds): A popBritish bandcalled "Spice Girl" has sung a song.

  8. Different errors – different solutions • Prepositions and articles: much contextual information needed • Over-regularized verb morphology: local information is enough • Noun number: local information (mass noun, quantifier etc) is enough • Machine learned approaches for (1), simple heuristics for (2) and (3). • Total number of error modules: 4 machine-learned modules, 19 heuristic models

  9. Modeling preposition and determiner errors • What data?

  10. Modeling preposition and determiner errors • Preprocessing: tokenization, POStagging • Heuristic algorithm (based on POS tags): find left edges of NPs (potential sites for prepositions and articles) • For each potential site of a preposition or article: • Target feature 1: preposition/article present or absent • Target feature 2: choice of preposition/article (if present) • Contextual features (POS tags to the left/right, tokens to the left/right) • Maximum Entropy classifier

  11. Modeling preposition and determiner errors Training data: 2.5M sentences: Encarta, Reuters, UN, EU, web scraped

  12. Adding a language model

  13. Adding web search • Observation: Non-native speakers often use the web to validate word choice

  14. Show suggestions and originals in context

  15. Evaluation (1): native text (correct usage of prepositions and determiners) • Splitting the original training data into 70% training, 30% test • Note: classification is split into two questions: • Should there be a determiner/preposition? • If yes, which one should it be? (Prepositions: limiting the set to 12 choices that are common in errors: about, as, at, by, for, from, in, like, of, on, since, to, with, "other“)

  16. Articles: results on native text

  17. Prepositions: results on native text

  18. Results on individual prepositions

  19. Evaluation(2): Human evaluation • Spellchecked Chinese Learners’ Corpus (CLEC) • Test set scraped from the web • User data

  20. Spellchecked Chinese Learners’ Corpus (CLEC) • 1 million words of English compositions • collected from Chinese learners of English in China with differing levels of proficiency: • senior secondary school students • English-major university students • non-English-major university students

  21. Web scraped data • collected by a vendor for MSR • Scraped from 489 personal web pages and blogs of non-native speakers/students of English, of Korean, Chinese, or Japanese L1 background • 6746 sentences, 1k selected randomly for our evaluation • Education level ranges from high school to graduate school, professionals are also included • Gender balanced

  22. Intermission: Pie charts

  23. Prepositions

  24. Articles

  25. Broader categories adj related verb related noun related prep related CLEC Web scraped

  26. Usage of the prototype and evaluation of user data

  27. Page views per day Live Translator snafu Beijing Olympics

  28. User location

  29. Users and Sessions

  30. Repeat users (2)

  31. Return visits

  32. Collected data

  33. User interactions

  34. 84% of squiggles are examined by the user

  35. Are users accepting the right suggestions? suggested accepted

  36. In summary • Large market for ESL proofing tools • Detecting and correcting non-native errors is a non-trivial and interesting research problem • We may already be at a point where the technology starts to be useful

  37. Some open questions • How does the accuracy of POStagging influence the accuracy of the overall system? • How can we best leverage the user behavior as a supervision signal?

  38. Some ideas • Using web result counts directly as an LM approximation • Using web result counts as (part of a) supervision signal for ML • Combining more sources of evidence: LMs trained on different data sets etc • Build one single model, including LM scores • Active learning to optimize thresholds

More Related