1 / 82

Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test

Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test. Roberto Navigli. http://lcl.uniroma1.it. ERC StG: Multilingual Joint Word Sense Disambiguation (MultiJEDI) Roberto Navigli. 1. Turing Test (1950). A test of a machine ’s ability to demonstrate intelligence

thisbe
Download Presentation

Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli http://lcl.uniroma1.it ERC StG: Multilingual Joint Word Sense Disambiguation (MultiJEDI) Roberto Navigli 1

  2. Turing Test (1950) • A test of a machine’s ability to demonstrateintelligence • Inspired by a party game, known as the “imitation game" (a man vs. a woman) “I propose to consider the question, Can machines think?[Since "thinking" is difficult to define, let’s] replace the question by another, which is closely related to it and is expressed in relatively unambiguous words. […] Are there imaginable digital computers which would do well in the imitation game?” -Alan Turing, “Computing Machinery and Intelligence” (1950) Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  3. Turing Test (1950) • A test of a machine’s ability to demonstrateintelligence Human judge Human Player Computer Player Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  4. Why a language-driven test? • Because intelligence is manifested through the ability to converse (as a sufficient (?)condition) Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  5. Why a language-driven test? • Because intelligence is manifested through the ability to converse (as a sufficient but not necessary condition) • Because Natural Language Understanding (NLU) is often thought of as an AI-complete problem • NLU requires extensive knowledge about the world and the ability to interact with it Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  6. ELIZA (Weizenbaum, 1964-66) • Simulates a Rogerian psychotherapist (Person-Centered Therapy) E> Hi, I’m Eliza. What problem do you want to talk about? U> Let’s talk about the SOFSEM conference E> What is your problem with the SOFSEM conference? U> No problem, really E> Why not? U> I just wanted to chat about the conference E> Is that the real reason? U> Yes E> I understand Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  7. ELIZA (Weizenbaum, 1964-66) • Simulates a Rogerian psychotherapist (Person-Centered Therapy) U> So what E> Have you asked such questions before? U> About a zillion times E> Can you elaborate on that? U> Ok, let’s talk about the Turing session of the conference E> Does that question interest you? U> Yes, tremendously, I’m here for giving an invited talk! E> Are you sure? Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  8. ELIZA (Weizenbaum, 1964-66) • An early example of computer program performing primitive natural language processing • Processes human replies to questions • Uses simple template matching by inserting keywords into template phrases • No semantics, really! Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  9. Shortcut #1: Flatter the user (AKA: Template Matching) Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  10. Mgonz (Humphrys, 1989) • Memoryless is good in a random conversation! U> finger M> cut this cryptic s**t speak in full sentences U> well hello M> ok thats it im not talking to you any more U> mark? M> mark isnt here and has left me to deal with cretins like you U> hello mark are you there? U> I can't tell if this thing is working. M> ah type something interesting or shut up Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  11. Shortcut #2: Insult the user (AKA: “type something interesting or shut up”) Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  12. Implementing the Turing Test: the Loebner Prize • Annual competition in Artificial Intelligence • Round: a judge converses with a human and a machine and decides which is which • Time limit: 5-20+ minutes • Risk: a winning system might aim at trickery rather than plausible intelligence Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  13. These systems do not really UNDERSTAND! • JUDGE> I liked "Betty Blue" very much. I like movies about crazy people. • PROGRAM> Like wow. I'm so flattered. I saw "The City of Lost Children" recently. Have you heard of it? • You just need a database of movies… Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  14. These systems do not really UNDERSTAND! • JUDGE> Yes, but I haven't seen it. Why did you say you were "so flattered"? • PROGRAM> OK. That's good... Does there have to be a reason? • Reminiscent of ELIZA, maybe? Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  15. These systems do not really UNDERSTAND! • JUDGE> Oh, I suppose not. Say anything which will convince me you are not a machine • PROGRAM> Wow - you suppose not? Me too! Listen. I don't know which convince you I am a machine will not, and I don't wish to be told either! So, have you ever visited Australia? • Alright, better than ELIZA at changing the subject when on a difficult topic • But linguistically challenged! Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  16. These systems do not really UNDERSTAND! • JUDGE> are you from australia? • PROGRAM> It's cool being from Australia I guess. Gives you something to chat about when talking with a Yank! • JUDGE> I am unsure of that reply--are you from australia? • PROGRAM> It's cool being from Australia I guess. Gives you something to chat about when talking with a Yank! • Oooops, a little crack in the system? Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  17. Let’s get SERIOUS! • JUDGE> I bought pens for my cattlein Perth, Australia. What do you think about that? • PROGRAM1> Check back later and see if I learn the answer to that one. • Oooops, a larger crack in the system? Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  18. Let’s get SERIOUS! • JUDGE> I bought pens for my cattlein Perth, Australia. What do you think about that? • PROGRAM2> Hey, are theywriting the next best seller? • Nope, it doesn’t work… Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  19. Shortcut #3 (generalizes shortcuts #1 and #2): Keep away from Natural Language Processing (AKA: “the intelligent parrot approach to the Turing Test”) Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  20. What knowledge does HAL 9000 need? • HAL (Heuristically programmed ALgorithmic computer) Dave Bowman: Hello, HAL. Do you read me, HAL? HAL: Affirmative, Dave. I read you. Dave Bowman: Open the pod bay doors, HAL. HAL: I'm sorry, Dave. I'm afraid I can't do that. Dave Bowman: What's the problem? HAL: I think you know what the problem is just as well as I do. Dave Bowman: What are you talking about, HAL? HAL: This mission is too important for me to allow you to jeopardize it. Dave Bowman: I don't know what you're talking about, HAL. syntax phonetics & phonology ? semantics discourse pragmatics morphology Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  21. So what can we (researchers?) do about automatic text understanding? Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  22. Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  23. The (Commercial) State of the Art Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli How to translate: “I like chocolate, so I bought a bar in a supermarket”? Google Translate: “Mi piace il cioccolato, così ho comprato un barin un supermercato”!

  24. Shortcut #4: Just use statistics (AKA: “tons of data will tell you”) Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  25. It’s not that the “big data” approach is bad, it’s just that mere statistics is not enough Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  26. Symbol vs. Concept Source: Little Britain. Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli 26 11/09/2014

  27. Symbol vs. Concept 粉塵 Source: Little Britain. Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli 27 11/09/2014

  28. Word Sense Disambiguation (WSD) • The task of computationally determining the senses (meanings) for words in context • “Springwater can be found at different altitudes” Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  29. Word Sense Disambiguation (WSD) • The task of computationally determining the senses (meanings) for words in context • “Springwater can be found at different altitudes” Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  30. Word Sense Disambiguation (WSD) • It is basically a classification task • The objective is to learn how to associate word senses with words in context • This task is strongly linked to Machine Learning • I like chocolate, so I bought a bar in a supermarket , , , … , bar = Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  31. Word Sense Disambiguation (WSD) bar (target word) “I like chocolate, so I bought a bar in a supermarket” (context) at the bar? no yes Corpora WSD system bar … court? coffee … bar? Lexical Knowledge Resources no no yes yes bar/counter bar/law bar/block resources output sense Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  32. Lexical sample vs. All words • Lexical sample • A system is asked to disambiguate a restricted set of words (typically one word per sentence): • We are fond of fruit such as kiwi and banana. • The kiwi is the national bird of New Zealand. • ... a perspective from a nativekiwi speaker (NZ). • All words • A system is asked to disambiguate all content words in a data set: • The kiwiis the national bird of New Zealand. • ... a perspective from a nativekiwispeaker (NZ). Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  33. But… is WSD any good? Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  34. But… is WSD any good? • Back in 2004 (Senseval-3, ACL): Many instances of just a few words State of the art: 65% accuracy Few instances of many words Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  35. What does “65% accuracy” mean? • You understand 65 (content) words out of 100 in a text: • You might behave the wrong way! • For instance: • I was arrestedafter mytherapistsuggestedI takesomething for my kleptomania. I usedto be a banker, but lostinterestin the work. Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  36. And you need a lot of training data! • Dozens (hundreds?) of person-years, at a throughput of one word instance per minute [Edmonds, 2000] • For each language of interest! Source: [Iraolak, 2004] NOT FEASIBLE Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  37. Why is WSD so difficult? “For three days and nights she danced in the streets with the crowd.” WordNet [Miller et al. 1990; Fellbaum, 1998]: • street1 - a thoroughfare (usually including sidewalks) that is lined with buildings • street2 - the part of a thoroughfare between the sidewalks; the part of the thoroughfare on which vehicles travel • street3 - the streets of a city viewed as a depressed environment in which there is poverty and crime and prostitution and dereliction • street5 - people living or working on the same street NIGHTMARE! Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  38. Inter-Tagger Agreement using WordNet • On average human sense taggers agreeon their annotations 67-72% of the time • Question 1: So why should we expect an automatic system to be able to do any better than us?! • Question 2: What can we do to improve this frustrating scenario? Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  39. From Fine-Grained to Coarse-Grained Word Senses • We need to remove unnecessarily fine-grained sense distinctions People living/working on the street Street without sidewalks Street with sidewalks Depressed environment Street with/without sidewalks? Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  40. From Fine-Grained to Coarse-Grained Word Senses (1) • Coarser sense inventories can be created manually [Hovy et al., NAACL 2006]: • Start with the WordNet sense inventory of a target word • Iteratively partition its senses • Until 90% agreement is reached among human annotators in a sense annotation task Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  41. From Fine-Grained to Coarse-Grained Word Senses (2) • Coarser sense inventories can be created automatically using WSD [Navigli, ACL 2006]: flat list of senses hierarchy of senses subsenses polysemous senses homonyms Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  42. Is coarse-grained better than fine-grained? • Let’s move forward to 2007 (Semeval-2007, ACL): training data training data Most Frequent Sense Baseline Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  43. The Most Frequent Sense (MFS) Baseline • Given an ambiguous word, simply always choose the most common (i.e., most frequently occurring) sense within a sense-annotated corpus Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  44. The Most Frequent Sense (MFS) Baseline • Given an ambiguous word, simply always choose the most common (i.e., most frequently occurring) sense within a sense-annotated corpus • High performance on open text (the predominant sense is most likely to be correct) • Looks like a pretty lazy way to tackle ambiguity! Sense #1 Sense #1 Sense #1 Sense #1 Sense #1 Sense #1 Sense #1 Sense #1 Sense #1 Sense #1 Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  45. But, hey, just think of the Web! There are at least 8 billion Web pages out there! On average each page contains ~500 words (2006) Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  46. MFS on 8G * 500 words = 4T words... • At 78.9% performance, it makes 21.1% of errors • So it chooses the wrong sense for 844G words (out of 4T) • Can we afford that? • So what’s the output on: • JUDGE> I bought pens for my cattlein Perth, Australia. What do you think about that? • PROGRAM2> Hey, are theywriting the next best seller? Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  47. Shortcut #5: Most Frequent Sense (AKA: “lazy about senses”) Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  48. Coarse-grained Word Sense Disambiguation[Navigli et al. 2007; Pradhan et al. 2007] • Semeval-2007, ACL: 83.2 SSI KB training data training data State of the art: 82-83%! large knowledge base Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  49. Knowledge-based WSD: the core idea • We view WordNet (or any other knowledge base) as a graph • We exploit the relational structure (i.e., edges) of the graph to perform WSD Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

  50. WordNet (Miller et al. 1990; Fellbaum, 1998) Don’t Take Shortcuts! Computational Lexical Semantics and the Turing Test Roberto Navigli

More Related