Computational Lexical Semantics

1. Computational Lexical Semantics Om Damani, IIT Bombay

2. Study of Word Meaning • Word Sense Disambiguation • Word Similarity • WordNet Relations • Do we really know the meaning of meaning • We will just take the dictionary definition as meaning

3. Word Sense Disambiguation (WSD) WSD Applications: Search, _____, ______

4. Sense Inventory • Wordnet, Dictionary etc. • Plant in English Wordnet (#senses ??): • Noun Senses: • plant, works, industrial plant (buildings for carrying on industrial labor) "they built a large plant to manufacture automobiles" • plant, flora, plant life ((botany) a living organism lacking the power of locomotion) • plant (an actor situated in the audience whose acting is rehearsed but seems spontaneous to the audience) • plant (something planted secretly for discovery by another) "the police used a plant to trick the thieves"; "he claimed that the evidence against him was a plant"

5. Sense Inventory .. • Plant (Verb Senses): • plant, set (put or set (seeds, seedlings, or plants) into the ground) "Let's plant flowers in the garden" • implant, engraft, embed, imbed, plant (fix or set securely or deeply) "He planted a knee in the back of his opponent"; "The dentist implanted a tooth in the gum" • establish, found, plant, constitute, institute (set up or lay the groundwork for) "establish a new department" • plant (place into a river) "plant fish" • plant (place something or someone in a certain position in order to secretly observe or deceive) "Plant a spy in Moscow"; "plant bugs in the dissident's apartment" • plant, implant (put firmly in the mind) "Plant a thought in the students' minds"

6. How many Senses of सच्चा • Noun: सत्यवादी, सच्चा, सत्यभाषी, सत्यवक्ता - वह जो सत्य बोलता हो "आधुनिक समाज में भी सत्यवादियों की कमी नहीं है / यथार्थवादी होने के कारण कई लोग श्याम के दुश्मन बन गए हैं" • Adjective(6) • सत्यवादी, सच्चा, सत्यभाषी, सत्यवक्ता - जो सत्य बोलता हो "युधिष्ठिर एक सत्यवादी व्यक्ति थे" • ईमानदार, छलहीन, निष्कपट, निःकपट, रिजु, ऋजु, दयानतदार, सच्चा, अपैशुन, सत्यपर - चित्त में सद्वृत्ति या अच्छी नीयत रखनेवाला, चोरी या छल-कपट न करनेवाला "ईमानदार व्यक्ति सम्मान का पात्र होता है" • वास्तविक, यथार्थ, सच्चा, सही, असली, वास्तव, अकाल्पनिक, अकल्पित, अकूट, प्रकृत - जो वास्तव में हो या हुआ हो या बिल्कुल ठीक "मैंने अभी-अभी एक अविश्वसनीय पर वास्तविक घटना सुनी है" • सच्चा, असली - जो झूठा या बनावटी न हो "वह भारत माँ का सच्चा सपूतहै" • खरा, चोखा, सच्चा - जो ईमानदारी, निष्पक्षता, न्याय आदि के आधार पर हो "हमें खरा सौदा करना चाहिए" • खरा, सच्चा, सीधा - बिना किसी बहाने या समझौता के यानि सीधा "वह इतना खरा नहीं है जितना दिखाता है“ • How do you know these are different senses • Hint: think translation

7. How many Senses of आदमी • आदमी, पुरुष, मर्द, नर - नर जाति का मनुष्य "आदमी और औरत की शारीरिक संरचनाएँ भिन्न होती हैं" • मानव, आदमी, इंसान, इन्सान, इनसान, मनुष्य, मानुष, मानुस, मनुष, नर - वह द्विपद प्राणी जो अपने बुद्धिबल के कारण सब प्राणियों में श्रेष्ठ है और जिसके अंतर्गत हम,आप और सब लोग हैं " आदमी अपनी बुद्धि के कारण सभी प्राणियों में श्रेष्ठ है" • व्यक्ति, मानस, आदमी, शख़्स, शख्स, जन, बंदा, बन्दा - मनुष्य जाति या समूह में से कोई एक "इस कार में दो ही आदमी बैठ सकते हैं" • नौकर, सेवक, दास, अनुचर, ख़ादिम, मुलाज़िम, मुलाजिम, आदमी, टहलुआ, पार्षद, लौंडा, अनुग, अनुचारक, अनुचारी, अनुयायी, पाबंद, पाबन्द, नफर, अभिचर, भृत्य, गण, अभिसर, अभिसारी - वह जो सेवा करता हो "मेरा आदमी एक हफ्ते के लिए घर गया है" • पति, मर्द, शौहर, घरवाला, मियाँ, आदमी, ख़सम, खसम, स्वामी, अधीश, नाथ, कांत, कंत, परिणेता, वारयिता, दयित - स्त्री की दृष्टि से उसका विवाहित पुरुष "शीला का आदमी किसानी करके परिवार का पालन-पोषण करता है“ • How do you know these are different senses • Hint: think translation

8. WSD: Problem Statement • Given a string of words (sentence, phrase, set of key-words), and a set of senses for each word, decide the appropriate sense for each word. • Example: Translate ‘Where can I get spare parts for textile plant ?’ to Hindi

9. Solution Approaches • Solution depends on what resources do you have: • Definition, Gloss • Topic/Category label for each sense definition • Selectional preference for each sense • Sense Marked Corpora • Parallel Sense-Marked Corpora

10. Combinatorial Explosion Problem • I saw a man who is 98 years old and can still walk and tell jokes • See(26), man(11), year(4), old(8), can(5). Still(4), walk(10), tell(8), joke(3). • 4,39,29,600 sense combinations • Solution: Viterbi ??

11. Dictionary-Based WSD

12. Dictionary-Based WSD The bank did not give loan to him though he offered to mortgage his boat. Thebank did not give loan to him though he offered to mortgage his boat.

13. How to improve the LESK further • Give an example where the algo fails – say for bank • “Thebank did not give loan to him though he offered his boat as collateral.” • Problem: collateral is related to the bank but the relation does not come out clearly • Solution: See if the definition of bank and definition of collateral share a term: • Collateral: security pledged for loan repayment • Problem: Can you give an example where the new algorithm fails too

14. LESK Algorithm Function Lesk (word, sentence) returns best sense of word context := set of words in sentence; for each sense insensesof word do sense.signature := GetSignature (sense); sense.relevance := ComputeRelevance ( sense.signature, context ); end best-sense := MaxRelevantSense () ; if ( best-sense.relevance == 0 ) best-sense :=GetDefaultSense (word); return best-sense; GetSignature ( sense ): Get all words in example and gloss of sense ComputeRelevance ( signature, context ): number of common words

15. GetSignature ( sense) • All words in example and gloss of sense • All words in gloss of sense • All words in gloss of all words in the gloss of the given sense • All words in gloss of all words in gloss of all words in gloss • ….. • Problem: • Including the right sense of each word in gloss needs WSD • Including all senses of all words in gloss will lead to sense-drift • Possible Solution: All context words in a sense marked corpora

16. Ideal Signature • For each word, get a Vector of all the words in the language • Work with a |V|x|V| Matrix • Iterate over it, till it converges

17. ComputeRelevance( {signature}, {context} ) Signature1: a financial institution that accepts deposits and gives loan Signature2: the slope beside a body of water Context: The bank did not give loan to him though he offered to mortgage his boat number of common words: | signature Intersection context | Favors longer signatures | signature Intersection context | / | signature Union context | Define Relevance between two words Synonyms Specialization, Generalization has to be accounted for – canoe, boat Even more general: credit, money Sum of Relevance between all word pairs Weigh different terms differently – maybe based on TF-IDF score FunctionComputeRelevance( {signature}, {context} ) relevance := 0; for each sig-word in signature do for each con-word in context do wordRelevance := WordRelevance ( sig-word, con-word ); relevance += wordRelevance *weight( sig-word); end end relevance /= Normalize ( signature, context ); return relevance;

18. GetDefaultSense ( word ) • The most frequent sense • The most frequent sense in a given domain • The most frequent sense as per the topic of the document

19. Power of the LESK Schema • Signature can even be a topic/domain code: finance, travel, geology, physics, civics • All variations of ComputeRelevance function are still applicable: Defn: the slope beside a body of water • Signature: geology, physics, geology Defn: a financialinstitution that accepts deposits and gives loan • Signature: finance, civics, finance, finance Sentence: The bank did not give loan to him though he offered to mortgage his boat • Context: finance, finance, travel • Problem: Various senses of a word have different topics • Solution: ??

20. Possible Improvements • LESK gives equal weightage to all senses - ‘right’ sense should be given more weight • Iterative fashion – one at a time – most certain first • Page Rank like algo • Give more weightage to Gloss than to Example in computing relevance

21. Page-Rank-LESK

22. Fundamental Limitation of Dictionary Based Methods • Depends too much on the exact word • Another dictionary may use different gloss and example • Use the context words from a tagged corpus as signature

23. Supervised Learning • Lesk-like methods depend too much on the exact word • Another dictionary may use different gloss and example • Use a sense-tagged corpora • Employ a machine learning algorithm

24. Supervised Learning • Machine can only learn what we ask it to • Collocation feature • Relative position (2 words to the left) • Words and POS • “An electric guitar and bass player stand off to one side, not really part of the scene, ...” • [wi−2,POSi−2,wi−1,POSi−1,wi+1,(20.2) POSi+1,wi+2,POSi+2] • [guitar, NN, and, CC, player, NN, stand, VB] • Bag-of-words feature • [fishing, big, sound, player, fly, rod, pound, double, runs, playing, guitar, band] • [0,0,0,1,0,0,0,0,0,0,1,0]

25. Naïve Bayes Classifier a simple binary bag of words vector defined over a vocabulary of 20 words would have --- possible feature vectors. Still the data sparsity problem remains Assumption: features are conditionally independent given the word sense

26. Computing Naïve Bayes Probabilities if a collocational feature such as [wi.2 = guitar] occurred 3 times for sense bass1, and sense bass1 itself occurred 60 times in training, theMLE estimate is P( f j |s)= 0.05. it’s hard for humans to examine Naïve Bayes’s workings and understand its decisions. Hence use Decision lists

27. Decision Lists Rule ⇒ Sense fish within window ⇒ bass1 striped bass ⇒ bass1 guitar within window ⇒ bass2 bass player ⇒ bass2 piano within window ⇒ bass2 tenor within window ⇒ bass2 sea bass ⇒ bass1

28. How to Create Decision Lists • Which feature has the most discrimination power • Seems same as max P (Sense | f) • Need Decision Trees

29. Selectional Restrictions and Selectional Preferences • “In our house, everybody has a career and none of them includes washing dishes,” he says. • In her kitchen, Ms. Chen works efficiently, cooking several simple dishes. • Wash[+WASHABLE], cook[+EDIBLE] • Used more often for elimination than selection • Problem: Gold-rush fell apart in 1931, perhaps because people realized you can’t eat gold for lunch if you’re hungry. • Solution: Use these preferences as features/probabilities

30. Selectional Preference Strength • eat ⇒ edible. • be ⇒ ?? • Strength: P(c) vs P(c|v) • Kullback-Leibler Divergence (Relative Entropy) • selectional association : contribution of that class to general selectional preference of the verb

31. Selection Association • a probabilistic measure of the strength of association between a predicate and a class dominating the argument to the predicate • Verb, Semantic Class, Assoc, Semantic Class, Assoc • read WRITING 6.80 ACTIVITY -.20 • write WRITING 7.26 COMMERCE 0 • see ENTITY 5.79 METHOD -0.01

32. How do we use Selection Association for WSD • Use as a relatedness model • select the sense with highest selectional association between one of its ancestor hypernyms and the predicate.

33. Minimally Supervised WSD • Supervised: needs sense tagged corpora • Dictionary based: needs large examples and gloss • Supervised approaches do better but are much more expensive • Can we get best of both words

34. Bootstrapping • Seed-set L0 of labeled instances, a much larger unlabeled corpus V0 • Train a decision-list classifier on seed-set L0 • Uses this classifier to label the corpus V0 • Add to the training set examples in V0 that it is confident about • Iterate { retrain decision-list classifier }

35. Bootstrapping Success Depends On • Choosing the initial seed-set • One sense per collocation • One sense per discourse • Samples of bass sentences extracted from the WSJ using the simple correlates play and fish. • We need more good teachers – right now, there are only a half a dozen who can play the free bass with ease. • And it all started when fishermen decided the striped bass in Lake Mead were too skinny. • Choosing the ‘confidence’ criterion

36. WSD: Summary • It is a hard problem • In part because it is not a well-defined problem • Or it cannot be well-defined • Because making sense of ‘Sense’ is hard

37. Hindi Wordnet • Wordnet - A lexical database • Inspired by the English WordNet • Built conceptually • Synset (synonym set) is the basic building block.

38. Entry in Hindi Wordnet • Synset {गाय,गऊ, गैया, धेनु} {gaaya ,gauu, gaiyaa, dhenu}, Cow • Gloss • Text definition सींगवाला एक शाकाहारी मादा चौपाया (siingwaalaa eka shaakaahaarii maadaa choupaayaa) (a horny, herbivorous, four-legged female animal) • Example sentence हिन्दू लोग गाय को गो माता कहते हैं एवं उसकी पूजा करते हैं। (hinduu loga gaaya ko go maataa kahate hain evam usakii puujaa karate hain) (The Hindus considers cow as mother and worship it.)

39. Subgraph for Noun चौपाया,पशु (chaupaayaa, pashu) Four-legged animal शाकाहारी (shaakaahaarii) herbivorous Hypernym पूँछ (puunchh ) Tail गाय, गऊ (gaaya ,gauu) Cow Attribute meronym सींगवाला एक शाकाहारी मादा चौपाया (siingwaalaa eka sakaahaarii maadaa choupaayaa) A horny, herbivorous, four-legged female animal) Gloss थन (thana) udder Hyponym Ability Verb पगुराना( paguraanaa) ruminate Antonym कामधेनु kaamadhenu A kind of cow मैनी गाय mainii gaaya A kind of cow बैल (baila) Ox

40. Subgraph for Verb भावाभिव्यक्ति करना (bhaavaabhivyakti karanaa) to express Hypernym रोना,रुदन करना (ronaa, rudan karanaa) to weep Causative Verb Antonym रुलाना (rulaanaa) to make cry हँसना (hansanaa) to laugh Entailment Gloss Troponym आँसू बहाना (aansuu bahaanaa) to weep सिसकना (sisakanaa) to sob

41. Marathi Wordnet (Noun) वनस्पती रान H Y P E R N Y M Y खोड HOLONYMY M E R O N Y M Y झाड,वृक्ष,तरू बाग मूळ GLOSS H Y P O N Y M Y मुळे,खोड,फांद्या,पाने इत्यादींनी युक्त असा वनस्पतिविशेष:"झाडे पर्यावरण शुद्ध करण्याचे काम करतात" लिंबू आंबा

42. Word Similarity • In Lesk and other algo, we need to measure how related two words are • Simplest measure: pathLength - #edges in shortest path between sense nodes c1 and c2 • sim (c1,c2) = −log pathlen(c1,c2) • wordsim(w1,w2) = max (c1∈senses(w1), c2∈senses(w2)) sim(c1,c2)

43. Path Length: Limitations • All edges are not equal • Compare medium of exchange and standard with coin and nickel • Need a distance measure on edges

44. Information Content Word Similarity LCS(c1,c2) = the lowest common subsumer, i.e., the lowest node in the hierarchy that subsumes (is a hypernym of) both c1 and c2 sim (c1,c2) = −log P(LCS(c1,c2))

45. IC Similarity: Limitations • A concept is not similar to itself using the previous defn • Word similarity is not about Information Contents. It is about commanality vs differences:

46. Overlap based Similarity • Previous methods may not work for words belonging to different classes: car and petrol • similarity(A,B) = overlap(gloss(A), gloss(B)) + overlap(gloss(hypo(A)), gloss(hypo(B)))+ overlap(gloss(A), gloss(hypo(B))) + overlap(gloss(hypo(A)),gloss(B))

47. WORD SIMILARITY: DISTRIBUTIONAL METHODSpointwise mutual information

48. Similarity using Feature Vectors

49. Cosine Distance Dot product favors long vectors

50. Conclusion • Lot of care is needed in defining similarity measures • Impressive results can be obtained once similarity is carefully defined