Download
1 / 19
190 likes | 479 Views
Aiding WSD by exploiting hypo/hypernymy relations in a restricted framework. MEANING project Experiment 6.H(d) Luis Villarejo and Llu í s M à rquez. Preface.
E N D
Aiding WSDby exploiting hypo/hypernymy relationsin arestricted framework MEANING projectExperiment 6.H(d)Luis Villarejo and Lluís Màrquez
Preface This document stands for the 1st draft of the description of Experiment 6.H(d): “Aiding WSDby exploiting hypo/hypernymy relationsin arestricted framework”. To be included in the Working Paper describing experiment 6.H: Bootstrapping.
Outline • Introduction • Our approach vs Mihalcea’s • Technical details • Experiments • Results • Conclusions • Future work
Introduction • “The Role of Non-Ambiguous Words in Natural Language Disambiguation” • Rada Mihalcea • University of North Texas • Task: Automatic resolution of ambiguity in NL. • Problem: The lack of large amounts of annotated data. • Proposal: Inducing knowledge from non-ambiguous words via equivalence classes to automatically build an annotated corpus.
Introduction In this experiment we explore whether training example sets can be enlarged with automatically extracted examples associated to each sense. Some work has been recently done in the direction of extracting examples, in a non/supervised manner, associated to word senses like the one presented by Mihalcea. R. in The Role of Non-Ambiguous Words in Natural Language Disambiguation where POS tagging, Named entity tagging and WSD were approached. We will only tackle here WSD. We not only did our best to reproduce the conditions, in which experiments were developed, described in the paper by Mihalcea. R. but also explored new possibilities not taken into account in the paper. However, our scenario is better since our unique source for obtaining the extra training examples is SemCor, and therefore, we do not need to perform any kind of disambiguation. The semantic relations, used to acquire the target words for the extra examples, were taken from the MCR.
Introduction • Synonyms ? • Hyperonyms ? • Hyponyms ? • Holonyms ? • Meronyms ? • Focuses on: • Part of Speech Tagging • Named Entity Tagging • Word Sense Disambiguation WordNet 1.6 “Equivalence classes consist of words semanticallyrelated.” What happens with the other two meanings of plant?? (actor and trick)
Our approach vs Mihalcea’s • Manually selected word set: • Our list: child, day, find, keep, live, material, play and serve. • Rada’s: bass, crane, motion, palm, plant and tank. • Only two, specially selected, and clearly differentiated senses per word: • {play 00727813}: play games, play sports “Tom plays tennis” • {play 01177316}: play a role or part “Gielgud played Hamlet” • Source for examples on the equivalent words: • Relation of equivalence between words: • Equivalent corpus sizes.
Our approach vs Mihalcea’s • Features used: • Technique used to learn: • Use of the examples coming from the equivalent words:
Technical details • SVM_light (Joachims): • Each word is a binary classification problem which has to decide between two possible labels (senses). • Positive examples of one sense are negative for the other. • 10-fold cross validation • Testing with a random folder from the originals. • Training with the rest of the originals plus the equivalents. • C parameter tuning by margin (5 pieces and 2 rounds) for each classification problem. • Linear kernel
Experiments • Baselines: • Examples from Original word set codified with all features. • Examples from Original word set codified only with BOW feature. • Experiments over each Baseline: • Examples from Equivalents codified with all features. • Examples from Equivalents codified only with BOW feature. • Examples from Equivalents added in equal proportions. • Relations explored over each experiment: • Hyponymy levels 1, 2 and both • Hyperonymy levels 1, 2 and both • Synonymy • Mixes: SHypo1, SHypo2, SHypo12, SHype1, SHype2, SHype12 • Total: 8 words * 2 baselines * 3 experim * 13 relations = 624 results
Results – O-all, A-both, keeping proportions II Note: Examples not randomly added.
Results • Other results: • Accuracy improves (slightly) when choosing manually which equivalents to take into account. • Experiments with a set of 41 words (nouns and verbs) with all senses per word (varying from 2 to 20) proved to have worse results (accuracies on all mixes of relations and features are below the baseline).
Conclusions The work presented by R. Mihalcea has some dark points. The criteria used to select the words, involved in the experiment, the criteria to select word senses, the restriction on the number of senses per word, the semantic relations used to get the monosemous equivalents or the features used to learn are not satisfactorily described. Although this, Mihalcea’s results showed that equivalents carry useful information to do WSD (better than MFS 76.60% against 70.61%). But is this information useful to improve the state-of-the-art in WSD?. Experiments carried out here showed that adding examples coming from the equivalents seems to improve the results in a restricted framework. This means using a small word set, only two senses per word and clearly differentiated senses. When we moved to an open framework, this means using a bigger word set is used (41 words), no special selection of words, no special selection of senses and no restriction on the number of senses per word (varying from 2 to 20), results proved to be worse.
Future Work • Do the differences between the features set used by Mihalcea and the one we used critically affected the results on the 41 words experiment? • Exploiting the feature extractor over SemCor to enrich the feature set used. Ideas are welcome. • Study the correlation between the number of examples added and the accuracy obtained. • Restrict the addition of examples coming from the equivalent words (second class examples). • Randomly select which examples to add when keeping proportions or restricting the addition.
