1 / 21

RTE2 2006 April 10, 2006

RTE2 2006 April 10, 2006. An approach based on Logic Forms and WordNet relationships to Textual Entailment performance. O. Ferrández, R. M. Terol, R. Muñoz, P. Martínez-Barco and M. Palomar {ofe,rafamt,rafael,patricio,mpalomar}@dlsi.ua.es

mirit
Download Presentation

RTE2 2006 April 10, 2006

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. RTE2 2006 April 10, 2006 An approach based on Logic Forms and WordNet relationships to Textual Entailment performance O. Ferrández, R. M. Terol, R. Muñoz, P. Martínez-Barco and M. Palomar {ofe,rafamt,rafael,patricio,mpalomar}@dlsi.ua.es GPLSI – Natural Language Processing and Information Systems Group

  2. Index • System Architecture • Derivation of the Logic Forms • Computation of Similarity Measures between Logic Forms • Result Analysis • Conclusions and Future Work

  3. System Architecture Text Hypothesis Derivation of the Logic Forms LF Text LF Hypothesis Computation of similarity measures between Logic Forms score Entailment? YES NO

  4. Derivation of the Logic Forms • The Logic Forms are derived through an analysis of dependency relationships between the words of the sentence • Employs a set of rules that infer several aspects such as the assert, its type, its identifier and the relationships between the different asserts in the logic form • The Logic Forms are based on the logic form format defined in the eXtended WordNet

  5. Derivation of the Logic Forms • As an example • “A shark attacked a human being” • Dependency tree • Logic Form • shark:NN(x1) attack:VB(e1,x1,x3) • human:NN(x2) NNC(x3,x2,x4) being:NN(x4) attack:V S obj shark:N being:N det Lex-mod det a:Det a:Det human:U

  6. Computation of Similarity Measures between LF • The method • Focused on the entailment between the verbs (verbs generally govern the meaning of sentences) • Firstly analyses the relation between the verbs of the two logic forms derived from the text and the hypothesis • Secondly, if there is a relation between the verbs, then the method will analyse the similarity relations between all predicates which depending on the two verbs

  7. Computation of Similarity Measures between LF simWeight = 0 Tvb = obtainVerbs(T) Hvb = obtainVerbs(H) for i = 0 ... size(Tvb) do for j = 0 ... size(Hvb) do if calcSim(Tvb(i),Hvb(j)) >0 then simWeight += calcSim(Tvb(i),Hvb(j)) Telem = obtainElem(Tvb(i)) Helem = obtainElem(Hvb(j)) simWeight += calcSim(Telem,Helem) end if end for end for if simWeight > threshold then return TRUE else return FALSE end if

  8. Computation of Similarity Measures between LF • In order to obtain the similarity between the predicates of the logic forms (calcSim(x,y)), two approaches have been implemented • Based on WordNet relations • Based on Lin’s measure • A Word Sense Disambiguation module was not employed • The first 50% of the WordNet senses were taken into account • The threshold, which determines if the text entails the hypothesis, has been obtained empirically using the development data

  9. Computation of Similarity Measures between LF • Based on WordNet relations • For which concept (word#sense): • Obtaining the relations among other concepts through the synsets • Each relation has an associated weight • Synonymy (0.9), Hypernymy (0.8), Hyponymy and Entailment (0.7), Meronymy and Holonymy (0.5) • The length of the path that relates the two different concepts must be lower or equal than 4 synsets

  10. Computation of Similarity Measures between LF • Based on WordNet relations • The weight of the path between two different concepts is calculated as the product of the weights associated to the relations connecting the intermediate synsets • This weight indicates the relation between two concepts

  11. Computation of Similarity Measures between LF • Based on WordNet relations • Example • cable_car#n#3 subway#n#3  0.5*0.8*0.7=0.28 railway#n#3 cable_car#n#1 Hyponymy Hypernymy 0.5 0.7 holonymy 0.8 subway#n#3 funicular#n#3

  12. Computation of Similarity Measures between LF • Based on Lin’s measure • Lin’s similarity measure as implemented in WordNet::Similarity • Open source software package • Lin’s similarity measure augments the information content of the least common subsumer (LCS is the most specific concept that two concepts share as an ancestor) of the two concepts with the sum of the information content of the concepts

  13. An example • T: Five US soldiers were killed in the capital and insurgents blasted polling stations across the country and:CC kill:VB blast:VB in:IN NNC NNC insurgent:NN Ø NNC five:NN capital:NN polling:U ... soldier:NN us:NN

  14. kill:VB NNC Ø NNC five:NN us:NN soldier:NN An example • H: Five US soldiers were killed

  15. kill and NNC Ø kill blast NNC five in NNC NNC insurgents us soldier Ø NNC five capital polling ... soldier us An example Verbs of H  kill Verbs of T  kill, blast kill#v kill#v  1 Relation ~ 1 kill#v blast#v  0,34 ... Accumulating weights > Threshold  ENTAILMENT

  16. Result Analysis • Results (RTE2 dev&test) • Run1 (Lin’s measure) • Development data • Accuracy: 0.5462 • Test data • Accuracy: 0.5563 • Average Precision: 0.6089 • Run2 (WN relations) • Development data • Accuracy: 0.5273 • Test data • Accuracy: 0.5475 • Average Precision: 0.5743

  17. Result Analysis • The empirical threshold of the development data • A value of 0.24

  18. Conclusions and Future Work • Our system derives the logic forms for the text/hypothesis pair and computes the similarity between them • The similarity is computed using two different approaches: • Lin’s similarity measure • WordNet relation-based similarity • Our system provides a score showing the semantic similarity between two logic forms

  19. Conclusions and Future Work • The run using Lin’s similarity measure achieves better results than the approach based onWordNet relations, both when tested on development, as well as test data • This slight loss of accuracy is due to the fact that our WordNet relations approach attempts to establish an objective semantic comparison between the logic forms rather than an entailment relation

  20. Conclusions and Future Work • As a future work: • Performing a deeper study about the most suitable WordNet relations for recognising textual entailment. Perhaps only hypernymy, synonymy and entailment relations between the text and the hypothesis would be more suitable for the entailment phenomenon • Testing how other natural language processing tools can help in detecting textual entailment. For example, using a Named Entity Recognizer could help in detecting entailment between two segment of text

  21. RTE2 2006 April 10, 2006 Thank you very much O. Ferrández, R. M. Terol, R. Muñoz, P. Martínez-Barco and M. Palomar {ofe,rafamt,rafael,patricio,mpalomar}@dlsi.ua.es GPLSI – Natural Language Processing and Information Systems Group

More Related