Toward dependency path based entailment
This presentation is the property of its rightful owner.
Sponsored Links
1 / 20

Toward Dependency Path based Entailment PowerPoint PPT Presentation


  • 55 Views
  • Uploaded on
  • Presentation posted in: General

Toward Dependency Path based Entailment. Rodney Nielsen, Wayne Ward, and James Martin. Why Entailment. Intelligent Tutoring Systems Student Interaction Analysis Are all aspects of the student’s answer entailed by the text and the gold standard answer

Download Presentation

Toward Dependency Path based Entailment

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Toward dependency path based entailment

Toward Dependency Path based Entailment

Rodney Nielsen, Wayne Ward, and James Martin


Why entailment

Why Entailment

  • Intelligent Tutoring Systems

  • Student Interaction Analysis

    • Are all aspects of the student’s answer entailed by the text and the gold standard answer

    • Are all aspects of the desired answer entailed by the student’s response


Dependency path based entailment

Dependency Path-based Entailment

  • DIRT (Lin and Pantel, 2001)

    • Unsupervised method to discover inference rules

      • “X is author of Y ≈ X wrote Y”

      • “X solved Y ≈ X found a solution to Y”

    • Based on Harris’ Distributional Hypothesis

      • words occurring in the same contexts tend to be similar

    • If two dependency paths tend to link the same sets of words, they hypothesize that their meanings are similar


Ml classification approach

ML Classification Approach

Dependency Path

Based Entailment

  • Features derived from corpus statistics

    • Unigram co-occurrence

    • Surface form bigram co-occurrence

    • Dependency-derived bigram co-occurrence

  • Mixture of experts:

    • About 18 ML classifiers from Weka toolkit

    • Classify by majority vote or average probability

Bag of Words

Graph Matching


Corpora

Corpora

  • 7.4M articles, 2.5B words, 347 words/doc

    • Gigaword (Graff, 2003) – 77% of documents

    • Reuters Corpus (Lewis et al., 2004)

    • TIPSTER

  • Lucene IR engine

    • Two indices

      • Word surface form

      • Porter stem filter

    • Stop words = {a, an, the}


Word alignment features

Word Alignment Features

  • Unigram word alignment


Core features

Core Features


Word alignment features1

Word Alignment Features

  • Bigram word alignment

  • Example:

    • <t>Newspapers choke on rising paper costs and falling revenue.</t><h>The cost of paper is rising.</h>

    • MLE(cost, t) = ncost of, costs of /ncosts of = 6086/35800 = 0.17


Dependency features

Hypothesis h

Text t

rising

choke

cost

is

Newspapers

on

The

of

costs

paper

rising

paper

and

revenues

falling

Dependency Features

  • Dependency bigram features


Dependency features1

Hypothesis h

Text t

rising

choke

cost

is

Newspapers

on

The

of

costs

paper

rising

paper

and

revenues

falling

Dependency Features

  • Descendent relation statistics


Dependency features2

Hypothesis h

Text t

rising

choke

cost

is

Newspapers

on

of

The

costs

paper

rising

paper

and

revenues

falling

Dependency Features

  • Descendent relation statistics


Dependency features3

Hypothesis h

Text t

rising

choke

cost

is

Newspapers

on

The

of

costs

paper

rising

paper

and

revenues

falling

Dependency Features

  • Descendent relation statistics


Dependency features4

Hypothesis h

Text t

rising

choke

cost

is

Newspapers

on

The

of

costs

paper

rising

paper

and

revenues

falling

Dependency Features

  • Descendent relation statistics


Verb dependency features

Hypothesis h

Text t

rising

choke

cost

is

Newspapers

on

The

of

costs

paper

rising

paper

and

revenues

falling

Verb Dependency Features

  • Combined verb descendent relation features

  • Worst verb descendent relation features


Subject dependency features

Hypothesis h

Text t

rising

choke

cost

is

Newspapers

on

The

of

costs

paper

rising

paper

and

revenues

falling

SubjectDependencyFeatures

  • Combined and worst subject descendent relations

  • Combined and worst subject-to-verb paths


Other dependency features

Other Dependency Features

  • Repeat these same features for:

    • Object

    • pcomp-n

    • Other descendent relations


Results

Results


Feature analysis

Feature Analysis

  • All feature sets are contributing according to cross validation on the training set

  • Most significant feature set:

    • Unigram stem based word alignment

  • Most significant core repeated feature:

    • Average Probability


Conclusions

Conclusions

  • While our current dependency path features are only a step in the direction of our proposed inference system, they provided a significant improvement over the best results from the first PASCAL Recognizing Textual Entailment challenge (RTE1)

  • Our system (after fixing a couple of bugs) ranked 6th in accuracy and 4th in average precision out of 23 entrants at this year’s RTE2 challenge

  • We believe our proposed system will provide an effective foundation for the detailed assessment of students’ responses to an intelligent tutor


Questions

choke

Newspapers

on

rising

costs

cost

is

rising

paper

and

revenues

The

of

falling

paper

Questions

Dependency Path

Based Entailment

  • Mixture of experts classifier using corpus co-occurrence statistics

  • Moving in the direction of DIRT

  • Domain of Interest: Student response analysis in intelligent tutoring systems

Bag of Words

Graph Matching

Hypothesis h

Text t


  • Login