Containment exclusion and implicativity a model of natural logic for textual inference
Download
1 / 50

Containment - PowerPoint PPT Presentation


  • 202 Views
  • Updated On :

Containment, Exclusion, and Implicativity: A Model of Natural Logic for Textual Inference. Bill MacCartney and Christopher D. Manning NLP Group Stanford University 14 February 2008.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Containment' - mike_john


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Containment exclusion and implicativity a model of natural logic for textual inference l.jpg

Containment, Exclusion, and Implicativity:A Model of Natural Logic for Textual Inference

Bill MacCartney and Christopher D. Manning

NLP Group

Stanford University

14 February 2008


The textual inference task l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

The textual inference task

  • Does premise P justify an inference to hypothesis H?

    • An informal, intuitive notion of inference: not strict logic

    • Focus on local inference steps, not long chains of deduction

  • Robust, accurate textual inference could enable:

    • Question answering [Harabagiu & Hickl 06]

    • Semantic search

    • Customer email response

    • Relation extraction (database building)

    • Document summarization

  • A limited selection of data sets: RTE, FraCaS, KBE


Some simple inferences l.jpg

OK

No western state completely forbids casino gambling.

No state completely forbids gambling.

Few or no states completely forbid casino gambling.

No

No state completely forbids casino gambling for kids.

No state restricts gambling.

No state or city completely forbids casino gambling.

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Some simple inferences

No state completely forbids casino gambling.

What kind of textual inference system could predict this?


Textual inference a spectrum of approaches l.jpg

Bos & Markert 2006

FOL &theoremproving

semantic

graph

matching

Hickl et al. 2006

MacCartney et al. 2006

Burchardt & Frank 2006

patternedrelationextraction

Romano et al. 2006

lexical/semanticoverlap

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Jijkoun & de Rijke 2005

Textual inference:a spectrum of approaches

deep,but brittle

naturallogic

robust,but shallow


What is natural logic l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

What is natural logic?

  • (natural logic  natural deduction)

  • Lakoff (1970) defines natural logic as a goal (not a system)

    • to characterize valid patterns of reasoning via surface forms (syntactic forms as close as possible to natural language)

    • without translation to formal notation:      

  • A long history

    • traditional logic: Aristotle’s syllogisms, scholastics, Leibniz, …

    • van Benthem & Sánchez Valencia (1986-91): monotonicity calculus

  • Precise, yet sidesteps difficulties of translating to FOL:

    idioms, intensionality and propositional attitudes, modalities, indexicals, reciprocals,scope ambiguities, quantifiers such as most, reciprocals, anaphoric adjectives, temporal and causal relations, aspect, unselective quantifiers, adverbs of quantification, donkey sentences, generic determiners, …


Monotonicity calculus s nchez valencia 1991 l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

+

+

+

+

+

Monotonicity calculus (Sánchez Valencia 1991)

  • Entailment as semantic containment:

    rat< rodent, eat< consume, this morning< today, most< some

  • Monotonicity classes for semantic functions

    • Upward monotone: some rats dream< some rodents dream

    • Downward monotone: no rats dream > no rodents dream

    • Non-monotone: most rats dream # most rodents dream

  • Handles even nested inversions of monotonicity

    Every state forbids shooting game without a hunting license

  • But lacks any representation of exclusion

    Garfield is a cat< Garfield is not a dog


Implicatives factives l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Implicatives & factives

  • Work at PARC, esp. Nairn et al. 2006

  • Explains inversions & nestings of implicatives & factives

    • Ed did not forget to force Dave to leave  Dave left

  • Defines 9 implication signatures

  • “Implication projection algorithm”

    • Bears some resemblance to monotonicity calculus

  • But, fails to connect to containment or monotonicity

    • John refused to danceJohn didn’t tango


Outline l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Outline

  • Introduction

  • Foundations of Natural Logic

  • The NatLog System

  • Experiments with FraCaS

  • Experiments with RTE

  • Conclusion


A theory of natural logic l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

A theory of natural logic

Three elements:

  • an inventory of entailment relations

    • semantic containment relations of Sánchez Valencia

    • plus semantic exclusion relations

  • a concept of projectivity

    • explains entailments compositionally

    • generalizes Sánchez Valencia’s monotonicity classes

    • generalizes Nairn et al.’s implication signatures

  • a weak proof procedure

    • composes entailment relations across chains of edits


Entailment relations in past work l.jpg

Yesentailment

Nonon-entailment

2-wayRTE1,2,3

Yesentailment

Unknownnon-entailment

Nocontradiction

3-wayFraCaS, PARC,RTE4?

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

P = Qequivalence

P < Qforwardentailment

P > Qreverseentailment

P # Qnon-entailment

containmentSánchez-Valencia

Entailment relations in past work


16 elementary set relations l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

16 elementary set relations

Q

Q

P

P

P and Q can representsets of entities (i.e., predicates)or of possible worlds (propositions)cf. Tarski’s relation algebra


16 elementary set relations12 l.jpg

P ^ Q

P _ Q

P > Q

P = Q

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

P < Q

P | Q

P # Q

16 elementary set relations

Q

Q

P

P

P and Q can representsets of entities (i.e., predicates)or of possible worlds (propositions)cf. Tarski’s relation algebra


7 basic entailment relations l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

7 basic entailment relations


Relations for all semantic types l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Relations for all semantic types

The entailment relations are defined for expressions of all semantic types (not just sentences)


Monotonicity of semantic functions l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Monotonicity of semantic functions

Sánchez Valencia’s monotonicity calculus assigns semantic functions to one of three monotonicity classes:

Upward-monotone (M)

The default: “bigger” inputs yield “bigger” outputs

Example: broken. Since chairfurniture, broken chairbroken furniture

Heuristic: in a M context, broadening edits preserve truth

Downward-monotone (M)

Negatives, restrictives, etc.: “bigger” inputs yield “smaller” outputs

Example: doesn’t. While hoverfly, doesn’t flydoesn’t hover

Heuristic: in a M context, narrowing edits preserve truth

Non-monotone (#M)

Superlatives, some quantifiers (most, exactly n): neither M nor M

Example: most. While penguinbird, most penguins # most birds

Heuristic: in a #M context, no edits preserve truth


Downward monotonicity l.jpg

explicit negation:no, n’t

restrictive quantifiers:no, few, at most n

didn’t dance< didn’t tango

at most two people< at most two men

negative & restrictive nouns:ban, absence [of], refusal

negative & restrictive verbs:lack, fail, prohibit, deny

drug ban< heroin ban

prohibit weapons< prohibit guns

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

prepositions & adverbs:without, except, only

the antecedent of a conditional

If stocks rise, we’ll get real paid<If stocks soar, we’ll get real paid

without clothes< without pants

Downward monotonicity

Downward-monotone constructions are widespread!


Generalizing to projectivity l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Generalizing to projectivity

  • How do the entailments of a compound expression depend on the entailments of its parts?

  • How does the entailment relation between (fx) and (fy) depend on the entailment relation between x and y(and the properties of f)?

  • Monotonicity gives partial answer (for =, <, >, #)

  • But what about the other relations (^, |, _)?

  • We’ll categorize semantic functions based on how they project the basic entailment relations


Example projectivity of not l.jpg

downwardmonotonicity

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

swaps

these too

Example: projectivity of not


Example projectivity of refuse l.jpg

downwardmonotonicity

switch

blocks,

not swaps

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Example: projectivity of refuse


Projecting entailment relations upward l.jpg

@

@

>

>

>

@

@

<

<

@

@

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

@

@

nobody

nobody

can

can

without

without

a shirt

clothes

enter

enter

Projecting entailment relations upward

Nobody can enter without a shirt < Nobody can enter without clothes

  • Assume idealized semantic composition trees

  • Propagate lexical entailment relations upward, according to projectivity class of each node on path to root


A weak proof procedure l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

A weak proof procedure

  • Find sequence of edits connecting P and H

    • Insertions, deletions, substitutions, …

  • Determine lexical entailment relation for each edit

    • Substitutions: depends on meaning of substituends

    • Deletions: < by default: red socks < socks

    • But some deletions are special: not hungry ^ hungry

    • Insertions are symmetric to deletions: > by default

  • Project up to find entailment relation across each edit

  • Compose entailment relations across sequence of edits


Composing entailment relations l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Composing entailment relations

  • Relation composition: if a R b and b S c, then a ? C

    • cf. Tarski’s relation algebra

  • Many compositions are intuitive

    = º =  = < º <  < < º =  < ^ º ^  =

  • Some less obvious, but still accessible

    | º ^  < fish | human, human ^ nonhuman,fish < nonhuman

  • But some yield unions of basic entailment relations!

    | º |  {=, <, >, |, #} (i.e. the non-exhaustive relations)

    • Larger unions convey less information (can approx. with #)

    • This limits power of proof procedure described


Implicatives factives23 l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Implicatives & factives

  • Nairn et al. 2006 define nine implication signatures

  • These encode implications (+, –, o) in + and – contexts

    • Refuse has signature –/o:refuse to danceimplies didn’t dancedidn’t refuse to dance implies neither danced nor didn’t dance

  • Signatures generate different relations when deleted

    • Deleting –/o generates |Jim refused to dance | Jim dancedJim didn’t refuse to dance _ Jim didn’t dance

    • Deleting o/– generates <Jim attempted to dance<Jim dancedJim didn’t attempt to dance>Jim didn’t dance

  • (Factives are only partly explained by this account)


Outline24 l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Outline

  • Introduction

  • Foundations of Natural Logic

  • The NatLog System

  • Experiments with FraCaS

  • Experiments with RTE

  • Conclusion


The natlog system l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

The NatLog system

textual inference problem

linguistic analysis

1

alignment

2

lexical entailment classification

3

entailment projection

4

entailment composition

5

prediction


Step 1 linguistic analysis l.jpg

No

S

completely

state

VP

forbid

NP ADVP NP

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

DT NNS RB VBD NN NN

casino

gambling

+

+

+

Step 1: Linguistic analysis

  • Tokenize & parse input sentences (future: & NER & coref & …)

  • Identify items w/ special projectivity & determine scope

  • Problem: PTB-style parse tree  semantic structure!

no

pattern: DT < /^[Nn]o$/

arg1: M on dominating NP

__ >+(NP) (NP=proj !> NP)

arg2: M on dominating S

__ > (S=proj !> S)

No state completely forbids casino gambling

  • Solution: specify scope in PTB trees using Tregex


Step 2 alignment l.jpg

MAT

INS

MAT

SUB

DEL

MAT

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Step 2: Alignment

  • Phrase-based alignments: symmetric, many-to-many

  • Can view as sequence of atomic edits: DEL, INS, SUB, MAT

  • Ordering of edits defines path through intermediate forms

    • Need not correspond to sentence order

    • Ordering of some edits can influence effective projectivity for others

    • We use heuristic reordering of edits to simplify this

  • Decomposes problem into atomic entailment problems

  • We haven’t (yet) invested much effort here

    • Experimental results use alignments from other sources


Running example l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Running example

OK, the example is contrived, but it compactly exhibits containment, exclusion, and implicativity


Step 3 lexical entailment classification l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Step 3: Lexical entailment classification

  • Predict basic entailment relation for each edit, based solely on lexical features, independent of context

  • Feature representation:

    • WordNet features: synonymy, hyponymy, antonymy

    • Other relatedness features: Jiang-Conrath (WN-based), NomBank

    • String and lemma similarity, based on Levenshtein edit distance

    • Lexical category features: prep, poss, art, aux, pron, pn, etc.

    • Quantifier category features

    • Implication signatures (for DEL edits only)

  • Decision tree classifier

    • Trained on 2,449 hand-annotated lexical entailment problems

    • >99% accuracy on training data — captures relevant distinctions


Running example30 l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Running example


Step 4 entailment projection l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Step 4: Entailment projection


Step 5 entailment composition l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

interesting

final answer

Step 5: Entailment composition


Outline33 l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Outline

  • Introduction

  • Foundations of Natural Logic

  • The NatLog System

  • Experiments with FraCaS

  • Experiments with RTE

  • Conclusion


The fracas test suite l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

The FraCaS test suite

  • FraCaS: mid-90s project in computational semantics

  • 346 “textbook” examples of textual inference problems

    • examples on next slide

  • 9 sections: quantifiers, plurals, anaphora, ellipsis, …

  • 3 possible answers: yes, no, unknown (not balanced!)

  • 55% single-premise, 45% multi-premise (excluded)


Fracas examples l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

FraCaS examples


Results on fracas l.jpg

27% error reduction NatLog System •

in largest category,

all but one correct

high accuracy

in sections

most amenable

to natural logic

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

high precision

even outsideareas of expertise

Results on FraCaS


Fracas confusion matrix l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

FraCaS confusion matrix

guess

gold


Outline38 l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Outline

  • Introduction

  • Foundations of Natural Logic

  • The NatLog System

  • Experiments with FraCaS

  • Experiments with RTE

  • Conclusion


The rte3 test suite l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

The RTE3 test suite

  • RTE: more “natural” textual inference problems

  • Much longer premises: average 35 words (vs. 11)

  • Binary classification: yes and no

  • RTE problems not ideal for NatLog

    • Many kinds of inference not addressed by NatLog:

      paraphrase, temporal reasoning, relation extraction, …

    • Big edit distance  propagation of errors from atomic model


Rte3 examples l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

RTE3 examples


Results on rte3 data l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Results on RTE3 data

  • Accuracy is unimpressive, but precision is relatively high

  • Maybe we can achieve high precision on a subset?

  • Strategy: hybridize with broad-coverage RTE system

    • As in Bos & Markert 2006

(each data set contains 800 problems)


A simple bag of words model l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

A simple bag-of-words model

H

P

similarity scores on [0, 1]for each pair of words

(I used a really simple-mindedsimilarity function based onLevenshtein string-edit distance)

max sim for each hyp word

how rare each word is

= (max sim)^IDF

= h P(h|P)


Results on rte3 data43 l.jpg

+10 NatLog System • Experiments with FraCaS • probs

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Results on RTE3 data

(each data set contains 800 problems)


Combining bow natlog l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Combining BoW & NatLog

  • MaxEnt classifier

  • BoW features: P(H|P), P(P|H)

  • NatLog features:7 boolean features encoding predicted entailment relation


Results on rte3 data45 l.jpg

+11 NatLog System • Experiments with FraCaS • probs

+3probs

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Results on RTE3 data

(each data set contains 800 problems)


Problem natlog is too precise l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Problem: NatLog is too precise?

  • Error analysis reveals a characteristic pattern of mistakes:

    • Correct answer is yes

    • Number of edits is large (>5) (this is typical for RTE)

    • NatLog predicts < or = for all but one or two edits

    • But NatLog predicts some other relation for remaining edits!

    • Most commonly, it predicts > for an insertion (e.g., RTE3_dev.71)

    • Result of relation composition is thus #, i.e. no

  • Idea: make it more forgiving, by adding features

    • Number of edits

    • Proportion of edits for which predicted relation is not < or =


Results on rte3 data47 l.jpg

+13 NatLog System • Experiments with FraCaS • probs

+8probs

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Results on RTE3 data

(each data set contains 800 problems)


Outline48 l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Outline

  • Introduction

  • Foundations of Natural Logic

  • The NatLog System

  • Experiments with FraCaS

  • Experiments with RTE

  • Conclusion


What natural logic can t do l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

What natural logic can’t do

  • Not a universal solution for textual inference

  • Many types of inference not amenable to natural logic

    • Paraphrase: Eve was let go = Eve lost her job

    • Verb/frame alternation: he drained the oil<the oil drained

    • Relation extraction: Aho, a trader at UBS…<Aho works for UBS

    • Common-sense reasoning: the sink overflowed<the floor got wet

    • etc.

  • Also, has a weaker proof theory than FOL

    • Can’t explain, e.g., de Morgan’s laws for quantifiers:

      Not all birds fly= Some birds don’t fly


What natural logic can do l.jpg

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

:-)

Thanks! Questions?

What natural logic can do

Natural logic enables precise reasoning about containment, exclusion, and implicativity, while sidestepping the difficulties of translating to FOL.

The NatLog system successfully handles a broad range of such inferences, as demonstrated on the FraCaS test suite.

Ultimately, open-domain textual inference is likely to require combining disparate reasoners, and a facility for natural logic is a good candidate to be a component of such a system.

Future work: phrase-based alignment for textual inference


ad