Loading in 2 Seconds...

Containment, Exclusion, and Implicativity: A Model of Natural Logic for Textual Inference

Loading in 2 Seconds...

- 209 Views
- Uploaded on

Download Presentation
## PowerPoint Slideshow about 'Containment, Exclusion, and Implicativity: A Model of Natural Logic for Textual Inference' - mike_john

**An Image/Link below is provided (as is) to download presentation**

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

### Containment, Exclusion, and Implicativity:A Model of Natural Logic for Textual Inference

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

What is natural logic?Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Implicatives & factivesIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

OutlineIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

A theory of natural logicIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

16 elementary set relationsIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

7 basic entailment relationsIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Relations for all semantic typesIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Monotonicity of semantic functionsIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Generalizing to projectivityIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Example: projectivity of refuseIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

A weak proof procedureIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Composing entailment relationsIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Implicatives & factivesIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

OutlineIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

The NatLog systemIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Step 2: AlignmentIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Running exampleIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Step 3: Lexical entailment classificationIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Running exampleIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Step 4: Entailment projectionIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

OutlineIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

The FraCaS test suiteIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

FraCaS examplesIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

FraCaS confusion matrixIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

OutlineIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

The RTE3 test suiteIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

RTE3 examplesIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Results on RTE3 dataIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

A simple bag-of-words modelIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Results on RTE3 dataIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Combining BoW & NatLogIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Results on RTE3 dataIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Problem: NatLog is too precise?Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Results on RTE3 dataIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

OutlineIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

What natural logic can’t doIntroduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Bill MacCartney and Christopher D. Manning

NLP Group

Stanford University

14 February 2008

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

The textual inference task- Does premise P justify an inference to hypothesis H?
- An informal, intuitive notion of inference: not strict logic
- Focus on local inference steps, not long chains of deduction
- Robust, accurate textual inference could enable:
- Question answering [Harabagiu & Hickl 06]
- Semantic search
- Customer email response
- Relation extraction (database building)
- Document summarization
- A limited selection of data sets: RTE, FraCaS, KBE

No western state completely forbids casino gambling.

No state completely forbids gambling.

Few or no states completely forbid casino gambling.

No

No state completely forbids casino gambling for kids.

No state restricts gambling.

No state or city completely forbids casino gambling.

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Some simple inferencesNo state completely forbids casino gambling.

What kind of textual inference system could predict this?

FOL &theoremproving

semantic

graph

matching

Hickl et al. 2006

MacCartney et al. 2006

Burchardt & Frank 2006

patternedrelationextraction

Romano et al. 2006

lexical/semanticoverlap

Introduction • Foundations of Natural Logic • The NatLog System • Experiments with FraCaS • Experiments with RTE • Conclusion

Jijkoun & de Rijke 2005

Textual inference:a spectrum of approachesdeep,but brittle

naturallogic

robust,but shallow

- (natural logic natural deduction)
- Lakoff (1970) defines natural logic as a goal (not a system)
- to characterize valid patterns of reasoning via surface forms (syntactic forms as close as possible to natural language)
- without translation to formal notation:
- A long history
- traditional logic: Aristotle’s syllogisms, scholastics, Leibniz, …
- van Benthem & Sánchez Valencia (1986-91): monotonicity calculus
- Precise, yet sidesteps difficulties of translating to FOL:

idioms, intensionality and propositional attitudes, modalities, indexicals, reciprocals,scope ambiguities, quantifiers such as most, reciprocals, anaphoric adjectives, temporal and causal relations, aspect, unselective quantifiers, adverbs of quantification, donkey sentences, generic determiners, …

+

–

+

–

–

–

+

+

+

Monotonicity calculus (Sánchez Valencia 1991)- Entailment as semantic containment:

rat< rodent, eat< consume, this morning< today, most< some

- Monotonicity classes for semantic functions
- Upward monotone: some rats dream< some rodents dream
- Downward monotone: no rats dream > no rodents dream
- Non-monotone: most rats dream # most rodents dream

- Handles even nested inversions of monotonicity

Every state forbids shooting game without a hunting license

- But lacks any representation of exclusion

Garfield is a cat< Garfield is not a dog

- Work at PARC, esp. Nairn et al. 2006
- Explains inversions & nestings of implicatives & factives
- Ed did not forget to force Dave to leave Dave left
- Defines 9 implication signatures
- “Implication projection algorithm”
- Bears some resemblance to monotonicity calculus
- But, fails to connect to containment or monotonicity
- John refused to danceJohn didn’t tango

- Introduction
- Foundations of Natural Logic
- The NatLog System
- Experiments with FraCaS
- Experiments with RTE
- Conclusion

Three elements:

- an inventory of entailment relations
- semantic containment relations of Sánchez Valencia
- plus semantic exclusion relations
- a concept of projectivity
- explains entailments compositionally
- generalizes Sánchez Valencia’s monotonicity classes
- generalizes Nairn et al.’s implication signatures
- a weak proof procedure
- composes entailment relations across chains of edits

Nonon-entailment

2-wayRTE1,2,3

Yesentailment

Unknownnon-entailment

Nocontradiction

3-wayFraCaS, PARC,RTE4?

P = Qequivalence

P < Qforwardentailment

P > Qreverseentailment

P # Qnon-entailment

containmentSánchez-Valencia

Entailment relations in past workQ

Q

P

P

P and Q can representsets of entities (i.e., predicates)or of possible worlds (propositions)cf. Tarski’s relation algebra

P _ Q

P > Q

P = Q

P < Q

P | Q

P # Q

16 elementary set relationsQ

Q

P

P

P and Q can representsets of entities (i.e., predicates)or of possible worlds (propositions)cf. Tarski’s relation algebra

The entailment relations are defined for expressions of all semantic types (not just sentences)

Sánchez Valencia’s monotonicity calculus assigns semantic functions to one of three monotonicity classes:

Upward-monotone (M)

The default: “bigger” inputs yield “bigger” outputs

Example: broken. Since chairfurniture, broken chairbroken furniture

Heuristic: in a M context, broadening edits preserve truth

Downward-monotone (M)

Negatives, restrictives, etc.: “bigger” inputs yield “smaller” outputs

Example: doesn’t. While hoverfly, doesn’t flydoesn’t hover

Heuristic: in a M context, narrowing edits preserve truth

Non-monotone (#M)

Superlatives, some quantifiers (most, exactly n): neither M nor M

Example: most. While penguinbird, most penguins # most birds

Heuristic: in a #M context, no edits preserve truth

restrictive quantifiers:no, few, at most n

didn’t dance< didn’t tango

at most two people< at most two men

negative & restrictive nouns:ban, absence [of], refusal

negative & restrictive verbs:lack, fail, prohibit, deny

drug ban< heroin ban

prohibit weapons< prohibit guns

prepositions & adverbs:without, except, only

the antecedent of a conditional

If stocks rise, we’ll get real paid<If stocks soar, we’ll get real paid

without clothes< without pants

Downward monotonicityDownward-monotone constructions are widespread!

- How do the entailments of a compound expression depend on the entailments of its parts?
- How does the entailment relation between (fx) and (fy) depend on the entailment relation between x and y(and the properties of f)?
- Monotonicity gives partial answer (for =, <, >, #)
- But what about the other relations (^, |, _)?
- We’ll categorize semantic functions based on how they project the basic entailment relations

swaps

these too

Example: projectivity of notswitch

blocks,

not swaps

@

>

>

>

@

@

<

<

@

@

@

@

nobody

nobody

can

can

without

without

a shirt

clothes

enter

enter

Projecting entailment relations upwardNobody can enter without a shirt < Nobody can enter without clothes

- Assume idealized semantic composition trees

- Propagate lexical entailment relations upward, according to projectivity class of each node on path to root

- Find sequence of edits connecting P and H
- Insertions, deletions, substitutions, …
- Determine lexical entailment relation for each edit
- Substitutions: depends on meaning of substituends
- Deletions: < by default: red socks < socks
- But some deletions are special: not hungry ^ hungry
- Insertions are symmetric to deletions: > by default
- Project up to find entailment relation across each edit
- Compose entailment relations across sequence of edits

- Relation composition: if a R b and b S c, then a ? C
- cf. Tarski’s relation algebra
- Many compositions are intuitive

= º = = < º < < < º = < ^ º ^ =

- Some less obvious, but still accessible

| º ^ < fish | human, human ^ nonhuman,fish < nonhuman

- But some yield unions of basic entailment relations!

| º | {=, <, >, |, #} (i.e. the non-exhaustive relations)

- Larger unions convey less information (can approx. with #)
- This limits power of proof procedure described

- Nairn et al. 2006 define nine implication signatures
- These encode implications (+, –, o) in + and – contexts
- Refuse has signature –/o:refuse to danceimplies didn’t dancedidn’t refuse to dance implies neither danced nor didn’t dance
- Signatures generate different relations when deleted
- Deleting –/o generates |Jim refused to dance | Jim dancedJim didn’t refuse to dance _ Jim didn’t dance
- Deleting o/– generates <Jim attempted to dance<Jim dancedJim didn’t attempt to dance>Jim didn’t dance
- (Factives are only partly explained by this account)

- Introduction
- Foundations of Natural Logic
- The NatLog System
- Experiments with FraCaS
- Experiments with RTE
- Conclusion

textual inference problem

linguistic analysis

1

alignment

2

lexical entailment classification

3

entailment projection

4

entailment composition

5

prediction

S

completely

state

VP

forbid

NP ADVP NP

DT NNS RB VBD NN NN

casino

gambling

+

–

–

–

+

+

Step 1: Linguistic analysis- Tokenize & parse input sentences (future: & NER & coref & …)
- Identify items w/ special projectivity & determine scope
- Problem: PTB-style parse tree semantic structure!

no

pattern: DT < /^[Nn]o$/

arg1: M on dominating NP

__ >+(NP) (NP=proj !> NP)

arg2: M on dominating S

__ > (S=proj !> S)

No state completely forbids casino gambling

- Solution: specify scope in PTB trees using Tregex

INS

MAT

SUB

DEL

MAT

- Phrase-based alignments: symmetric, many-to-many
- Can view as sequence of atomic edits: DEL, INS, SUB, MAT

- Ordering of edits defines path through intermediate forms
- Need not correspond to sentence order
- Ordering of some edits can influence effective projectivity for others
- We use heuristic reordering of edits to simplify this
- Decomposes problem into atomic entailment problems
- We haven’t (yet) invested much effort here
- Experimental results use alignments from other sources

OK, the example is contrived, but it compactly exhibits containment, exclusion, and implicativity

- Predict basic entailment relation for each edit, based solely on lexical features, independent of context
- Feature representation:
- WordNet features: synonymy, hyponymy, antonymy
- Other relatedness features: Jiang-Conrath (WN-based), NomBank
- String and lemma similarity, based on Levenshtein edit distance
- Lexical category features: prep, poss, art, aux, pron, pn, etc.
- Quantifier category features
- Implication signatures (for DEL edits only)
- Decision tree classifier
- Trained on 2,449 hand-annotated lexical entailment problems
- >99% accuracy on training data — captures relevant distinctions

interesting

final answer

Step 5: Entailment composition- Introduction
- Foundations of Natural Logic
- The NatLog System
- Experiments with FraCaS
- Experiments with RTE
- Conclusion

- FraCaS: mid-90s project in computational semantics
- 346 “textbook” examples of textual inference problems
- examples on next slide
- 9 sections: quantifiers, plurals, anaphora, ellipsis, …
- 3 possible answers: yes, no, unknown (not balanced!)
- 55% single-premise, 45% multi-premise (excluded)

in largest category,

all but one correct

high accuracy

in sections

most amenable

to natural logic

high precision

even outsideareas of expertise

Results on FraCaSguess

gold

- Introduction
- Foundations of Natural Logic
- The NatLog System
- Experiments with FraCaS
- Experiments with RTE
- Conclusion

- RTE: more “natural” textual inference problems
- Much longer premises: average 35 words (vs. 11)
- Binary classification: yes and no
- RTE problems not ideal for NatLog
- Many kinds of inference not addressed by NatLog:

paraphrase, temporal reasoning, relation extraction, …

- Big edit distance propagation of errors from atomic model

- Accuracy is unimpressive, but precision is relatively high
- Maybe we can achieve high precision on a subset?
- Strategy: hybridize with broad-coverage RTE system
- As in Bos & Markert 2006

(each data set contains 800 problems)

H

P

similarity scores on [0, 1]for each pair of words

(I used a really simple-mindedsimilarity function based onLevenshtein string-edit distance)

max sim for each hyp word

how rare each word is

= (max sim)^IDF

= h P(h|P)

(each data set contains 800 problems)

- MaxEnt classifier
- BoW features: P(H|P), P(P|H)
- NatLog features:7 boolean features encoding predicted entailment relation

+3probs

(each data set contains 800 problems)

- Error analysis reveals a characteristic pattern of mistakes:
- Correct answer is yes
- Number of edits is large (>5) (this is typical for RTE)
- NatLog predicts < or = for all but one or two edits
- But NatLog predicts some other relation for remaining edits!
- Most commonly, it predicts > for an insertion (e.g., RTE3_dev.71)
- Result of relation composition is thus #, i.e. no
- Idea: make it more forgiving, by adding features
- Number of edits
- Proportion of edits for which predicted relation is not < or =

+8probs

(each data set contains 800 problems)

- Introduction
- Foundations of Natural Logic
- The NatLog System
- Experiments with FraCaS
- Experiments with RTE
- Conclusion

- Not a universal solution for textual inference
- Many types of inference not amenable to natural logic
- Paraphrase: Eve was let go = Eve lost her job
- Verb/frame alternation: he drained the oil<the oil drained
- Relation extraction: Aho, a trader at UBS…<Aho works for UBS
- Common-sense reasoning: the sink overflowed<the floor got wet
- etc.
- Also, has a weaker proof theory than FOL
- Can’t explain, e.g., de Morgan’s laws for quantifiers:

Not all birds fly= Some birds don’t fly

:-)

Thanks! Questions?

What natural logic can doNatural logic enables precise reasoning about containment, exclusion, and implicativity, while sidestepping the difficulties of translating to FOL.

The NatLog system successfully handles a broad range of such inferences, as demonstrated on the FraCaS test suite.

Ultimately, open-domain textual inference is likely to require combining disparate reasoners, and a facility for natural logic is a good candidate to be a component of such a system.

Future work: phrase-based alignment for textual inference

Download Presentation

Connecting to Server..