QuASI: Question Answering using Statistics, Semantics, and Inference

1. QuASI:Question Answering using Statistics, Semantics, and Inference Marti Hearst, Jerry Feldman, Chris Manning, Srini Narayanan Univ. of California-Berkeley / ICSI / Stanford University

2. Dynamic Probabilistic Inference for event structure Srini Narayanan Jerry Feldman ICSI and UC Berkeley Jan-June 2003

3. Scenario Question (CNS data) • How has Al-Qaida conducted its efforts to acquire WMD capability and what are the results of this endeavor? • Even with perfect parsing, to answer this question, we have to go beyond words in the input in at least the following ways: • Multiple sources (reports, evidence, news) • Fusing information from unreliable sources (P(Information = true | source)) • Non-monotonicity. Previous assertions or predictions may have to be retracted in the light of new evidence. • Modeling complex events • Evolving events with complex dynamics including sequence, concurrency, coordination, interruptions and resources.

4. Reasoning about Events for QA • Reasoning about dynamics • Complex event structure • Multiple stages, interruptions, resources • Evolving events • Conditional events, presuppositions. • Nested temporal and aspectual references • Past, future event references • Metaphoric references • Use of motion domain to describe complex events. • Reasoning with Uncertainty • Combining Evidence from Multiple, unreliable sources • Non-monotonic inference • Retracting previous assertions • Conditioning on partial evidence

5. Previous work • Models of event structure that are able to deal with the temporal and aspectual structure of events • Based on an active semantics of events and a factorized graphical model of complex states. • Models event stages, embedding, multi-level perspectives and coordination. • Event model based on a Stochastic Petri Net representation with extensions allowing hierarchical decomposition. • State is represented as a Temporal Bayes Net (T(D)BN).

9. Factorized Inference

10. Quantifying the model

11. Pilot System Results • Captures fine grained distinctions needed for interpretation • Frame-based Inferences (COLING02) • Aspectual Inferences (Cogsci98, IJCAI 99, COLING02) • Metaphoric Inferences (AAAI 99) • Sufficient Inductive bias for verb learning (Bailey97, CogSci99), construction learning (Chang02, to Appear) • Model for DAML-S (WWW02, Computer Networks 03)

12. Extensions to Pilot System • Scalable Data Resources • Language Resources/Ontology • Lexicon (Open Source, WordNet, FrameNet) • Conceptual Relations: • Schemas, Maps, Frames, Mental Space • General Principle: Use Semantic Web resources • (DAML, DAML-S, OpenCYC, IEEE SUMO) • Language Analyzer • Construction Parser (ICSI/EML) • Statistical techniques (UCB/Stanford, CU,UTD) • Scalable Domain Representation • Coordinated Probabilistic Relational Models

13. Problems with DBN • Scaling up to relational structures • Supports linear (sequence) but not branching (concurrency, coordination) dynamics

14. Structured Probabilistic Inference

15. Probabilistic inference for QA • Filtering • P(X_t | o_1…t,X_1…t) • Update the state based on the observation sequence and state set • MAP Estimation • Argmaxh1…hnP(X_t | o_1…t, X_1…t) • Return the best assignment of values to the hypothesis variables given the observation and states • Smoothing • P(X_t-k | o_1…t, X_1…t) • modify assumptions about previous states, given observation sequence and state set • Projection/Prediction/Reachability • P(X_t+k | o_1..t, X_1..t) • Predict future states based on observation sequence and state set

16. PRM (and DBN) inference is hard • Exact Inference Techniques (NP): • Variable Elimination (VE) • Junction-Tree Methods • Approximate inference (NP): • Variational Approximations • Loopy propagation (loses information)

17. Tractable inference and net topology • Polytree-inference is tractable (Pearl 1990) • Proportional to Network Size • SCFG-inference can be modeled as extended Polytree inference (Narayanan 99) • For more complicated models, exploit relational structure (Pfeffer 99, Kohler et al 00, 02).

18. Probabilistic Relation Inference • Scalable Representation of • States, domain knowledge, ontologies • (Pfeffer 2000, Koller et al. 2001) • Merges relational database technology with Probabilistic reasoning based on Graphical Models. • Domain entities and relations. • Inter-entity relations are probabilistic functions • Can capture complex dependencies with both simple and composite slot (chains). • Inference exploits structure of the domain

19. Inference With PRMs SVE inference for a PRM P with q query variables and N attributes is O(Nkbk(m+2)bq) (Pfeffer 2000) • k is the maximum number of interface variables • q is the number of query variables • m is the maximum tree width for any object in P (related to the markov blanket).

20. Controlling PRM inference • The number of interface variables, k, is related to the number of relations that a variable participates in as well as the number of slot chains that the variable participates in • Careful selection of relations (only part-of) can make inference tractable. • The tree width m depends on the markov blanket of an attribute. • Control of network topology can reduce this.

21. Adding Time to PRM’s • Since time is another relation, doesn’t increase expressive power. • Significant impact of inference tractability since both k and m may become quite large. • New Algorithm: Exploit the structure of time using the interface and frontier algorithm (Murphy 2002). • Variables at slice t with links to variables at t+1 form the interface • Interface variables d-separate the past (< t) from the future slices (> t). • Allows for on-line inference algorithms similar to inside-outside algorithm for SCFG’s.

22. The CPRM algorithm • Combines insights from • the SVE algorithm for PRMs (Pfeffer 2000) • the frontier algorithms for temporal models (Murphy 2002) and • Inference algorithms for complex, coordinated events (Narayanan 1999) • Expressive Probabilistic Modeling paradigm with relations and branching dynamics. • Offers principled methods to bound inferential complexity.

23. Status of CPRM inference • Spring-Summer 2003 • Design Dynamic Probabilistic Relational Models (DPRM) • Initial Design of CPRM inference algorithm • Integrate Parser with existing Pilot System • Steve Sinha • Summer/Fall 2003 • Implement CPRM to replace Pilot System • Nathaniel Smith, Eva Mok • Test CPRM for QA (UTD) • Related Work • Probabilistic OWL (PrOWL) • Probabilistic FrameNet

24. Conclusion • QA with complex scenarios (such as the CNS scenario/data) needs complex inference that deals with • Relational Structure • Uncertain source and domain knowledge • Complex dynamics and evolving events • We have developed a representation and inference algorithm that is capable of tractable inference for a variety of domains. • We are collaborating with UTD (Sanda Harabagiu) to apply these techniques to QA systems.