1 / 21

Mastering Question Answering: Acquired Skill Unveiled

Dive deep into the art of question answering with insights from Chakrabarti and experts. Understand techniques, benefits, and challenges in acquiring this skill. Explore strategies for improving information retrieval and mastering contextual understanding.

tanith
Download Presentation

Mastering Question Answering: Acquired Skill Unveiled

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Is Question Answeringan Acquired Skill? Soumen Chakrabarti G. RamakrishnanD. ParanjpeP. Bhattacharyya IIT Bombay

  2. Web search and QA • Information need – words relating “things” + “thing” aliases = telegraphic Web queries • Cheapest laptop with wireless  best price laptop 802.11 • Why is the sky blue?  sky blue reason • When was the Space Needle built? “Space Needle” history • Entity and relation extraction technology better than ever (SemTag, KnowItAll) • Ontology extension (e.g., is a kind of) • List extraction (e.g., is an instance of) • Slot-filling (author X wrote book Y) Chakrabarti

  3. Factoid QA • Specialize given domain to a token related to ground constants in the query • What animal is Winnie the Pooh? • hyponym(“animal”) NEAR “Winnie the Pooh” • When was television invented? • instance-of(“time”) NEAR “television” NEAR synonym(“invented”) • Three kinds of useful question tokens • Appear unchanged in passage (selector) • Specialize to answer tokens (atype) • Improve belief in answer via synonymy etc. Chakrabarti

  4. A new relational view of QA Question Atypeclues Attributeor columnname Selectors Locate whichcolumn to read Directsyntacticmatch Entity class IS-A Limit searchto certain rows “Landingzone” Questionwords Answerpassage “Landing zone” • Entity class or atype may be expressed by • A finite IS-A hierarchy (e.g. WordNet, TAP) • A surface pattern matching infinitely many strings (e.g. “digit+”, “Xx+”, “preceded by a preposition”) • Match selectors, specialize atype to answer tokens Chakrabarti

  5. Benefits of the relational view • “Scaling up by dumbing down” • Next stop after vector-space • Far short of real knowledge representation and inference • Barely getting practical at (near) Web scale • Can set up as a learning problem: train with questions and answers embedded in passage context • Transparent, self-tuning, easy to deploy • Feature extractors used in entity taggers • Relational/graphical learning on features Chakrabarti

  6. Subproblems • Identify atype clues • Easy: who, when, where, how many, how tall… • Harder: What…, which…, name… • Map atype clues to likely entity classes • Data- and task-driven question classification • Train quickly on new corpus and QA samples • Identify selectors for keyword query • Based on question context and global stats • Get candidate passages from IR system • Re-rank candidate passages Chakrabarti

  7. Mapping “self-evident” atypes • Whoperson, whentime, whereplace • Not always trivial: how_many vs. when • Question classification + handcrafted map • Needs task knowledge and skilled effort • Laborious to move to new corpus, language… • Task-driven information extraction • Enough info in training QA pairs to learn map • Map clue to a generalization of the answer • Surface patterns: hasDigit, [in] DDDD, NNP, CD • WordNet-based: region#n#3, quantity#n#1 Chakrabarti

  8. Mapping examples how who abstraction#n#6NNS NNP, person fast far many rich wrote first rate#n#2 explorer mile#n#3linear_unit#n#1 paper_money#n#1 currency#n#1 WordNet writer, composer,artist, musician measure#n#3definite_quantity#n#1 rate#n#2magnitude_relation#n#1 A cheetah can chase its preyat up to 90 km/h Nothing moves faster than186,000 miles per hour, thespeed of light How fast can a cheetah run? How fast does light travel? Chakrabarti

  9. What…, which…, name… atype clues • Assumption: Question sentence has a wh-word and a main/auxiliary verb • Observation: Atype clues are embedded in a noun phrase (NP) adjoining the main or auxiliary verb • Heuristic: Atype clue = head of this NP • Use a shallow parser and apply rule • Head can have attributes • Which (American(general)) is buried in Salzburg? • Name (Saturn’s (largest (moon))) Chakrabarti

  10. Atype clue extraction stats • Simple heuristic surprisingly effective • If successful, extracted atype is mapped to WordNet synset (mooncelestial body etc.) • If no atype of this form available, try the “self-evident” atypes (who, when, where, how_X etc.) Chakrabarti

  11. Learning selectors • Which question words are likely to appear (almost) unchanged in an answer passage? • Constants in select-clauses of SQL queries • Guides backoff policy for keyword query • Local and global features • POS of word, POS of adjacent words, case info, proximity to wh-word • Suppose word is associated with synset set S • NumSense: size of S (how polysemous is the word?) • NumLemma: average #lemmas describing sS POS@-1 POS@0 POS@1 Chakrabarti

  12. Selector results • Decision trees better than logistic regression • F1=81% as against LR F1=75% • Intuitive decision branches • But logistic regression gives scores for query backoff • Global features (IDF, NumSense, NumLemma) essential for accuracy • Best F1 accuracy with local features alone: 71—73% • With local and global features: 81% Chakrabarti

  13. Putting together a QA system Learning tools TrainingCorpus Shallow parser Wordnet QASystem POSTagger N-E Tagger Chakrabarti

  14. Noun andverb markers Taggedquestion Tokenizer POS Tagger ShallowParser AtypeExtractor Atype clues SelectorLearner • Learning to rerank passages • Sample features: • Do selectors match? How many? • Is some non-selector passage token a specialization of the question’s atype clue? • Min, avg linear token distance between candidate token and matched selectors Is QA pair? Taggedpassage Tokenizer POS TaggerEntity Extractor LogisticRegression Rerankedpassages Putting together a QA system Question Keyword querygenerator Keyword query PassageIndex Candidatepassage Sentence splitterPassage indexer Corpus Chakrabarti

  15. Surface pattern hasDigits selector match WordNet match 5 tokens apart 1 Learning to re-rank passages • Remove passage tokens matching selectors • User already knows these are in passage • Find passage token/s specializing atype • For each candidate token collect • Atype of question, original rank of passage • Min, avg linear distances to matched selectors • POS and entity tag of token if available How many inhabitants live in the town of Ushuaia Ushuaia, a port of about 30,000 dwellers set between the Beagle Channel and … Chakrabarti

  16. Effect of re-ranking results • Categorical andnumeric attributes • Logistic regression • Good precision,poor recall • Use logit score tore-rank passages • Rank of first correctpassage shifts substantially Log scale Chakrabarti

  17. Mean reciprocal rank studies • nq = smallest rank among answer passages • Re-ranking reduces nqdrastically • MRR = (1/|Q |) qQ(1/nq) • Substantial gain in MRR • TREC 2000 top MRRs:0.76 0.71 0.46 0.46 0.31 Chakrabarti

  18. Generalization across corpora • Across-year numbers close to train/test split on a single year • Features and model seem to capture corpus-independent linguistic Q+A artifacts Chakrabarti

  19. Re-ranking benefits by question type • All question types benefit from re-ranking • Benefits differ by question type • Large benefits for “what” and “which” questions, thanks to WordNet • Without WordNet customization Chakrabarti

  20. Conclusion • A clean-room view of QA as feature extraction plus learning • Recover structure info from question • Learn correlations between question structure and passage features • Competitive accuracy with negligible domain expertise or manual intervention • Ongoing work • Use redundancy available from the Web • Model how selector and atype are related • Treat all question types uniformly Chakrabarti

More Related