Generic Soft Pattern Models for Definitional Question Answering
Download
1 / 28

Patterns Are Everywhere - PowerPoint PPT Presentation


  • 249 Views
  • Uploaded on

Generic Soft Pattern Models for Definitional Question Answering Hang Cui Min-Yen Kan Tat-Seng Chua Department of Computer Science National University of Singapore. Patterns Are Everywhere. Information Extraction (IE) noun preposition <noun_phrase> e.g. bomb against <target>. Lexico-syntactic

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Patterns Are Everywhere' - medwin


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Slide1 l.jpg

Generic Soft Pattern Models for Definitional Question Answering

Hang CuiMin-Yen KanTat-Seng ChuaDepartment of Computer ScienceNational University of Singapore

Generic Soft Pattern Models for Definitional QA


Patterns are everywhere l.jpg
Patterns Are Everywhere Answering

Information Extraction (IE)

noun preposition <noun_phrase>

e.g. bomb against <target>

Lexico-syntactic

Patterns

Question Answering (QA)

<search_term> , DT$ NNP ,

e.g. Gunter Blobel , a biologist

at … , said …

Other tasks

<subj> passive-verb

e.g. <subj> was satisfied

Generic Soft Pattern Models for Definitional QA


Two methods of pattern matching l.jpg
Two Methods of Pattern Matching Answering

  • Hard Matching

    • Rule induction

    • Generalizing training instances into regular expression represented rules

    • Performing slot by slot matching

  • Soft Matching

    • Hidden Markov Models (HMM)

    • Soft pattern matching for definitional QA (Cui et al., 2004)

Generic Soft Pattern Models for Definitional QA


Hard matching l.jpg
Hard Matching Answering

Lack of flexibility in matching

  • Can’t deal with gaps between rules and test instances

<PersonIN>,NNP ,BE$named to<POST>

Bob Lloyd ,president and chief operating officer ,wasnamed to the chief executive.

Lee Abraham ,65 years old ,former chairman and chief executive officer of Associated Merchandising Corp. ,New York ,wasnamed to the board of the footwear manufacturer.

Gaps by insertion

Generic Soft Pattern Models for Definitional QA


Soft matching cui et al 2004 l.jpg
Soft Matching (Cui Answeringet al., 2004)

Training

…… The channel Iqra is owned by the … …… severance packages, known as golden parachutes, included …… A battery is a cell which can provide electricity.

DT$ NN <Search_Term> BE$ owned by

known as <Search_Term> , VB

<Search_Term> BE$ DT$

…… <Slot-2> <Slot-1> <Search_Term> <Slot1> <Slot2> ……

NN 0.12 NN 0.11 , 0.40 DT$ 0.2

known 0.09 as 0.20 BE$ 0.2 VB 0.1

DT$ 0.04 owned 0.09

… is known as Wicca, a neo-pagan nature religion, includes the use of herbal magic and witchcraft in its practice.

Testing

known as<Search_Term>, DT$

P ( Ins ) = P(“known”|S-2) + P(“as”|S-1) + P(“,”|S1) + P(“DT$”|S2)

+ P(“known as”) + P(“, DT$”)

Generic Soft Pattern Models for Definitional QA


Weakness of current soft matching models l.jpg
Weakness of Current Soft Matching Models Answering

  • Ad-hoc in model parameter estimation

    • Cui et al., 2004: Lack of formalization

  • Not generic

    • Task specific topology for HMM

      • Difficult to port to other applications

Generic Soft Pattern Models for Definitional QA


In this paper l.jpg
In This Paper … Answering

  • Propose two generic soft pattern models

    • Bigram model

      • Formalization of our previous model

    • Profile Hidden Markov Model (PHMM)

      • More complex model that handles gaps better

  • Parameter estimation by EM algorithm

  • Evaluations on definitional question answering

    • Can be applied to other pattern matching applications

Generic Soft Pattern Models for Definitional QA


Outline l.jpg
Outline Answering

  • Overview of Definitional QA

  • Bigram Soft Pattern Model

  • PHMM Soft Pattern Model

  • Evaluations

  • Conclusions

Generic Soft Pattern Models for Definitional QA


Outline9 l.jpg
Outline Answering

  • Overview of Definitional QA

  • Bigram Soft Pattern Model

  • PHMM Soft Pattern Model

  • Evaluations

  • Conclusions

Generic Soft Pattern Models for Definitional QA


Definitional qa l.jpg
Definitional QA Answering

(1) … that Wicca _ whose practitioners call themselves witches and believe in the dual deity of god and goddess _ is not a religion and should not be practiced on military bases.

(2) … , Wicca, as contemporary witchcraft is often called, has been growing in the United States and abroad.

(3) The Wiccans, whose religion is a reconstruction of nature worship from tribal Europe and other parts of the world, had to meet the same criteria as other religions to conduct services on the base, including sponsorship by a legally incorporated church, in this case one in San Antonio.

(4) Wiccaadherents celebrate eight major sabbats, festivals that mark the change of seasons and agricultural cycles, and believe in both god and goddess.

  • To answer questions like “Who is Gunter Blobel” or “What is Wicca”.

  • Why evaluating on definition sentence retrieval?

    • Diverse patterns

    • Definitional QA is one of the least explored areas in QA

Generic Soft Pattern Models for Definitional QA


Pattern matching for definitional qa l.jpg
Pattern Matching for Definitional QA Answering

  • Manually constructed patterns

    • Appositive

      • e.g. Gunter Blobel , a cellular and molecular biologist,…

    • Copulas

      • e.g. Battery is a kind of electronic device …

    • Predicates (relations)

      • e.g. TB is usually caused by …

Generic Soft Pattern Models for Definitional QA


Outline12 l.jpg
Outline Answering

  • Overview of Definitional QA

  • Bigram Soft Pattern Model

  • PHMM Soft Pattern Model

  • Evaluations

  • Conclusions

Generic Soft Pattern Models for Definitional QA


Bigram soft pattern model l.jpg
Bigram Soft Pattern Model Answering

  • To estimate the interpolation mixture weight λ

    • Expectation Maximization (EM) algorithm

  • Count words and general tags separately

    • Avoid overwhelming frequency count of general tags

Bigram prob

Slot-aware unigram prob

P ( Ins ) = P(“known”|S-2) + P(“as”|S-1) + P(“,”|S1) + P(“DT$”|S2) + P(“known as”) + P(“, DT$”)

Generic Soft Pattern Models for Definitional QA


Bigram model in dealing with gaps l.jpg
Bigram Model in Dealing with Gaps Answering

  • Bigram model can deal with gaps

    • Unseen tokens have small smoothing probabilities in specific positions

Pattern

<Search_Term> which is known for DT$ NNP

Test sentence:

<Search_Term> , whose book is known for

P(“,”|S1) = P(“whose”|S2) … = small smoothing prob

P(“known”|S3) = 0.3 P(“for”|S4) = 0.21

P(“,”|S1) = P(“whose”|S2) = P(“book”|S3) = P(“is”|S4) =

Not too good!

Generic Soft Pattern Models for Definitional QA


Outline15 l.jpg
Outline Answering

  • Overview of Definitional QA

  • Bigram Soft Pattern Model

  • PHMM Soft Pattern Model

  • Evaluations

  • Conclusions

Generic Soft Pattern Models for Definitional QA


Phmm soft pattern model l.jpg
PHMM Soft Pattern Model Answering

  • Better solution for dealing with gaps

  • Left to right Hidden Markov Model with insertion and deletion states

Generic Soft Pattern Models for Definitional QA


How phmm deals with gaps l.jpg
How PHMM Deals with Gaps Answering

  • Calculating generative probability given a test instance

    • Find the most probable path by Viterbi algorithm

    • Efficient calculation by forward-backward algorithm

NNP

known

as

DT$

Generic Soft Pattern Models for Definitional QA


Estimation of phmm l.jpg
Estimation of PHMM Answering

  • Estimated by Baum-Welch algorithm

    • Using the most probable path during training

  • Random or uniform initialization may lead to unsatisfactory model

    • Extreme diversity of definition patterns and not sufficient training data

    • Assume path should favor match states over others

      • P( token | Match ) > P ( token | Insertion )

    • Using smoothed ML estimates

Generic Soft Pattern Models for Definitional QA


Outline19 l.jpg
Outline Answering

  • Overview of Definitional QA

  • Bigram Soft Pattern Model

  • PHMM Soft Pattern Model

  • Evaluations

    • Overall performance evaluation

    • Sensitivity to model length

    • Sensitivity to size of training data

  • Conclusions

Generic Soft Pattern Models for Definitional QA


Evaluation setup l.jpg
Evaluation Setup Answering

  • Data set

    • Test data: TREC-13 question answering task data

      • AQUAINT corpus and 64 definition questions with answers

    • Training data

      • 761 manually labeled definition sentences from TREC-12 question answering task data

  • Comparison systems

    • State-of-the-art manually constructed patterns

      • Most comprehensive manually constructed patterns to our knowledge

    • Previously proposed soft pattern in Cui et al., 2004

Generic Soft Pattern Models for Definitional QA


Evaluation metrics l.jpg
Evaluation Metrics Answering

  • Manually checked F3 measure

    • Based on essential/acceptable answer nuggets

      • NR – proportion of returned essential answer nuggets

      • NP – penalty to longer answers

      • Weighting NR 3 times as NP

    • Subject to inconsistent scoring among assessors

  • Automatic ROUGE score

    • Gold standard: sentences containing answer nuggets

    • Counting the trigrams shared in the gold standard and system answers

    • ROUGE-3-ALL (R3A) and ROUGE-3-ESSENTIAL (R3E)

Generic Soft Pattern Models for Definitional QA


Performance evaluation l.jpg
Performance Evaluation Answering

  • Soft pattern matching outperforms hard matching

  • Bigram and PHMM models perform better than the previously proposed soft pattern method

    • Previous soft pattern method is not optimized

  • Manual F3 score correlate well with automatic R3 scores

Generic Soft Pattern Models for Definitional QA


Sensitivity to model length l.jpg
Sensitivity to Model Length Answering

  • PHMM is less sensitive to model length

  • PHMM may handle longer sequences

Generic Soft Pattern Models for Definitional QA


Sensitivity to the amount of training data l.jpg
Sensitivity to the Amount of Training Data Answering

  • PHMM requires more training data to improve

2.28%

7.22%

Generic Soft Pattern Models for Definitional QA


Discussions on both models l.jpg
Discussions on Both Models Answering

  • Capture the same information

    • The importance of a token’s position in the context of the search term

    • The sequential order of tokens

  • Different in complexity

    • Bigram model

      • Simplified Markov model with each token as a state

      • Captures token sequential information by bigram probabilities

    • PHMM model

      • More complex – aggregated token sequential information by hidden state transition probabilities

  • Experimental results show

    • PHMM is less sensitive to model length

    • PHMM may benefit more by using more training data

Generic Soft Pattern Models for Definitional QA


Outline26 l.jpg
Outline Answering

  • Overview of Definitional QA

  • Bigram Soft Pattern Model

  • PHMM Soft Pattern Model

  • Evaluations

  • Conclusions

Generic Soft Pattern Models for Definitional QA


Conclusions l.jpg
Conclusions Answering

  • Proposed Bigram model and PHMM model

    • Generic in the forms

    • Systematic parameter estimation by EM algorithm

  • These two models can be applied to other applications using surface text patterns

    • Soft patterns have been applied to information extraction (Xiao et al., 2004)

    • Can deal with diversified patterns

    • PHMM is more flexible in dealing with gaps, but requires more training data to converge

Generic Soft Pattern Models for Definitional QA


Slide28 l.jpg
Q & A Answering

Thanks!

Generic Soft Pattern Models for Definitional QA


ad