Empirical development of an exponential probabilistic model
Download
1 / 49

Empirical Development of an Exponential Probabilistic Model - PowerPoint PPT Presentation


  • 78 Views
  • Uploaded on

Empirical Development of an Exponential Probabilistic Model. Using Textual Analysis to Build a Better Model Jaime Teevan & David R. Karger CSAIL (LCS+AI), MIT. Goal: Better Generative Model. Generative v. discriminative model Applies to many applications Information retrieval (IR)

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Empirical Development of an Exponential Probabilistic Model' - mandelina


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Empirical development of an exponential probabilistic model

Empirical Development of anExponential Probabilistic Model

Using Textual Analysis to Build a Better Model

Jaime Teevan & David R. Karger

CSAIL (LCS+AI), MIT


Goal better generative model
Goal: Better Generative Model

  • Generative v. discriminative model

  • Applies to many applications

    • Information retrieval (IR)

      • Relevance feedback

      • Using unlabeled data

    • Classification

  • Assumptions explicit


Using a model for ir
Using a Model for IR

Hyper-learn

  • Define model

  • Learn parameters from query

  • Rank documents

  • Better model improves applications

    • Trickle down to improve retrieval

    • Classification, relevance feedback, …

  • Corpus specific models


Overview
Overview

  • Related work

  • Probabilistic models

    • Example: Poisson Model

    • Compare model to text

  • Hyper-learning the model

    • Exponential framework

    • Investigate retrieval performance

  • Conclusion and future work


Related work
Related Work

  • Using text for retrieval algorithm

    • [Jones, 1972], [Greiff, 1998]

  • Using text to model text

    • [Church & Gale, 1995], [Katz, 1996]

  • Learning model parameters

    • [Zhai & Lafferty, 2002]

Hyper-learn the model from text!


Probabilistic models
Probabilistic Models

  • Rank documents by RV =Pr(rel|d)

  • Naïve Bayesian models

RV =Pr(rel|d)


Probabilistic models1
Probabilistic Models

  • Rank documents by RV =Pr(rel|d)

  • Naïve Bayesian models

# occs in doc

= Pr(dt|rel)

features t

RV =Pr(rel|d)

Pr(d|rel)

8

words

  • Open assumptions

    • Feature definition

    • Feature distribution family

Defines the model!


Using a na ve bayesian model
Using a Naïve Bayesian Model

  • Define model

  • Learn parameters from query

  • Rank documents


Using a na ve bayesian model1
Using a Naïve Bayesian Model

  • Define model

  • Learn parameters from query

  • Rank documents

Pr(dt|rel) =


Using a na ve bayesian model2
Using a Naïve Bayesian Model

  • Define model

  • Learn parameters from query

  • Rank documents

  • Poisson Model

  • θ: specifies term distribution

dt

θ e

Pr(dt|rel) =

dt!


Example poisson distribution
Example Poisson Distribution

+

θ=0.0006

Pr(dt|rel)

Pr(dt|rel)≈1E-15

Term occurs exactlydt times


Using a na ve bayesian model3
Using a Naïve Bayesian Model

  • Define model

  • Learn parameters from query

  • Rank documents

  • Learn a θ for each term

  • Maximum likelihood θ

    • Term’s average number of occurrence

  • Incorporate prior expectations


Using a na ve bayesian model4
Using a Naïve Bayesian Model

  • Define model

  • Learn parameters from query

  • Rank documents


Using a na ve bayesian model5
Using a Naïve Bayesian Model

  • Define model

  • Learn parameters from query

  • Rank documents

  • For each document, find RV

  • Sort documents by RV

= Pr(dt|rel). words t

RV


Using a na ve bayesian model6
Using a Naïve Bayesian Model

  • Define model

  • Learn parameters from query

  • Rank documents

Which step goes wrong?

  • For each document, find RV

  • Sort documents by RV

= Pr(dt|rel). words t

RV


Using a na ve bayesian model7
Using a Naïve Bayesian Model

  • Define model

  • Learn parameters from query

  • Rank documents


Using a na ve bayesian model8
Using a Naïve Bayesian Model

  • Define model

  • Learn parameters from query

  • Rank documents

dt

θ e

Pr(dt|rel) =

dt!


How good is the model
How Good is the Model?

+

θ=0.0006

Pr(dt|rel)

15 times

Term occurs exactlydt times


How good is the model1
How Good is the Model?

+

θ=0.0006

Pr(dt|rel)

Misfit!

15 times

Term occurs exactlydt times


Hyper learning a better fit through textual analysis

Hyper-learning a Better FitThrough Textual Analysis

Using an Exponential Framework


Hyper learning framework
Hyper-Learning Framework

  • Need framework for hyper-learning

Mixtures

Poisson

Bernoulli

Normal


Hyper learning framework1
Hyper-Learning Framework

  • Need framework for hyper-learning

  • Goal: Same benefits as Poisson Model

    • One parameter

    • Easy to work with (e.g., prior)

Mixtures

Poisson

Bernoulli

Normal

One parameter exponential families


Exponential framework
Exponential Framework

  • Well understood, learning easy

    • [Bernardo & Smith, 1994], [Gous, 1998]

      Pr(dt|rel) = f(dt)g(θ)e

  • Functions f(dt) and h(dt) specify family

    • E.g., Poisson: f(dt) = (dt!)-1,h(dt) = dt

  • Parameter θ term’s specific distribution

θh(dt)


Using a hyper learned model
Using a Hyper-learned Model

  • Define model

  • Learn parameters from query

  • Rank documents


Using a hyper learned model1
Using a Hyper-learned Model

  • Hyper-learn model

  • Learn parameters from query

  • Rank documents


Using a hyper learned model2
Using a Hyper-learned Model

  • Hyper-learn model

  • Learn parameters from query

  • Rank documents

  • Want “best” f(dt) and h(dt)

  • Iterative hill climbing

    • Local maximum

    • Poisson starting point


Using a hyper learned model3
Using a Hyper-learned Model

  • Hyper-learn model

  • Learn parameters from query

  • Rank documents

  • Data: TREC query result sets

    • Past queries to learn about future queries

  • Hyper-learn and test with different sets


Recall the poisson distribution
Recall the Poisson Distribution

+

Pr(dt|rel)

15 times

Term occurs exactlydt times


Poisson starting point h d t
Poisson Starting Point - h(dt)

+

h(dt)

Pr(dt|rel) =f(dt)g(θ)e

θh(dt)

dt


Hyper learned model h d t
Hyper-learned Model - h(dt)

Hyper-learned Model - h(dt)

+

h(dt)

Pr(dt|rel) =f(dt)g(θ)e

θh(dt)

dt


Poisson distribution
Poisson Distribution

+

Pr(dt|rel)

15 times

Term occurs exactlydt times


Hyper learned distribution
Hyper-learned Distribution

Hyper-learned Distribution

+

Pr(dt|rel)

15 times

Term occurs exactlydt times


Hyper learned distribution1
Hyper-learned Distribution

Hyper-learned Distribution

+

Pr(dt|rel)

5 times

Term occurs exactlydt times


Hyper learned distribution2
Hyper-learned Distribution

Hyper-learned Distribution

+

Pr(dt|rel)

30 times

Term occurs exactlydt times


Hyper learned distribution3
Hyper-learned Distribution

Hyper-learned Distribution

+

Pr(dt|rel)

300 times

Term occurs exactlydt times


Performing retrieval
Performing Retrieval

  • Hyper-learn model

  • Learn parameters from query

  • Rank documents


Performing retrieval1
Performing Retrieval

Labeled docs

  • Hyper-learn model

  • Learn parameters from query

  • Rank documents

θh(dt)

Pr(dt|rel) = f(dt)g(θ)e

  • Learn θ for each term


Learning
Learning θ

  • Sufficient statistics

    • Summarize all observed data

    • τ1: # of observations

    • τ2: Σobservations d h(dt)

  • Incorporating prior easy

  • Map τ1 and τ2θ

20 labeled documents


Performing retrieval2
Performing Retrieval

  • Hyper-learn model

  • Learn parameters from query

  • Rank documents


Results labeled documents
Results: Labeled Documents

Results: Labeled Documents

Precision

Recall


Results labeled documents1
Results: Labeled Documents

Results: Labeled Documents

Precision

Recall


Performing retrieval3
Performing Retrieval

  • Hyper-learn model

  • Learn parameters from query

  • Rank documents

Short query


Retrieval query
Retrieval: Query

Retrieval: Query

  • Query = single labeled document

  • Vector space-like equation

    RV = Σa(t,d) + Σb(q,d)

  • Problem: Document dominates

  • Solution: Use only query portion

    • Another solution: Normalize

t in doc q in query


Retrieval query1
Retrieval: Query

Precision

Recall


Retrieval query2
Retrieval: Query

Precision

Recall


Retrieval query3
Retrieval: Query

Precision

Recall


Conclusion
Conclusion

  • Probabilistic models

    • Example: Poisson Model

  • Hyper-learning the model

    • Exponential framework

    • Learned a better model

    • Investigate retrieval performance

- Bad text model

- Easy to work with

- Heavy tailed!

- Better …


Future work
Future Work

  • Use model better

  • Use for other applications

    • Other IR applications

    • Classification

  • Correct for document length

  • Hyper-learn on different corpora

    • Test if learned model generalizes

    • Different for genre? Language? People?

  • Hyper-learn model better


Questions
Questions?

Contact us with questions:

Jaime Teevan

[email protected]

David Karger

[email protected]


ad