Empirical development of an exponential probabilistic model
This presentation is the property of its rightful owner.
Sponsored Links
1 / 49

Empirical Development of an Exponential Probabilistic Model PowerPoint PPT Presentation


  • 47 Views
  • Uploaded on
  • Presentation posted in: General

Empirical Development of an Exponential Probabilistic Model. Using Textual Analysis to Build a Better Model Jaime Teevan & David R. Karger CSAIL (LCS+AI), MIT. Goal: Better Generative Model. Generative v. discriminative model Applies to many applications Information retrieval (IR)

Download Presentation

Empirical Development of an Exponential Probabilistic Model

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Empirical development of an exponential probabilistic model

Empirical Development of anExponential Probabilistic Model

Using Textual Analysis to Build a Better Model

Jaime Teevan & David R. Karger

CSAIL (LCS+AI), MIT


Goal better generative model

Goal: Better Generative Model

  • Generative v. discriminative model

  • Applies to many applications

    • Information retrieval (IR)

      • Relevance feedback

      • Using unlabeled data

    • Classification

  • Assumptions explicit


Using a model for ir

Using a Model for IR

Hyper-learn

  • Define model

  • Learn parameters from query

  • Rank documents

  • Better model improves applications

    • Trickle down to improve retrieval

    • Classification, relevance feedback, …

  • Corpus specific models


Overview

Overview

  • Related work

  • Probabilistic models

    • Example: Poisson Model

    • Compare model to text

  • Hyper-learning the model

    • Exponential framework

    • Investigate retrieval performance

  • Conclusion and future work


Related work

Related Work

  • Using text for retrieval algorithm

    • [Jones, 1972], [Greiff, 1998]

  • Using text to model text

    • [Church & Gale, 1995], [Katz, 1996]

  • Learning model parameters

    • [Zhai & Lafferty, 2002]

Hyper-learn the model from text!


Probabilistic models

Probabilistic Models

  • Rank documents by RV =Pr(rel|d)

  • Naïve Bayesian models

RV =Pr(rel|d)


Probabilistic models1

Probabilistic Models

  • Rank documents by RV =Pr(rel|d)

  • Naïve Bayesian models

# occs in doc

= Pr(dt|rel)

features t

RV =Pr(rel|d)

Pr(d|rel)

8

words

  • Open assumptions

    • Feature definition

    • Feature distribution family

Defines the model!


Using a na ve bayesian model

Using a Naïve Bayesian Model

  • Define model

  • Learn parameters from query

  • Rank documents


Using a na ve bayesian model1

Using a Naïve Bayesian Model

  • Define model

  • Learn parameters from query

  • Rank documents

Pr(dt|rel) =


Using a na ve bayesian model2

Using a Naïve Bayesian Model

  • Define model

  • Learn parameters from query

  • Rank documents

  • Poisson Model

  • θ: specifies term distribution

dt

θ e

Pr(dt|rel) =

dt!


Example poisson distribution

Example Poisson Distribution

+

θ=0.0006

Pr(dt|rel)

Pr(dt|rel)≈1E-15

Term occurs exactlydt times


Using a na ve bayesian model3

Using a Naïve Bayesian Model

  • Define model

  • Learn parameters from query

  • Rank documents

  • Learn a θ for each term

  • Maximum likelihood θ

    • Term’s average number of occurrence

  • Incorporate prior expectations


Using a na ve bayesian model4

Using a Naïve Bayesian Model

  • Define model

  • Learn parameters from query

  • Rank documents


Using a na ve bayesian model5

Using a Naïve Bayesian Model

  • Define model

  • Learn parameters from query

  • Rank documents

  • For each document, find RV

  • Sort documents by RV

= Pr(dt|rel). words t

RV


Using a na ve bayesian model6

Using a Naïve Bayesian Model

  • Define model

  • Learn parameters from query

  • Rank documents

Which step goes wrong?

  • For each document, find RV

  • Sort documents by RV

= Pr(dt|rel). words t

RV


Using a na ve bayesian model7

Using a Naïve Bayesian Model

  • Define model

  • Learn parameters from query

  • Rank documents


Using a na ve bayesian model8

Using a Naïve Bayesian Model

  • Define model

  • Learn parameters from query

  • Rank documents

dt

θ e

Pr(dt|rel) =

dt!


How good is the model

How Good is the Model?

+

θ=0.0006

Pr(dt|rel)

15 times

Term occurs exactlydt times


How good is the model1

How Good is the Model?

+

θ=0.0006

Pr(dt|rel)

Misfit!

15 times

Term occurs exactlydt times


Hyper learning a better fit through textual analysis

Hyper-learning a Better FitThrough Textual Analysis

Using an Exponential Framework


Hyper learning framework

Hyper-Learning Framework

  • Need framework for hyper-learning

Mixtures

Poisson

Bernoulli

Normal


Hyper learning framework1

Hyper-Learning Framework

  • Need framework for hyper-learning

  • Goal: Same benefits as Poisson Model

    • One parameter

    • Easy to work with (e.g., prior)

Mixtures

Poisson

Bernoulli

Normal

One parameter exponential families


Exponential framework

Exponential Framework

  • Well understood, learning easy

    • [Bernardo & Smith, 1994], [Gous, 1998]

      Pr(dt|rel) = f(dt)g(θ)e

  • Functions f(dt) and h(dt) specify family

    • E.g., Poisson: f(dt) = (dt!)-1,h(dt) = dt

  • Parameter θ term’s specific distribution

θh(dt)


Using a hyper learned model

Using a Hyper-learned Model

  • Define model

  • Learn parameters from query

  • Rank documents


Using a hyper learned model1

Using a Hyper-learned Model

  • Hyper-learn model

  • Learn parameters from query

  • Rank documents


Using a hyper learned model2

Using a Hyper-learned Model

  • Hyper-learn model

  • Learn parameters from query

  • Rank documents

  • Want “best” f(dt) and h(dt)

  • Iterative hill climbing

    • Local maximum

    • Poisson starting point


Using a hyper learned model3

Using a Hyper-learned Model

  • Hyper-learn model

  • Learn parameters from query

  • Rank documents

  • Data: TREC query result sets

    • Past queries to learn about future queries

  • Hyper-learn and test with different sets


Recall the poisson distribution

Recall the Poisson Distribution

+

Pr(dt|rel)

15 times

Term occurs exactlydt times


Poisson starting point h d t

Poisson Starting Point - h(dt)

+

h(dt)

Pr(dt|rel) =f(dt)g(θ)e

θh(dt)

dt


Hyper learned model h d t

Hyper-learned Model - h(dt)

Hyper-learned Model - h(dt)

+

h(dt)

Pr(dt|rel) =f(dt)g(θ)e

θh(dt)

dt


Poisson distribution

Poisson Distribution

+

Pr(dt|rel)

15 times

Term occurs exactlydt times


Hyper learned distribution

Hyper-learned Distribution

Hyper-learned Distribution

+

Pr(dt|rel)

15 times

Term occurs exactlydt times


Hyper learned distribution1

Hyper-learned Distribution

Hyper-learned Distribution

+

Pr(dt|rel)

5 times

Term occurs exactlydt times


Hyper learned distribution2

Hyper-learned Distribution

Hyper-learned Distribution

+

Pr(dt|rel)

30 times

Term occurs exactlydt times


Hyper learned distribution3

Hyper-learned Distribution

Hyper-learned Distribution

+

Pr(dt|rel)

300 times

Term occurs exactlydt times


Performing retrieval

Performing Retrieval

  • Hyper-learn model

  • Learn parameters from query

  • Rank documents


Performing retrieval1

Performing Retrieval

Labeled docs

  • Hyper-learn model

  • Learn parameters from query

  • Rank documents

θh(dt)

Pr(dt|rel) = f(dt)g(θ)e

  • Learn θ for each term


Learning

Learning θ

  • Sufficient statistics

    • Summarize all observed data

    • τ1: # of observations

    • τ2: Σobservations d h(dt)

  • Incorporating prior easy

  • Map τ1 and τ2θ

20 labeled documents


Performing retrieval2

Performing Retrieval

  • Hyper-learn model

  • Learn parameters from query

  • Rank documents


Results labeled documents

Results: Labeled Documents

Results: Labeled Documents

Precision

Recall


Results labeled documents1

Results: Labeled Documents

Results: Labeled Documents

Precision

Recall


Performing retrieval3

Performing Retrieval

  • Hyper-learn model

  • Learn parameters from query

  • Rank documents

Short query


Retrieval query

Retrieval: Query

Retrieval: Query

  • Query = single labeled document

  • Vector space-like equation

    RV = Σa(t,d) + Σb(q,d)

  • Problem: Document dominates

  • Solution: Use only query portion

    • Another solution: Normalize

t in doc q in query


Retrieval query1

Retrieval: Query

Precision

Recall


Retrieval query2

Retrieval: Query

Precision

Recall


Retrieval query3

Retrieval: Query

Precision

Recall


Conclusion

Conclusion

  • Probabilistic models

    • Example: Poisson Model

  • Hyper-learning the model

    • Exponential framework

    • Learned a better model

    • Investigate retrieval performance

- Bad text model

- Easy to work with

- Heavy tailed!

- Better …


Future work

Future Work

  • Use model better

  • Use for other applications

    • Other IR applications

    • Classification

  • Correct for document length

  • Hyper-learn on different corpora

    • Test if learned model generalizes

    • Different for genre? Language? People?

  • Hyper-learn model better


Questions

Questions?

Contact us with questions:

Jaime Teevan

[email protected]

David Karger

[email protected]


  • Login