correlated topic models by blei and lafferty nips 2005 l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Correlated Topic Models By Blei and Lafferty (NIPS 2005) PowerPoint Presentation
Download Presentation
Correlated Topic Models By Blei and Lafferty (NIPS 2005)

Loading in 2 Seconds...

play fullscreen
1 / 16

Correlated Topic Models By Blei and Lafferty (NIPS 2005) - PowerPoint PPT Presentation


  • 273 Views
  • Uploaded on

Correlated Topic Models By Blei and Lafferty (NIPS 2005). Presented by Chunping Wang ECE, Duke University August 4 th , 2006. Outlines. Introduction Latent Dirichlet Allocation (LDA) Correlated Topic Models (CTM) Experimental Results Conclusions. Introduction(1).

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Correlated Topic Models By Blei and Lafferty (NIPS 2005)' - finn


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
correlated topic models by blei and lafferty nips 2005

Correlated Topic ModelsBy Blei and Lafferty (NIPS 2005)

Presented by Chunping Wang

ECE, Duke University

August 4th, 2006

slide2

Outlines

  • Introduction
  • Latent Dirichlet Allocation (LDA)
  • Correlated Topic Models (CTM)
  • Experimental Results
  • Conclusions
slide3

Introduction(1)

Topic models: generative probabilistic models which use a small number of distributions over a vocabulary to describe text collections and other discrete data (such as image). Normally, some latent variables are introduced to capture abstract notions such as topics.

Applications: document modeling, text classification, image processing, collaborative filtering, etc.

Latent Dirichlet Allocation (LDA): allows each document to exhibit multiple topics, but ignores the correlation between topics.

Correlated Topic Models (CTM): is based on LDA and dresses the limitation of LDA.

slide4

Introduction(2)

Notation and terminology (text collections)

  • Word:the basic unit from a vocabulary of size V (includes V distinct words). The vth word is represented by
  • Document: a sequence of N words.
  • Corpus: a collection of M documents.

Assumptions:

  • The words in a document are exchangeable;
  • Documents are also exchangeable.
slide5

Latent Dirichlet Allocation (LDA) (1)

fixed known parameters

fixed unknown parameters

Random variables (w are observable)

Generative process for each document W in a corpus D:

  • Choose
  • For each of the N words
  • Choose a topic index
  • Choose a word
  • are document-level variables, z and w are word-level variables.
slide6

Latent Dirichlet Allocation (LDA) (2)

  • Pros:
  • The Dirichlet distribution is in the exponential family and conjugate to the multinomial distribution --- variational inference is tractable.
  • are document-specific, so the variational parameters of could be regarded as the representation of a document --- feature set is reduced.
  • are sampled repeatedly within a document --- one document can be associated with multiple topics.
  • Cons:
  • Because of the independence assumption implicit in the Dirichlet distribution, LDA is unable to capture the correlation between different topics.
slide7

Correlated Topic Models (CTM) (1)

Key point: the topic proportions are drawn from a logistic normal distribution rather than a Dirichlet distribution.

Definition of logistic normal distribution

Let denote k-dimensional real space, the (k-1)-dimensional positive simplex defined by

Suppose that follows a multinormal distribution over . The logistic transformation from to can be used to define a logistic distribution over .

slide8

Logistic transformation

1

1

Log ratio transformation

1

Correlated Topic Models (CTM) (2)

The density function of

The logistic normal distribution is defined over the simplex as Dirichlet distribution and it allows correlation between components.

slide9

Correlated Topic Models (CTM) (3)

Generative process for each document W in a corpus D:

  • Choose
  • For each of the N words
  • Choose a topic
  • (b) Choose a word
slide10

Correlated Topic Models (CTM) (4)

Posterior inference (for in each document) – variational inference

where

Difficulty: the logistic normal is not exponential conjugate.

Solution: we lower bound it with a Taylor expansion

concave

slide11

Correlated Topic Models (CTM) (5)

Parameters estimation (for ) – maximizing the likelihood of the entire corpus of documents

Variational EM

1. (E-step) For each document, we maximize the lower bound with respect to the variational parameters ;

2. (M-step) Maximize the lower bound of the likelihood of the entire corpus with respect to the model parameters and

slide12

Experimental Results (1)

Example: Modeling Science

slide13

Experimental Results (2)

Comparison with LDA - Document modeling

slide14

Experimental Results (3)

Comparison with LDA – Collaborative filtering

To evaluate how well the models predict the remaining words after observing a portion of the document, we need to define a measure to compare .

Lower numbers denote more predictive power.

slide15

Conclusions

  • The main contribution of this paper is that the CTM directly model correlation between topics via the logistic normal distribution.
  • At the same time, the nonconjugacy of the logistic normal distribution adds complexity to the variational inference process.
  • As the LDA, the CTM allows multiple topics for each document; its variational parameters could serve as features of the document.
slide16

Reference:

J. Aitchison and S.M. Shen. Logistic-Normal Distributions: Some Properties and Uses. Biometrika, vol.67, no.2, pp.261-272, 1980.

D. Blei, A. Ng and M. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3: 993-1022, 2003.