Variational Bayes Model Selection for Mixture Distribution

Download Presentation

Variational Bayes Model Selection for Mixture Distribution

Loading in 2 Seconds...

- 98 Views
- Uploaded on
- Presentation posted in: General

Variational Bayes Model Selection for Mixture Distribution

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Variational Bayes Model Selectionfor Mixture Distribution

Authors: Adrian Corduneanu & Christopher M. Bishop

Presented by Shihao Ji

Duke University Machine Learning Group

Jan. 20, 2006

Outline

- Introduction – model selection
- Automatic Relevance Determination (ARD)
- Experimental Results
- Application to HMMs

Introduction

- Cross validation
- Bayesian approaches
- MCMC and Laplace approximation
- (Traditional) variational method
- (Type II) variational method

Automatic Relevance Determination (ARD)

- relevance vector regression
- Given a dataset , we assume is Gaussian

Likelihood:

Prior:

Posterior:

Determination of hyperparameters:

Type II ML

Automatic Relevance Determination (ARD)

- mixture of Gaussian
- Given an observed dataset , we assume each data point is drawn
- independently from a mixture of Gaussian density

Likelihood:

Prior:

Posterior:

VB

Determination of mixing coefficients:

Type II ML

Automatic Relevance Determination (ARD)

- model selection

Bayesian method: ,

Component elimination: if ,

i.e.,

Experimental Results

- Bayesian method vs. cross-validation

600 points drawn from a mixture of 5 Gaussians.

Experimental Results

- Component elimination

Initially the model had 15 mixtures, finally was pruned down to 3 mixtures

Experimental Results

Automatic Relevance Determination (ARD)

- hidden Markov model
- Given an observed dataset , we assume each data sequence is
- generated independently from an HMM

Likelihood:

Prior:

Posterior:

VB

Determination of p and A:

Type II ML

Automatic Relevance Determination (ARD)

- model selection

Bayesian method: ,

State elimination: if ,

Define -- visiting frequency

where

Experimental Results (1)

Experimental Results (2)

Experimental Results (3)

Questions?