Variational Bayes Model Selection for Mixture Distribution

1. Variational Bayes Model Selectionfor Mixture Distribution Authors: Adrian Corduneanu & Christopher M. Bishop Presented by Shihao Ji Duke University Machine Learning Group Jan. 20, 2006

2. Outline • Introduction – model selection • Automatic Relevance Determination (ARD) • Experimental Results • Application to HMMs

3. Introduction • Cross validation • Bayesian approaches • MCMC and Laplace approximation • (Traditional) variational method • (Type II) variational method

4. Automatic Relevance Determination (ARD) • relevance vector regression • Given a dataset , we assume is Gaussian Likelihood: Prior: Posterior: Determination of hyperparameters: Type II ML

5. Automatic Relevance Determination (ARD) • mixture of Gaussian • Given an observed dataset , we assume each data point is drawn • independently from a mixture of Gaussian density Likelihood: Prior: Posterior: VB Determination of mixing coefficients: Type II ML

6. Automatic Relevance Determination (ARD) • model selection Bayesian method: , Component elimination: if , i.e.,

7. Experimental Results • Bayesian method vs. cross-validation 600 points drawn from a mixture of 5 Gaussians.

8. Experimental Results • Component elimination Initially the model had 15 mixtures, finally was pruned down to 3 mixtures

9. Experimental Results

10. Automatic Relevance Determination (ARD) • hidden Markov model • Given an observed dataset , we assume each data sequence is • generated independently from an HMM Likelihood: Prior: Posterior: VB Determination of p and A: Type II ML

11. Automatic Relevance Determination (ARD) • model selection Bayesian method: , State elimination: if , Define -- visiting frequency where

12. Experimental Results (1)

14. Experimental Results (2)

16. Experimental Results (3)

18. Questions?