Em algorithm and mixture of gaussians l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 32

EM Algorithm and Mixture of Gaussians PowerPoint PPT Presentation


  • 339 Views
  • Uploaded on
  • Presentation posted in: General

EM Algorithm and Mixture of Gaussians. Collard Fabien - 20046056 김진식 (Kim Jinsik) - 20043152 주찬혜 (Joo Chanhye) - 20043595. Summary. Hidden Factors EM Algorithm Principles Formalization Mixture of Gaussians Generalities Processing Formalization Other Issues

Download Presentation

EM Algorithm and Mixture of Gaussians

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Em algorithm and mixture of gaussians l.jpg

EM AlgorithmandMixture of Gaussians

Collard Fabien - 20046056

김진식 (Kim Jinsik) - 20043152

주찬혜 (Joo Chanhye) - 20043595


Summary l.jpg

Summary

  • Hidden Factors

  • EM Algorithm

    • Principles

    • Formalization

  • Mixture of Gaussians

    • Generalities

    • Processing

    • Formalization

  • Other Issues

    • Bayesian Network with hidden variables

    • Hidden Markov models

    • Bayes net structures with hidden variables

2


The problem hidden factors l.jpg

Hidden factors

The Problem : Hidden Factors

  • Unobservable / Latent / Hidden

  • Make them as variables

  • Simplicity of the model

3


Simplicity details graph1 l.jpg

162

54

54

486

54

Symptom 1

Symptom 2

Symptom 3

Hidden factors

Simplicity details (graph1)

2

2

2

Smoking

Diet

Exercise

708 Priors !

4


Simplicity details graph2 l.jpg

Heart Disease

54

Hidden factors

Simplicity details (Graph2)

2

2

2

Smoking

Diet

Exercise

78 Priors

6

6

6

Symptom 1

Symptom 2

Symptom 3

5


A solution em algorithm l.jpg

EM Algorithm

A Solution : EM Algorithm

  • Expectation

  • Maximization

6


Principles generalities l.jpg

EM Algorithm

Principles : Generalities

  • Given :

    • Cause (or Factor / Component)

    • Evidence

  • Compute :

    • Probability in connection table

7


Principles the two steps l.jpg

E Step : For each evidence (E),

Use parameters to compute probability distribution

Weighted Evidence :

P(causes/evidence)

M Step : Update the estimates of parameters

Based on weighted evidence

EM Algorithm

Principles : The two steps

Parameters :

P(effects/causes)

P(causes)

8


Principles the e step l.jpg

EM Algorithm

Principles : the E-Step

  • Perception Step

  • For each evidence and cause

    • Compute probablities

    • Find probable relationships

9


Principles the m step l.jpg

EM Algorithm

Principles : the M-Step

  • Learning Step

  • Recompute the probability

    • Cause event / Evidence event

    • Sum for all Evidence events

  • Maximize the loglikelihood

  • Modify the model parameters

10


Formulae notations l.jpg

EM Algorithm

Formulae : Notations

  • Terms

    •  : underlying probability distribution

    • x : observed data

    • z : unobserved data

    • h : current hypothesis of 

    • h’ : revised hypothesis

    • q : a hidden variable distribution

  • Task : estimate  from X

    • E-step:

    • M-step:

11


Formulae the log likelihood l.jpg

EM Algorithm

Formulae : the Log Likelihood

  • L(h) estimates the fitting of the parameter h to the data x with the given hidden variables z :

  • Jensen's inequality for any distribution of hidden states q(z) :

  • Defines the auxiliary function A(q,h):

    • Lower bound on the log likelihood

    • What we want to optimize

12


Formulae the e step l.jpg

EM Algorithm

Formulae : the E-step

  • Lower bound on log likelihood :

  • H(q) entropy of q(z),

  • Optimize A(q,h)

    • By distribute data over hidden variables

13


Formulae the m step l.jpg

EM Algorithm

Formulae : the M-step

  • Maximise A(q,h)

    • By choosing the optimal parameters

  • Equivalent to optimize likelihood

14


Formulae convergence 1 2 l.jpg

EM Algorithm

Formulae : Convergence (1/2)

  • EM increases the log likelihood of the data at every iteration

  • Kullback-Liebler (KL) divergence

    • Non negative

    • Equals 0 iff q(z)=p(z/x,h)

15


Formulae convergence 2 2 l.jpg

Formulae : Convergence (2/2)

  • Likelihood increases at each iteration

  • Usually, EM converges to a local optimum of L

16


Problem of likelihood l.jpg

Problem of likelihood

  • Can be high dimensional integral

  • Latent variables  additional dimensions

  • Likelihood term can be complicated

17


The issue mixture of gaussian l.jpg

Mixture of Gaussians

The Issue : Mixture of Gaussian

  • Unsupervised clustering

    • Set of data points (Evidences)

      • Data generated from mixture distribution

      • Continuous data : Mixture of Gaussians

  • Not easy to handle :

    • Number of parameters is Dimension-squared

18


Gaussian mixture model 2 2 l.jpg

Mixture of Gaussians

Gaussian Mixture model (2/2)

  • Distribution

  • Likelihood of Gaussian Distribution :

  • Likelihood given a GMM :

    • N number of Gaussians

    • wi the weight of Gaussian I

      • All weights positive

      • Total weight = 1

19


Em for gaussian mixture model l.jpg

EM for Gaussian Mixture Model

  • What for ?

    • Find parameters:

      • Weights: wi=P(C=i)

      • Means: i

      • Covariances: i

  • How ?

    • Guess the priority Distribution

      • Guess components (Classes -or Causes)

      • Guess the distribution function

20


Processing em initialization l.jpg

Mixture of Gaussians

Processing : EM Initialization

  • Initialization :

    • Assign random value to parameters

21


Processing the e step 1 2 l.jpg

Mixture of Gaussians

Processing : the E-Step (1/2)

  • Expectation :

    • Pretend to know the parameter

    • Assign data point to a component

22


Processing the e step 2 2 l.jpg

Mixture of Gaussians

Processing : the E-Step (2/2)

  • Competition of Hypotheses

    • Compute the expected values of Pij of hidden indicator variables.

  • Each gives membership weights to data point

  • Normalization

  • Weight = relative likelihood of class membership

23


Processing the m step 1 2 l.jpg

Mixture of Gaussians

Processing : the M-Step (1/2)

  • Maximization :

    • Fit the parameter to its set of points

24


Processing the m step 2 2 l.jpg

Mixture of Gaussians

Processing : the M-Step (2/2)

  • For each Hypothesis

    • Find the new value of parameters to maximize the log likelihood

    • Based on

      • Weight of points in the class

      • Location of the points

    • Hypotheses are pulled toward data

25


Applied formulae the e step l.jpg

Mixture of Gaussians

Applied formulae : the E-Step

  • Find Gaussian for every data point

    • Use Bayes’ rule:

26


Applied formulae the m step l.jpg

Maximize A

For each parameter of h, search for :

Results :

μ*

σ2*

w*

Mixture of Gaussians

Applied formulae : the M-Step

27


Eventual problems l.jpg

Mixture of Gaussians

Eventual problems

  • Gaussian Component shrinks

    • Variance 0

    • Likelihood infinite

  • Gaussian Components merge

    • Same values

    • Share the data points

  • A Solution : reasonable prior values

28


Bayesian networks l.jpg

Other Issues

Bayesian Networks

29


Hidden markov models l.jpg

Other Issues

Hidden Markov models

  • Forward-Backward Algorithm

  • Smooth rather than filter

30


Bayes net with hidden variables l.jpg

Other Issues

Bayes net with hidden variables

  • Pretend that data is complete

  • Or invent new hidden variable

    • No label or meaning

31


Conclusion l.jpg

Conclusion

  • Widely applicable

    • Diagnosis

    • Classification

    • Distribution Discovery

  • Does not work for complex models

    • High dimension

  •  Structural EM

32


  • Login