Incomplete Graphical Models. Nan Hu. Outline . Motivation K-means clustering Coordinate Descending algorithm Density estimation EM on unconditional mixture Regression and classification EM on conditional mixture A general formulation of EM Algorithm. K-means clustering.
Problem: Given a set of observations
how to group them into a set of K clustering, supposing the value of K is given.
by setting the partial derivatives to zero
Problem: If the given sample data demonstrate multimodal densities, how to estimate the true density?
Fit a single density with this bimodal case.
Although algorithm converges, the results bear little relationship to the truth.
Multinomial node taking on one of K values
Assign a density model for each subpopulation, overall density is
Suppose we observed the latent variables ,
the data set becomes completely observed, the likelihood is defined as the complete log likelihood
We treat the as random variables and take expectations conditioned on X and .
Note are binary r.v., where
Use this as the “best guess” for , we have
Expected complete log likelihood
For regression and classification
The relationship between X and Z can be modeled in a discriminative classification way, e.g. softmax func.
Latent variable Z, multinomial node taking on one of K values
Where is the logistic function:
Complete log likelihood :
Use expectation as the “best guess”, we have
Summary of EM algorithm for conditional mixture
Suppose is observed, the ML estimate is
However, is in fact not observed
Complete log likelihood
Incomplete log likelihood
Note:is the upper bound of
non-negative and uniquely minimized at
Alternating minimization algorithm