1 / 28

Independent Component Analysis (ICA) and Factor Analysis (FA)

Independent Component Analysis (ICA) and Factor Analysis (FA). Amit Agrawal. Outline. Motivation for ICA Definitions, restrictions and ambiguities Comparison of ICA and FA with PCA Estimation Techniques Applications Conclusions. Motivation.

vin
Download Presentation

Independent Component Analysis (ICA) and Factor Analysis (FA)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Independent Component Analysis (ICA)and Factor Analysis(FA) Amit Agrawal ENEE 698A Seminar

  2. Outline • Motivation for ICA • Definitions, restrictions and ambiguities • Comparison of ICA and FA with PCA • Estimation Techniques • Applications • Conclusions ENEE 698A Seminar

  3. Motivation • Method for finding underlying components from multi-dimensional data • Focus is on Independent and Non-Gaussian components in ICA as compared to uncorrelated and gaussian components in FA and PCA ENEE 698A Seminar

  4. Cocktail-party Problem • Multiple speakers in room (independent) • Multiple sensors receiving signals which are mixture of original signals • Estimate original source signals from mixture of received signals • Can be viewed as Blind-Source Separation as mixing parameters are not known ENEE 698A Seminar

  5. ICA Definition • Observe n random variables which are linear combinations of n random variables which are mutually independent • In Matrix Notation, X = AS • Assume source signals are statistically independent • Estimate the mixing parameters and source signals • Find a linear transformation of observed signals such that the resulting signals are as independent as possible ENEE 698A Seminar

  6. Restrictions and Ambiguities • Components are assumed independent • Components must have non-gaussian densities • Energies of independent components can’t be estimated • Sign Ambiguity in independent components ENEE 698A Seminar

  7. Gaussian and Non-Gaussian components • If some components are gaussian and some are non-gaussian. • Can estimate all non-gaussian components • Linear combination of gaussian components can be estimated. • If only one gaussian component, model can be estimated ENEE 698A Seminar

  8. Why Non-Gaussian Components • Uncorrelated Gaussian r.v. are independent • Orthogonal mixing matrix can’t be estimated from Gaussian r.v. • For Gaussian r.v. estimate of model is up to an orthogonal transformation • ICA can be considered as non-gaussian factor analysis ENEE 698A Seminar

  9. ICA vs. PCA • PCA • Find smaller set of components with reduced correlation. Based on finding uncorrelated components • Needs only second order statistics • ICA • Based on finding independent components • Needs higher order statistics ENEE 698A Seminar

  10. Factor Analysis • Based on a generative latent variable model • where Y is zero mean, gaussian and uncorrelated • N is zero mean gaussian noise • Elements of Y are the unobservable factors • Elements of A are called factor loadings • In practice, have a good estimated of covariance of X • Solve for A and Noise Covariance • Variables should have high loadings on a small number of factors ENEE 698A Seminar

  11. FA vs. PCA • PCA • Not based on generative model, although can be derived from one • Linear transformation of observed data based on variance maximization or minimum mean-square representation • Invertible, if all components are retained • FA • Based on generative model • Value of factors cannot be directly computed from observations due to noise • Rows of Matrix A (factor loadings) are NOT proportional to eigenvectors of covariance of X • Both are based on second order statistics due to the assumption of gaussianity of factors ENEE 698A Seminar

  12. Whitening as Preprocessing for ICA • Elements are uncorrelated and have unit variances • Decorrelation followed by scaling • Any orthogonal transformation of whitened r.v. will be white • So whitening gives components up to orthogonal transformation. • Useful as preprocessing step for ICA. • Search is restricted to orthogonal mixing matrices • Parameters reduced from ENEE 698A Seminar

  13. ICA Techniques • Maximization of non-gaussianity • Maximum Likelihood Estimation • Minimization of Mutual Information • Non-Linear Decorrelation ENEE 698A Seminar

  14. ICA by Maximization of non-gaussianity • S is linear combination of observed signals X • By Central Limit Theorem (CLT), sum is more gaussian. So maximize non-gaussianity of mixture of observed signals • How to measure non-gaussianity • Kurtosis • Negentropy ENEE 698A Seminar

  15. Non-gaussianity using Kurtosis • Kurtosis = • Kurtosis = 0 for gaussian r.v. • Kurtosis < 0 sub-gaussian e.g. uniform • Kurtosis > 0 super-gaussian e.g. laplacian • Simple to compute • Whiten observed data x to get z; z = Vx • Maximize absolute value (or square) of kurtosis of wTz subject to ||w|| = 1 ENEE 698A Seminar

  16. Gradient Algorithm using Kurtosis • Start from some initial w • Compute direction in which absolute value of kurtosis of y = wTz is increasing • Move vector w in that direction (Gradient Descent) ENEE 698A Seminar

  17. FastICA Algorithm • Usual problems with gradient descent such as learning rate, slow convergence • FastICA is a fixed point algorithm • Main Idea • At convergence, gradient must point in direction of true w or • Use • Normalize w to unit vector after each step ENEE 698A Seminar

  18. Non-gaussianity using Negentropy • Kurtosis is sensitive to outliers. Not a robust measure • Negentropy based on information-theoretic concepts • Underlying Principle • Gaussian r.v. has largest entropy among all r.v.’s of equal variance. So gaussian r.v. is most random • Negentropy = H(Ygauss) – H(Y) where Ygauss is a gaussian r.v. with same variance as Y • Negentropy >= 0. Is an optimal estimator of nongaussianity • Computationally difficult. Require an estimate of pdf. Approximate using higher order statistics ENEE 698A Seminar

  19. Negentropy… • This is same as square of kurtosis if first term is zero ( for r.v. with symmetric pdf ) • Suffers from same problems as kurtosis • Generalize higher order cumulant information • Replace y^2 and y^3 by some other functions • where v is a gaussian r.v. with zero mean and unit variance and G is a non-quadratic function • Useful choices ENEE 698A Seminar

  20. ICA using Maximum Likelihood (ML) Estimation • Express the Likelihood as a function of parameters of model, i.e. the elements of the mixing matrix ENEE 698A Seminar

  21. ML estimation • But the likelihood is also a function of densities of independent components. Hence semi-parametric estimation. • Estimation is easier • If prior information on densities is available. • Likelihood is a function of mixing parameters only • If the densities can be approximated by a family of densities which are specified by a limited no. of parameters ENEE 698A Seminar

  22. ICA by Minimizing Mutual Information • In many cases, we can’t assume that data follows ICA model • This approach doesn’t assume anything about data • ICA is viewed as a linear decomposition that minimizes the dependence measure among components or finding maximally independent components • Mutual Information >=0 and zero if and only if the variables are statistically independent ENEE 698A Seminar

  23. Mutual Information • Minimization of mutual information is equivalent to maximizing the sum of nongausssianities of estimates of independent components • But in maximizing sum of nongausssianities, the estimates are forced to be uncorrelated ENEE 698A Seminar

  24. ICA by Non-Linear Decorrelation • Independent components can be found as nonlinearly uncorrelated linear combinations • Non-Linear Correlation is defined as where f and g are two functions with at least one of them being non-linear • Y1 and Y2 are independent if and only if • Assume Y1 and Y2 are non-linearly decorrelated i.e. • A sufficient condition for this is that Y1 and Y2 are independent and for one of them the non-linearity is an odd function such that has zero mean ENEE 698A Seminar

  25. Applications • Feature Extraction • Taking windows from signals and considering them as multi-dimensional signals • Medical Applications • Removing artifacts (due to muscle activity) from Electroencephalography (EEG) and MEG data in Brain Imaging • Removal of artifacts from cardio graphic signals • Telecommunications: CDMA signal model can be cast in form of a ICA model • Econometrics: Finding hidden factors in financial data ENEE 698A Seminar

  26. Conclusions • General purpose technique • Formulated as estimation of a generative model • Problem can be simplified by whitening of data • Estimated techniques include ML estimation, non-gaussianity maximization, minimization of mutual information • Can be applied in diverse fields ENEE 698A Seminar

  27. References • A. Hyvarinen, J. Karhunen, E. Oja, Independent Component Analysis, Wiley Interscience, 2001 • A. Hyvarinen, E. Oja, Independent Component Analysis: A tutorial, April 1999 (http://www.cis.hut.fi/projects/ica/) ENEE 698A Seminar

  28. Thank You ENEE 698A Seminar

More Related