ICA

1. ICA Alphan Altinok

2. Outline • PCA • ICA • Foundation • Ambiguities • Algorithms • Examples • Papers

3. PCA & ICA • PCA • Projects d-dimensional data onto a lower dimensional subspace in a way that is optimal in Σ|x0 – x|2. • ICA • Seek directions in feature space such that resulting signals show independence.

4. PCA • Compute d-dimensional μ (mean). • Compute d x d covariance matrix. • Compute eigenvectors and eigenvalues. • Choose k largest eigenvalues. • k is the inherent dimensionality of the subspace governing the signal and (d – k) dimensions generally contain noise. • Form a d x k matrix A with k columns of eigenvalues. • The representation of data by principal components consists of projecting data into k-dimensional subspace by x = At (x – μ).

5. PCA • A simple 3-layer neural network can form such a representation when trained.

6. ICA • While PCA seeks directions that represents data best in a Σ|x0 – x|2 sense, ICA seeks such directions that are most independent from each other. • Used primarily for separating unknown source signals from their observed linear mixtures. • Typically used in Blind Source Separation problems. ICA is also used in feature extraction.

7. ICA – Foundation • q source signals s1(k), s2(k), …, sq(k) • with 0 means • k is the discrete time index or pixels in images • scalar valued • mutually independent for each value of k • h measured mixture signals x1(k), x2(k), …, xh(k) • Statistical independence for source signals • p[s1(k), s2(k), …, sq(k)] = П p[si(k)]

8. ICA – Foundation • The measured signals will be given by • xj(k) = Σsi(k)aij + nj(k) • For j = 1, 2, …, h, the elements aij are unknown. • Define vectors x(k) and s(k), and matrix A • Observed: x(k) = [x1(k), x2(k), …, xh(k)] • Source: s(k) = [s1(k), s2(k), …, sq(k)] • Mixing matrix: A = [a1, a2, …, aq] • The equation above can be stated in vector-matrix form • x(k) = As(k) + n(k) = Σsi(k)ai + n(k)

9. Ambiguities with ICA • The ICA expansion • x(k) = As(k) + n(k) = Σsi(k)ai + n(k) • Amplitudes of separated signals cannot be determined. • There is a sign ambiguity associated with separated signals. • The order of separated signals cannot be determined.

10. ICA – Using NNs • Prewhitening – transform input vectors x(k) by • v(k) = Vx(k) • Whitening matrix V can be obtained by NN or PCA • Separation (NN or contrast approximation) • Estimation of ICA basis vectors (NN or batch approach)

11. ICA – Fast Fixed Point Algorithm • FFPA converges rapidly to the most accurate solution allowed by the data structure.

12. ICA – Example

13. ICA – Example

14. ICA – Example

15. ICA – Example

16. ICA – Example

17. ICA – Example • BSS of recorded speech and music signals. http://www.cnl.salk.edu/~tewon/ica_cnl.html

18. ICA – Example • Source images • separation demo http://www.open.brain.riken.go.jp/demos/researchBSRed.html

19. ICA – Papers • Hinton – A New View of ICA • Interprets ICA as a probability density model. • Overcomplete, undercomplete, and multi-layer non-linear ICA becomes simpler. • Cardoso – Blind Signal Separation, Statistical Principles • Modelling identifiability. • Contrast functions. • Estimating functions. • Adaptive algorithms. • Performance issues.

20. ICA – Papers • Hyvarinen – ICA Applied to Feature Extraction from Color and Stereo Images • Seeks to extend ICA by contrasting it to the processing done in neural receptive fields. • Hyvarinen – Survey on ICA • Lawrence – Face Recognition, A Convolutional Neural Network Approach • Combines local image sampling, a SOM, and a convolutional NN that provides partial invariance to translation, rotation, scaling, and deformations.

21. ICA – Papers • Sejnowski – Independent Component Representations for Face Recognition • Sejnowski – A Comparison of Local vs Global Image Decompositions for Visual Speechreading • Bartlett – Viewpoint Invariant Face Recognition Using ICA and Attractor Networks • Bartlett – Image Representations for Facial Expression Coding

22. ICA – Links • http://sig.enst.fr/~cardoso/ • http://www.cnl.salk.edu/~tewon/ica_cnl.html • http://nucleus.hut.fi/~aapo/ • http://www.salk.edu/faculty/sejnowski.html