newton method for the ica mixture model n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Newton Method for the ICA Mixture Model PowerPoint Presentation
Download Presentation
Newton Method for the ICA Mixture Model

Loading in 2 Seconds...

play fullscreen
1 / 21

Newton Method for the ICA Mixture Model - PowerPoint PPT Presentation


  • 87 Views
  • Uploaded on

Newton Method for the ICA Mixture Model. Jason A. Palmer 1 Scott Makeig 1 Ken Kreutz-Delgado 2 Bhaskar D. Rao 2 1 Swartz Center for Computational Neuroscience 2 Dept of Electrical and Computer Engineering University of California San Diego, La Jolla, CA. Introduction.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Newton Method for the ICA Mixture Model' - caia


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
newton method for the ica mixture model

Newton Method for theICA Mixture Model

Jason A. Palmer1 Scott Makeig1Ken Kreutz-Delgado2 Bhaskar D. Rao2

1 Swartz Center for Computational Neuroscience2 Dept of Electrical and Computer EngineeringUniversity of California San Diego, La Jolla, CA

introduction
Introduction
  • Want to model sensor array data with multiple independent sources — ICA
  • Non-stationary source activity — mixture model
  • Want the adaptation to be computationally efficient — Newton method
outline
Outline
  • ICA mixture model
  • Basic Newton method
  • Positive definiteness of Hessian when model source densities are true source densities
  • Newton for ICA mixture model
  • Example applications to analysis of EEG
ica mixture model toy example
ICA Mixture Model—toy example
  • 3 models in two dimensions, 500 points per model
  • Newton method converges < 200 iterations, natural gradient fails to converge, has difficulty on poorly conditioned models
ica mixture model
ICA Mixture Model
  • Want to model observations x(t), t = 1,…,N, different models “active” at different times
  • Bayesian linear mixture model, h = 1, . . . , M :
  • Conditionally linear given the model, :
  • Samples are modeled as independent in time:
source density mixture model
Source Density Mixture Model
  • Each source density mixture component has unknown location, scale, and shape:
  • Generalizes Gaussian mixture model, more peaked, heavier tails
ica mixture model invariances
ICA Mixture Model—Invariances
  • The complete set of parameters to be estimated is:

h = 1, . . ., M, i = 1, . . ., n, j = 1, . . ., m

  • Invariances: W row norm/source density scale and model centers/source density locations:
basic ica newton method
Basic ICA Newton Method
  • Transform gradient (1st derivative) of cost function using inverse Hessian (2nd derivative)
  • Cost function is data log likelihood:
  • Gradient:
  • Natural gradient (positive definite transform):
newton method hessian
Newton Method – Hessian
  • Take derivative of (i,j)th element of gradient with respect to (k,l)th element of W :
  • This defines a linear transform :
  • In matrix form, this is:
newton method hessian1
Newton Method – Hessian
  • To invert: rewrite the Hessian transformation in terms of the source estimates:
  • Define , , :
  • Want to solve linear equation :
newton method hessian2
Newton Method – Hessian
  • The Hessian transformation can be simplified using source independence and zero mean:
  • This leads to 2x2 block diagonal form:
newton direction
Newton Direction
  • Invert Hessian transformation, evaluate at gradient:
  • Leads to the following equations:
  • Calculate the Newton direction:
positive definiteness of hessian
Positive Definiteness of Hessian
  • Conditions for positive definiteness:
  • Always true for true when model source densities match true densities:

1)

2)

3)

newton for ica mixture model
Newton for ICA Mixture Model
  • Similar derivation applies to ICA mixture model:
convergence rates
Convergence Rates
  • Convergence is really much faster than natural gradient. Works with step size 1!
  • Need correct source density model

log likelihood

iteration

iteration

segmentation of eeg experiment trials
Segmentation of EEG experiment trials

3 models

4 models

trial

trial

time

time

log

likelihood

log

likelihood

iteration

iteration

applications to eeg epilepsy
Applications to EEG—Epilepsy

1 model

5 models

log

likelihood

time

time

log

likelihood

difference

from

single model

time

conclusion
Conclusion
  • We applied method of Amari, Cardoso and Laheld, to formulate a Newton method for the ICA mixture model
  • Arbitrary source densities modeled with non-gaussian source mixture model
  • Non-stationarity modeled with ICA mixture model (multiple mixing matrices learned)
  • It works! Newton method is substantially faster (superlinear). Also Newton can converge when Natural Gradient fails
slide19
Code
  • There is Matlab code available!!
    • Generate toy mixture model data for testing
    • Full method implemented: mixture sources, mixture ICA, Newton
  • Extended version of paper in preparation, with derivation of mixture model Newton updates
  • Download from:

http://sccn.ucsd.edu/~jason

acknowledgements
Acknowledgements
  • Thanks to Scott Makeig, Howard Poizner, Julie Onton, Ruey-Song Hwang, Rey Ramirez, Diane Whitmer, and Allen Gruber for collecting and consulting on EEG data
  • Thanks to Jerry Swartz for founding and providing ongoing support the Swartz Center for Computational Neuroscience
  • Thanks for your attention!