1 / 15

Input Space Regularization Stabilizes Pre-Images For Kernel PCA De-Noising

Input Space Regularization Stabilizes Pre-Images For Kernel PCA De-Noising. Trine Julie Abrahamsen and Lars Kai Hansen IEEE International Workshop on Machine Learning for Signal Processing, September 3, 2009. Outline. Introduction Kernel PCA The Pre-image Problem

denali
Download Presentation

Input Space Regularization Stabilizes Pre-Images For Kernel PCA De-Noising

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Input Space Regularization Stabilizes Pre-Images For Kernel PCA De-Noising Trine Julie Abrahamsen and Lars Kai Hansen IEEE International Workshop on Machine Learning for Signal Processing, September 3, 2009

  2. Outline • Introduction • Kernel PCA • The Pre-image Problem • Input Space Distance Regularization • Conclusions

  3. Introduction Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions Definition of kernel function The Gaussian kernel • Extensive studies showed that current methods suffered from instabilities for very non-linear kernels • We therefore suggest to stabilize the pre-image estimate by including input space distance regularization Mika et al. 1999 Schölkopf et al. 2001

  4. Kernel Principal Component Analysis Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions Linear PCA is performed in feature space. Thus, the first PC can be found as the normal direction, v1 , by All solutions must lie in the span of the training images, hence, The projection of onto the i’thPC can be found as While the projection onto the subspace spanned by the first q PCs is given by Schölkopf et al. 1998

  5. The Pre-Image Problem Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions The pre-image problem = reconstruction of point in input space from feature space point Ill-posed due to properties of the -map. Relax search to find approximate pre-image Common methods seek to minimize the feature space distance where Mika et al. 1999 Schölkopf et al. 1999

  6. Overview of Current Estimation Schemes Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions Mika et al. (1999) Kwok & Tsang (2004) Dambreville et al. (2006)

  7. Input Space Distance Regularization Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions Which is equivalent to minimizing For RBF kernels the costfunction (whichshouldbemaximized) reduces to

  8. Experiments on the USPS Data Set Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions USPS digitsGaussiannoiseadded Mika et al. Input spaceregularization Hull 1994

  9. Experiments on the USPS Data Set Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions Evaluating the stability by confidence intervals on the mean squared error. Kwok & Tsang (2004) Mika et al. (1999) Dambreville et al. (2006) Input Space Reg.

  10. Experiments on the USPS Data Set Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions Evaluating the sensitivity to initialization by the mean pairwise distance between de-noised images when the methods are initialized in random training points.

  11. Conclusions Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions • Several of the current pre-image algorithms were shown to suffer from instabilities • Adding input space distance regularization stabilizes the pre-image with limited sacrifice in terms of de-noising efficiency Future Work • Derive guidelines for choosing λ • Introduce other types of regularization

  12. References

  13. Input Space Regularization Stabilizes Pre-Images For Kernel PCA De-Noising Trine Julie Abrahamsen and Lars Kai Hansen IEEE International Workshop on Machine Learning for Signal Processing, September 3, 2009

  14. Distance distortions for non-linearkernels Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions

  15. Many local minima with almost equal value Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions

More Related