1 / 9

Dimensionality Reduction with Principal Component Analysis

Learn about Principal Component Analysis (PCA), a technique used to find a new feature space that adequately describes the original feature space. Discover how to calculate basis vectors, minimize error, and utilize eigenvectors and eigenvalues.

quigleye
Download Presentation

Dimensionality Reduction with Principal Component Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ECE471-571 – Pattern RecognitionLecture 7 – Dimensionality Reduction – Principal Component Analysis Hairong Qi, Gonzalez Family Professor Electrical Engineering and Computer Science University of Tennessee, Knoxville http://www.eecs.utk.edu/faculty/qi Email: hqi@utk.edu

  2. Pattern Classification Statistical Approach Non-Statistical Approach Supervised Unsupervised Decision-tree Basic concepts: Baysian decision rule (MPP, LR, Discri.) Basic concepts: Distance Agglomerative method Syntactic approach Parameter estimate (ML, BL) k-means Non-Parametric learning (kNN) Winner-takes-all LDF (Perceptron) Kohonen maps NN (BP) Mean-shift Support Vector Machine Deep Learning (DL) Dimensionality Reduction FLD, PCA Performance Evaluation ROC curve (TP, TN, FN, FP) cross validation Stochastic Methods local opt (GD) global opt (SA, GA) Classifier Fusion majority voting NB, BKS

  3. Principal Component Analysis or K-L Transform • How to find a new feature space (m-dimensional) that is adequate to describe the original feature space (d-dimensional). Suppose m<d x2 y1 y2 x1

  4. K-L Transform (1) • Describe vector x in terms of a set of basis vectors bi. • The basis vectors (bi) should be linearly independent and orthonormal, that is,

  5. K-L Transform (2) • Suppose we wish to ignore all but m (m<d) components of y and still represent x, although with some error. We will thus calculate the first m elements of y and replace the others with constants Error:

  6. K-L Transform (3) • Use mean-square error to quantify the error

  7. K-L Transform (4) • Find the optimal ai to minimize e2 • Therefore, the error is now equal to

  8. K-L Transform (5) • The optimal choice of basis vectors is the eigenvectors of Sx • The expansion of a random vector in terms of the eigenvectors of the covariance matrix is referred to as the Karhunen-Loeve expansion, or the “K-L expansion” • Without loss of generality, we will sort the eigenvectors bi in terms of their eigenvalues. That is l1 >= l2 >= … >= ld. Then we refer to b1, corresponding to l1, as the “major eigenvector”, or “principal component”

  9. Summary • Raw data  covariance matrix  eigenvalue  eigenvector  principal component • How to use error rate?

More Related