Principal Component Analysis

1. PrincipalComponentAnalysis Step by Step Walk Through Paul Biliniski

2. Purpose • Find patterns in data with many dimensions • Reduces the number of dimensions, analysis becomes easier • ONLY WORKS ON SQUARE MATRICES

3. Mathematical Concepts • Measures of Spread in 1 Dimension • Standard Deviation – spread of data from mean • Variance • Measure of Spread in 2 Dimensions • Covariance – Variance between 2 data sets, to see if they change at similar rates; sign is important • Covariance Matrix • Matrix of all of the covariances between each pair of data sets • Eigenvector • Transformation vector that creates a reflection of a data set onto itself • Eigenvalue • Amount by which the original vector is scaled

4. Step 1: Data, Subtract Means • Find the Mean of each component of the data set • Subtract that mean from each of the components • Mean of Height: 173.9366667 Mean of OFC: 57.59333333

5. Step 1: Data Graphed Clearly, there is a linear relationship

6. Step 1: Subtract Means

7. Step 2: Covariance Matrix • Calculate the covariance matrix • The diagonal should be the variance in each data set, the anti-diagonal should be the covariance • All positive values tells us that we expect to see that as data set 1 increases, data set 2 also increases. Verified by graph

8. Step 3: Calculate Eigens • Eigenvector x times matrix A equals Eigenvalue (Λ) times x • Ax = Λx • The eigenvalue is found with: • det (A – ΛI) = 0 • It is the determinant (performs a linear transformation of vector space) of the original matrix – Λ in the diagonals • So with the original matrix: • The Eigenvalue can be found with the equation: • Use the Quadratic to solve

9. Step 3: Calculate Eigens • So for our situation, we use the 2x2 matrix of the covariances: • So the determinant for this is: • (104.901023 – Λ) * (2.791678161 – Λ) - 15.36128736^2 =0 • So, the eigenvalues are 0.54 and 107.161! det

10. Step 3: Calculate Eigens • With the Eigens solved, we can now solve for the vectors in the null space, getting our vector to Row Echelon form; first just use one of the eigenvalues as the Λ • For row echelon form, get the item in the second row(15.36), first column to equal – 105.44, or super close to it. • So, multiply the second row by 6.86!

11. Step 3: Calculate Eigens • Row 1 stays the same, Row 2 = Row 1 – 6.86*(Row2) • So, do the multiplication: • 105.44X +15.36Y = 0 • We can define 105.44X = S, the variable S. This means that 15.36Y = 0.14S • The Vector for the value 0.54 is • Apply this same technique to the eigenvalue of 107, and get

12. Step 3: Calculate Eigens • Now lets see how our Eigens look on our graph of subtracted means data… looks like a line of best fit, reasonable.

13. Step 3: Calculate Eigens • The eigenvalue with the highest value is considered the principle component of the data set. Set up a matrix with your two eigenvalues, so you can transform the data. • Note that column 1 is the eigenvector associated with our 107 eigenvalue, the bigger one • Now, we get back to the data…

14. Step 4: Transform Data • Using the 2x2 matrix of eigenvectors, multiply each matrix of the data (each should be a 1 column 2 row matrix, X value on top of Y value) • Continue this for EVERY set of points…

15. Step 4: Transform Data • This is what the data should look like after this eigenvector multiplication step.

16. Step 5: Define Noise • One of the axes is going to be the noise that we assume occurs as a result of sampling. Choose one, in this case the Y values.

17. Step 6: Getting back the Data • Use the new noise-less data to plot your new points. Multiply the data without noise by the eigen matrix • Repeat for all of your points. Now, add the means of each back to the data:

18. Step 7: Victory • Plot your new data points. They are now one line without noise. This is now the new axis against which you can plot another component. Keep adding variables into each component until there is no longer a linear relationship. That will show you what components cause the variation in your data.