Machine Learning CS 165B Spring 2012

1. Machine LearningCS 165BSpring 2012

2. Course outline • Introduction (Ch. 1) • Concept learning (Ch. 2) • Decision trees (Ch. 3) • Ensemble learning • Neural Networks (Ch. 4) • Linear classifiers • Support Vector Machines • Bayesian Learning (Ch. 6) • Bayesian Networks • Clustering • Computational learning theory Midterm on Wednesday

3. Midterm Wednesday May 2 • Topics (till today’s lecture) • Content • (40%) Short questions • (20%) Concept learning and hypothesis spaces • (20%) Decision trees • (20%) Artificial Neural Networks • Practice midterm will be posted today • Can bring one regular 2-sided sheet & calculator

4. Background on Probability & Statistics • Random variable, sample space, event (union, intersection) • Probability distribution • Discrete (pmf) • Continuous (pdf) • Cumulative (cdf) • Conditional probability • Bayes Rule • P(C ≥ 2 | M = 0) • Independence of random variables • Are C and M independent? • Choose which of two envelopes contains a higher number • Allowed to peak at one of them 3 coins C is the count of heads M =1 iff all coins match

5. Background on Probability & Statistics • Common distributions • Bernoulli • Uniform • Binomial • Gaussian (Normal) • Poisson • Expected value, variance, standard deviation

6. Approaches to classification • Discriminant functions: • Learn the boundary between classes. • Infer conditional class probabilities: • Choose the most probable class What kind of classifier is logistic regression?

7. Nearest Neighbor Decision Tree Nonlinear Functions Linear Functions Discriminant Functions • They can be arbitrary functions of x, such as: Sometimes, transform the data and then learn a linear function

8. High-dimensional data Gene expression Face images Handwritten digits

9. Why feature reduction? • Most machine learning and data mining techniques may not be effective for high-dimensional data • Curse of Dimensionality • Query accuracy and efficiency degrade rapidly as the dimension increases. • The intrinsic dimension may be small. • For example, the number of genes responsible for a certain type of disease may be small.

10. Why feature reduction? • Visualization: projection of high-dimensional data onto 2D or 3D. • Data compression: efficient storage and retrieval. • Noise removal: positive effect on query accuracy.

11. Applications of feature reduction • Face recognition • Handwritten digit recognition • Text mining • Image retrieval • Microarray data analysis • Protein classification

12. Feature reduction algorithms • Unsupervised • Latent Semantic Indexing (LSI): truncated SVD • Independent Component Analysis (ICA) • Principal Component Analysis (PCA) • Supervised • Linear Discriminant Analysis (LDA)

13. Principal Component Analysis (PCA) • Summarization of data with many variables by a smaller set of derived (synthetic, composite) variables • PCA based on SVD • So, look at SVD first

14. Singular Value Decomposition (SVD) • Intuition: find the axis that shows the greatest variation, and project all points to this axis f2 e2 e1 f1 14

15. SVD: mathematical formulation • Let A be an m x n real matrix of m n-dimensional points • SVD decomposition • A = U x L x VT • U(m x m) is orthogonal: UTU = I • V(n x n) is orthogonal: VTV = I • L(m x n) has r positive non-zero singular values in descending order on its diagonal • Columns of U are the orthogonal eigenvectors of AAT (called the left singular vectors of A) • AAT = (U x L x VT ) (U x L x VT )T = U x L x LTx UT = U x L2x UT • Columns of V are the orthogonal eigenvectors of ATA (called the right singular vectors of A) • ATA = (U x L x VT )T (U x L x VT )= V x LTx L x VT = V x L2x VT • L contains the square root of the eigenvalues of AAT (or ATA) • These are called the singular values (positive real) • r is the rank of A, AAT , ATA • U defines the column space of A, V the row space.

16. SVD - example

17. x x = v1 SVD - example • A = ULVT

18. SVD - example • A = ULVT variance (‘spread’) on the v1 axis x x =

19. x x = Dimensionality reduction

20. Dimensionality reduction • set the smallest singular values to zero: x x =

21. Dimensionality reduction x x ~

22. Dimensionality reduction x x ~

23. Dimensionality reduction x x ~

24. Dimensionality reduction ~

25. Dimensionality reduction ‘spectral decomposition’ of the matrix: x x =

26. Dimensionality reduction ‘spectral decomposition’ of the matrix: l1 x x = u1 u2 l2 v1T v2T

27. l1 l2 u1 u2 v1T v2T Dimensionality reduction ‘spectral decomposition’ of the matrix: n = + +... m

28. l1 l2 u1 u2 vT1 vT2 Dimensionality reduction ‘spectral decomposition’ of the matrix: n r terms = + +... m m x 1 1 x n

29. l1 l2 u1 u2 vT1 vT2 Dimensionality reduction approximation / dim. reduction: by keeping the first few terms (how many?) m = + +... n assume: l1 >= l2 >= ...

30. l1 l2 u1 u2 vT1 vT2 Dimensionality reduction A heuristic: keep 80-90% of ‘energy’ (= sum of squares of li’s) m = + +... n assume: l1 >= l2 >= ...

31. Dimensionality reduction • Matrix V in the SVD decomposition • (A = UΛVT ) is used to transform the data. • AV (= UΛ) defines the transformed dataset. • For a new data element x, xV defines the transformed data. • Keeping the first k (k < n) dimensions, amounts to keeping only the first k columns of V.

32. å √åλi2 = = 2 A A [ i , j ] F - £ - A A A B k 2 2 Optimality of SVD • Let A = U L VT • A = ∑ λiuiviT • TheFrobenius norm of an m x n matrix M is • Let Ak = the above summation using the k largest eigenvalues. Theorem: [Eckart and Young] Among all m x n matrices B of rank at most k, we have that: • “Residual” variation is information in A that is not retained. Balancing act between • clarity of representation, ease of understanding • oversimplification: loss of important or relevant information. - £ - A A A B k F F

33. Principal Components Analysis (PCA) • Transfer the dataset to the center by subtracting the means: let matrix A be the result. • Compute the covariance matrix ATA. • Project the dataset along a subset of the eigenvectors of ATA. • Matrix V in the SVD decomposition contains these. • Also known as K-L transform.

34. Principal Component Analysis (PCA) • Takes a data matrix of m objects by n variables, which may be correlated, and summarizes it by uncorrelated axes (principal components or principal axes) that are linear combinations of the original n variables • The first k components display as much as possible of the variation among objects.

35. 2D Example of PCA

36. Configuration is Centered • each variable is adjusted to a mean of zero (by subtracting the mean from each value).

37. Principal Components are Computed • PC 1 has the highest possible variance (9.88) • PC 2 has a variance of 3.03 • PC 1 and PC 2 have zero covariance.

38. PC 1 PC 2 • Each principal axis is a linear combination of the original two variables

39. Feature reduction algorithms • Unsupervised • Latent Semantic Indexing (LSI): truncated SVD • Independent Component Analysis (ICA) • Principal Component Analysis (PCA) • Supervised • Linear Discriminant Analysis (LDA)

40. Course outline • Introduction (Ch. 1) • Concept learning (Ch. 2) • Decision trees (Ch. 3) • Ensemble learning • Neural Networks (Ch. 4) • Linear classifiers • Support Vector Machines • Bayesian Learning (Ch. 6) • Bayesian Networks • Clustering • Computational learning theory

41. Midterm analysis • Grade distribution • Solution to ANN problem • Makeup problem on Wednesday • 20 minutes • 15 points • Bring a calculator

42. Fisher’s linear discriminant • A simple linear discriminant function is a projection of the data down to 1-D. • So choose the projection that gives the best separation of the classes. What do we mean by “best separation”? • An obvious direction to choose is the direction of the line joining the class means. • But if the main direction of variance in each class is not orthogonal to this line, this will not give good separation (see the next figure). • Fisher’s method chooses the direction that maximizes the ratio of between class variance to within class variance. • This is the direction in which the projected points contain the most information about class membership (under Gaussian assumptions)

43. Fisher’s linear discriminant When projected onto the line joining the class means, the classes are not well separated. Fisher chooses a direction that makes the projected classes much tighter, even though their projected means are less far apart.

44. Fisher’s linear discriminant (derivation) Find the best direction w for accurate classification. A measure of the separation between the projected points is the difference of the sample means. If mi is the d-dimensional sample mean from Di given by the sample mean from the projected points Yigiven by the difference of the projected sample means is:

45. Fisher’s linear discriminant (derivation) Define scatterfor the projection: Choose w in order to maximize is called the total within-class scatter. Define scatter matricesSi(i = 1, 2) and Sw by

46. Fisher’s linear discriminant (derivation) We obtain

47. Fisher’s linear discriminant (derivation) where In terms of SB and Sw, J(w) can be written as:

48. Fisher’s linear discriminant (derivation)

49. Fisher’s linear discriminant (derivation) A vector w that maximizes J(w) must satisfy In the case that Sw is nonsingular,

50. Linear discriminant • Advantages: • Simple: O(d) space/computation • Knowledge extraction: weighted sum of attributes; positive/negative weights, magnitudes (credit scoring)