Presented by: Joanna Giforos CS8980: Topics in Machine Learning 9 March, 2006

1. An Introduction to Kernel-Based Learning AlgorithmsK.-R. Muller, S. Mika, G. Ratsch, K. Tsuda and B. Scholkopf Presented by: Joanna Giforos CS8980: Topics in Machine Learning 9 March, 2006

2. Outline • Problem Description • Nonlinear Algorithms in Kernel Feature Space • Supervised Learning: • Nonlinear SVM • Kernel Fisher Discriminant Analysis • Supervised Learning: • Kernel Principle Component Analysis • Applications • Model specific kernels

3. Problem Description • 2-class Classification: estimate a function, using input-output training data such that will correctly classify unseen examples. • i.e., Find a mapping: • Assume: Training and test data are drawn from the same probability distribution

4. Problem Description • A learning machine is a family of functions • For a task of learning two classes f(x,) 2 {-1,1}, 8 x, • Too complex ) Overfitting • Not complex enough ) Underfitting • Want to find the right balance between accuracy and complexity

5. Problem Description • Best is the one that minimizes the expected error: • Empirical Risk (training error): • Remp()! R() as n!1

6. Structural Risk Minimization • Construct a nested family of function classes , with non-decreasing VC dimension. • Let be the solutions of the empirical risk minimization. • SRM chooses the function class and the function such that an upper bound on the generalization error is minimized.

7. Nonlinear Algorithms in Kernel Feature Space • Via a non-linear mapping the data is mapped into a potentially much higher dimensional feature space. • Given this mapping, we can compute scalar products in feature spaces using kernel functions. • does not need to be known explicitly ) every linear algorithm that only uses scalar products can implicitly be executed in by using kernels.

8. Nonlinear Algorithms in Kernel Feature Space:Example

9. Supervised Learning: Nonlinear SVM • Consider linear classifiers in feature space using dot products. • Conditions for classification without training error: • GOAL: Find and b such that the empirical risk and regularization term are minimized. • But we cannot explicitly access w in the feature space, so we introduce Lagrange multipliers, i, one for each of the above constraints.

10. Supervised Learning: Nonlinear SVM • Last class we saw that the nonlinear SVM primal problem is: • Which leads to the dual:

11. Supervised Learning: Nonlinear SVM • Using KKT second order optimality conditions on the dual SVM problem, we obtain: • The solution is sparse in ) many patterns are outside the margin area and the optimal i’s are zero. • Without sparsity, SVM would be impractical for large data sets.

12. Supervised Learning: Nonlinear SVM • The dual problem can be rewritten as: • Where • Since objective function is convex, every local max is a global max, but there can be several optimal solutions (in terms of i) • Once i’s are found using QP solvers, simply plug into prediction rule:

13. Supervised Learning: KFD • Discriminant analysis seeks to find a projection of the data in a direction that is efficient for discrimination. Image from: R.O. Duda, P.E. Hart and D.G. Stork, Pattern Classification, John Wiley & Sons, INC., 2001.

14. Supervised Learning: KFD • Solve Fisher’s linear discriminant in kernel feature space. • Aims at finding linear projections such that the classes are well separated. • How far are the projected means apart? (should be large) • How big is the variance of the data in this direction? (should be small) • Recall, that this can be achieved by maximizing the Rayleigh quotient: • where

15. Supervised Learning: KFD • In kernel feature space , express w in terms of mapped training patterns: • To get: • Where,

16. Supervised Learning: KFD • Projection of a test point onto the discriminant is computed by: • Can solve the generalized eigenvalue problem: • But N and M may be large and non-sparse, can transform KFD into a convex QP problem. • Question – can we use numerical approximations to the eigenvalue problem?

17. Supervised Learning: KFD • Can reformulate as constrained optimization problem. • FD tries to minimize the distance between the variance of the data along the projection whilst maximizing the distance between the means: • This QP is equivalent to J() since • M is a matrix of rank 1 (columns are linearly dependent) • Solutions w in J() are invariant under scaling. )Can fix the distance of the means to some arbitrary, positive value and just minimize the variance.

18. Connection Between Boosting and Kernel Methods • Can show that Boosting maximizes the smallest margin . • Recall, SVM attempts to maximize w • In general, using an arbitrary lp norm constraint on the weight vector leads to maximizing the lq distance between the hyperplane and the training points. • Boosting uses l1 norm • SVM uses l2 norm

19. Unsupervised Methods: Linear PCA • Principal Components Analysis (PCA) attempts to efficiently represent the data by finding orthonormal axes which maximally decorrelate the data • Given centered observations: • PCA finds the principal axes by diagonalizing the covariance matrix • Note that C is positive definite,and thus can be diagonalized with nonnegative eigenvalues.

20. Unsupervised Methods: Linear PCA • Eigenvectors lie in the span of x1, …, xn: • Thus it can be shown that, • But is just a scalar, so all solutions v with   0 lie in the span of x1, …, xn, i.e.

21. Unsupervised Methods: Kernel PCA • If we first map the data into another space, • Then assuming we can center the data, we can write the covariance matrix as: • Which can be diagonalized with nonnegative eigenvalues satisfying:

22. Unsupervised Methods: Kernel PCA • As in linear PCA, all solutions v with  0 lie in the span of (xi), …, (xm) i.e. • Substituting, we get: • Where K is the inner product kernel: • Premultiplying both sides by (xk)T, we finally get:

23. Unsupervised Methods: Kernel PCA • The resulting set of eigenvectors are then used to extract the Principle Components of a test point by:

24. Unsupervised Methods: Kernel PCA • Nonlinearities only enter the computation at two points: • In the calculation of the matrix K • In the evaluation of new points • Drawback of PCA: • For large data sets, storage and computational complexity issues. • Can use sparse approximations of K. • Question: Can we think of other unsupervised methods which can make use of kernels? • Kernel k-means, Kernal ICA, Spectral Clustering

25. Unsupervised Methods: Linear PCA

26. Unsupervised Methods: Kernel PCA

27. Applications • Support Vector Machines and Kernel Fisher Discriminant: • Bioinformatics: protein classification • OCR • Face Recognition • Content based image retrieval • Decision Tree Predictive Modeling • … • Kernel PCA • Denoising • Compression • Visualization • Feature extraction for classification

28. Kernels for Specific Applications • Image Segmentation: Gaussian weighted 2-distance between local color histograms. • Can be shown to be robust for color and texture discrimination • Text classification: Vector Space kernels • Structured Data (strings, trees, etc.): Spectrum kernels • Generative models: P-kernels, Fisher kernels