Resourceful Guide to Sampling Techniques for Kernel Classifiers

1. Sampling Techniques for Kernel Classifiers Vijaya V Saradhi

2. Organization of Presentation • Supervised learning • Introduction • Various methods • Criteria for choosing a supervised learning algorithm? • Bootstrapping • Advantages and Disadvantages Sampling Techniques for Kernel Classifiers

3. Organization of Presentation Contd. • My problem: • Can I use bootstrapping for kernel methods? • Mathematical formulations • Bootstrapped Support Vector Machines (B-SVMs) • Results Sampling Techniques for Kernel Classifiers

4. Supervised Learning

5. Introduction Class I x x x x x x x x x x Decision Surface x x x x x x x x Class II o x x x x o o o o o o o o o o o o o o o Sampling Techniques for Kernel Classifiers

6. Statistical approach to Supervised Learning • Nearest neighbor classifier • Support Vector Machines Sampling Techniques for Kernel Classifiers

7. Bootstrapping • A class of procedures that resample given data • Used in pattern recognition literature as error estimator • Hamamoto et. al for the first time used this technique for nearest neighbor classifier design Sampling Techniques for Kernel Classifiers

8. Nearest Neighbor Classification Class II Class III Test point Class I Class IV Sampling Techniques for Kernel Classifiers

9. Notation • Set of training data points: • Each data point is a vector of D dimensions • There are m number of distinct classes Sampling Techniques for Kernel Classifiers

10. Bootstrapping Algorithm Weighted Average Class I x x x x x x x x x x x x x x x x x x x x x x x o x x x x o o o o o o o o o o o o o o o Class II Sampling Techniques for Kernel Classifiers

11. Outliers Get Eliminated x o o o x o o x xb x o x o o o o o o o o o Outlier Sampling Techniques for Kernel Classifiers

12. Advantages and Drawbacks • Advantages • Eliminates outliers • NNC performance better when used bootstrap samples • Drawbacks • Redundancy increases as r increases • Data points get corrupted Sampling Techniques for Kernel Classifiers

13. Introduction to SVMs Support vectors Class I x x x x x x x x x x Optimal hyper plane x x x x x x x x Class II o x x x x o o o o o o o o o o o o ρ o o o Support vectors Sampling Techniques for Kernel Classifiers

14. XOR Problem XOR Problem Sampling Techniques for Kernel Classifiers

15. Hyper plane parameter Slack variables User parameter Data point and corresponding class label Non linear mapping into feature space from input space Objective Function Formulation Number of training data points Subjected To (ST) Dual variables Dual Formulation Sampling Techniques for Kernel Classifiers

16. Our Problem: Using bootstrapping for kernel classifier’s design • Applied for Support vector machines • Why SVMs? • SVMs prone to pick outliers as potential support vectors (SVs) • What is expected? • Outliers shouldn’t be picked as SVs • Reducing #SVs • Not corrupting bootstrapped data points • Decreasing classification time Sampling Techniques for Kernel Classifiers

17. Bootstrapped SVM Formulation Select data point from set of training data points : Find r nearest neighbors of Compute bootstrapped sample Where is a weight and Sampling Techniques for Kernel Classifiers

18. Objective Function Formulation Subjected To (ST) Sampling Techniques for Kernel Classifiers

19. Dual Problem Formulation Dual variables Sampling Techniques for Kernel Classifiers

20. Theorem • is a positive semi-definite matrix • Proof: Sampling Techniques for Kernel Classifiers

21. H Matrix Rank • Kuhn-Tucker conditions are: • Can be re-written as follows: • Nth Lagrange multiplier is given by • Re-writing above KKT equation we get, Where And Where Sampling Techniques for Kernel Classifiers

22. Properties of Bootstrapped SVM • Theorem: Rank of H matrix decreases as the value of number of neighbors ‘r’ increases • Comment on generalization Sampling Techniques for Kernel Classifiers

23. Artificial Data Set Class 2 Class 1 Outliers Sampling Techniques for Kernel Classifiers

24. Artificial Data Set SVs with r = 3 Class 2 Support Vectors Class 1 Outlier as support vectors? Sampling Techniques for Kernel Classifiers

25. r Vs Number of SVs Variation of SVs with value of r Sampling Techniques for Kernel Classifiers

26. r Vs Classification Accuracy Effect of r on Classification Accuracy Sampling Techniques for Kernel Classifiers

27. Iris Data Set Outliers Class 3 Outliers Class 1 Class 2 Sampling Techniques for Kernel Classifiers

28. Iris SVs with r = 3 Outliers Class 3 Class 2 Outliers Support vectors Class 1 Sampling Techniques for Kernel Classifiers

29. r Vs Classification Accuracy Performance on real data set Sampling Techniques for Kernel Classifiers

30. Focus on Future • Intended to work on the following • Bootstrapping for Bayes Point Machine and other kernel classifiers • For support vector regression problem • Analyzing the possibilities of using other sampling techniques for kernel classifiers Sampling Techniques for Kernel Classifiers

31. Bootstrapping Select data point from set of training data points Find r nearest neighbors of : Compute bootstrapped sample Where is a weight and Sampling Techniques for Kernel Classifiers

32. Classification Based on Concept Hierarchy Vijaya V Saradhi

33. Organization of Presentation • Text Classification • Introduction • Motivation • Approach • Our Problem Classification Based on Concept Hierarchy

34. Classification of Documents Based on Concept Hierarchy • Documents might not always be classified into a single category • Use Domain Knowledge for classification of documents • An example of concept hierarchy: • ACM Computing Classification System • An example for our problem Classification Based on Concept Hierarchy

35. Arithmetic & Logic I/O Data Communications Performance and Reliability Performance of Systems Metrics Performance Concept Hierarchy Doc on: Performance Analysis ACM Computing Classification System General Literature Hardware Computer Systems Organization Software Data Theory of Computing Mathematics of Computing Information Systems Computing methodologies Computer Applications Computing Milieux Classification Based on Concept Hierarchy

36. Concept Hierarchy document Domain Knowledge document Black Box Classification Based on Concept Hierarchy

37. Problems to Address • What is the nature of Domain Knowledge • Ontology • Trees • Forests • Graphs • How to use Domain Knowledge in classification? • What are the strategies to come up with multi-category outputs? • What is the error measure? • What is classification measure? Classification Based on Concept Hierarchy

38. References • Anil K Jain, Richard C. Dubes and Chaur-Chin Chen, “Bootstrap Techniques for Error Estimation”, IEEE PAMI Vol. 5, No. 5, Sep 1987 • Y. Hamamoto and S. Tomita, “Bootstrap Technique for Nearest Neighbor Classifier Design”, IEEE PAMI Vol. 19, No. 1, Jan 1997 • B. Efron, “Bootstrap Methods: Another Look at the Jackknife”, Annual Statistics, Vol. 7, pp: 1-26, 1979 • Cortes C and Vapnik V. N, “Support Vector Network”, Machine Learning, 20, pp: 1 – 25 • Massimilaiano Pontil and Alessandro Verri, “Properties of Support Vector Machines”, Neural Computation, Vol. 10, pp. 955 – 974, 1998. • Rudd. M. Bolle, Nalini K. Ratha and Sharath Pankanti, “Error Analysis of Pattern Recognition Systems – the Subset Bootstrap”, Compute Vision and Image Understanding, Vol. 93, pp: 1 – 33, 2004 • Ralf Herbrich, Thore Graepel and Colin Campbell, “Bayes Point Machine”, Journal of Machine Learning Research, Vol. pp. 245-279, Aug 2001. • [Authors], “Bootstrapping Neural Networks”, Neural Computation, 1998, Vol. ?? pp. ?? - ?? • Anil K. Jain, Robert P. W. Dulin and Jianchang Mao, “Statistical Pattern Recogntion: A Review”, IEEE PAMI Vol. 22, No. 1, Jan 2000 Sampling Techniques for Kernel Classifiers

39. Questions? Thank You Classification Based on Concept Hierarchy