1 / 45

Support Vector Machines

Support Vector Machines. Pattern Recognition Sergios Theodoridis Konstantinos Koutroumbas Second Edition A Tutorial on Support Vector Machines for Pattern Recognition Data Mining and Knowledge Discovery, 1998 C. J. C. Burges. Separable Case. Maximum Margin Formulation. Separable Case.

Download Presentation

Support Vector Machines

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Support Vector Machines Pattern Recognition Sergios TheodoridisKonstantinos Koutroumbas Second Edition A Tutorial on Support Vector Machines for Pattern RecognitionData Mining and Knowledge Discovery, 1998C. J. C. Burges

  2. Separable Case

  3. Maximum Margin Formulation

  4. Separable Case • Label the training data • Hyperplane satisfy • w:normal to the hyperplane • |b|/||w||:perpendicular distance from the hyperplane to the origin • d+ (d-):margin

  5. Separable Case positive example negative example d+ d-

  6. Separable Case • Suppose that all the training data satisfy the following constraints • These can be combines into one set of inequalities • Distance of a point from a hyperplane class 1 class 2

  7. Separable Case • Having a margin of • Task • compute the parameter w, b of the hyperplane maximize

  8. Separable Case • Karush-Kuhn-Tucker (KKT) conditions • : vector of the Langrange multiplier • : Langrangian function

  9. Separable Case • Wolfe dual representation form

  10. Image Categorization by Learning and Reasoning with Regions Yixin ChenUniversity of New Orleans James Z. WangThe Pennsylvania State University Journal of Machine Learning Research 5 (2004) (Submitted 7/03; Revised 11/03; Published 8/04)

  11. Introduction • Automatic image categorization • Difficulties • Variable & uncontrolled image conditions • Complex and hard-to-describe objects in image • Objects occluding other objects • Applications • Digital libraries, Space science, Web searching, Geographic information systems, Biomedicine, Surveillance and sensor system, Commerce, Education

  12. Overview • Give a set of labeled images, can a computer program learn such knowledge or semantic concepts form implicit information of objects contained in image?

  13. Related Work • Multiple-Instance Learning • Diverse Density Function (1998) • MI-SVM (2003) • Image Categorization • Color Histograms (1998-2001) • Subimage-based Methods (1994-2004)

  14. Motivation • Correct categorization of an image depends on identifying multiple aspects of the image • Extension of MIL→A bag must contain a number of instances satisfying various properties

  15. A New Formulation of Multiple-Instance Learning • Maximum margin problem in a new feature space defined by the DD function • DD-SVM • In the instance feature space, a collection of feature vectors, each of which is called an instance prototype, is determined according to DD

  16. A New Formulation of Multiple-Instance Learning • Instance prototype: • A class of instances (or regions) that is more likely to appear in bags (or images) with the specific label than in the other bags • Maps every bag to a point in bag feature space • Standard SVMs are the trained in the bag feature space

  17. Outline • Image segmentation & feature representation • DD-SVM, and extension of MIL • Experiments & result • Conclusions & future work

  18. Image Segmentation • Partitions the image into non-overlapping blocks of size 4x4 pixels • Each feature vector consists of six features • Average color components in a block • LUV color space • Square root of the second order moment of wavelet coefficients in high-frequency bands

  19. HL LL k, l 2x2 coefficients LH HH Image Segmentation • Daubechies-4 wavelet transform • Moments of wavelet coefficients in various frequency bands are effective for representing texture (Unser, 1995)

  20. Image Segmentation • k-means algorithm: cluster the feature vectors into several classes with every class corresponding to one “region” • Adaptively select N by gradually increasing N until a stopping criterion is met (Wang et al. 2001)

  21. Segmentation Results

  22. Image Representation • :the mean of the set of feature vectors corresponding to each region Rj • Shape properties of each region • Normalized inertia of order 1, 2, 3 (Gersho, 1979)

  23. Image Representation • Shape feature of region Rj as • An image Bi • Segmentation: {Rj : j = 1, …, Ni} • Feature vectors: { xij : j = 1, …, Ni} 9-dimensional feature vector

  24. An extension of Multiple-Instance Learning • Maximum margin formulation of MIL in a bag feature space • Constructing a bag feature space • Diverse density • Learning instance prototypes • Computing bag features

  25. Maximum Margin Formulation of MIL in a Bag Feature Space • Basic idea of new MIL framework: • Map every bag to a point in a new feature space, named the bag feature space • To train SVMs in the bag feature space subject to

  26. Constructing a Bag Feature Space • Clues for classifier design: • What is common in positive bags and does not appear in the negative bags • Instance prototypes computed from the DD function • A bag feature space is then constructed using the instance prototypes

  27. Diverse Density (Maron and Lozano-Perez, 1998) • A function defined over the instance space • DD value at a point in the feature space • The probability that the point agrees with the underlying distribution of positive and negative bags

  28. Diverse Density • It measures a co-occurrence of instances from different (diverse) positive bags

  29. Learning Instance Prototype • An instance prototype represents a class of instances that is more likely to appear in positive bags than in negative bags • Learning instance prototypes then becomes an optimization problem • Finding local maximizers of the DD function in a high-dimensional

  30. Learning Instance Prototype • How do we find the local maximizers? • Start an optimization at every instance in every positive bag • Constraints: • Need to be distinct from each other • Have large DD values

  31. Computing Bag Features • Let be the collection of instance prototypes • Bag features,

  32. Experimental Setup for Image Categorization • COREL Corp: 2,000 images • 20 image categories • JPEG format, size 384*256 (256*384) • Each category are randomly divided into a training set and a test set (50/50) • SVMLight[Joachims, 1999] software is used to train the SVMs

  33. Sample Images (COREL)

  34. Image Categorization Performance • 5 random test sets, 95% confidence intervals • The images belong to Cat.0 ~ Cat.9 Chapelle et al., 1999 14.8% Andrews et al., 2003 6.8%

  35. Image Categorization Experiments

  36. Sensitivity to Image Segmentation • k-means clustering algorithmwith 5 different stopping criteria • 1,000 images for Cat.0 ~ Cat.9

  37. 13.8% 9.5% 11.7% 6.8% 27.4% Robustness to Image Segmentation

  38. Robustness to the Number of Categories in a Data Set 81.5% 6.8% 67.5% 12.9

  39. Difference in Average Classification accuracies

  40. Sensitivity to the Size of Training Images

  41. Sensitivity to the Diversity of Training Images Varies

  42. MUSK Data Sets

  43. Speed • 40 minutes • Training set of 500 images (4.31 regions per image) • Pentium III 700MHz PC running the Linux operating system • Algorithm is implemented in Matlab, C programming language • The majority is spent on learninginstance prototypes

  44. Conclusions • A region-based image categorization method using an extension of MIL → DD-SVM • Image → collection of regions → k-means alg. • Image → a point in a bag feature space (defined by a set of instance prototypes learned with the DD func.) • SVM-based image classifiers are trained in the bag feature space • DD-SVM outperforms two other methods • DD-SVM generates highly competitive results on MUSK data set

  45. Future Work • Limitations • Region naming (Barnard et al., 2003) • Texture dependence • Improvement • Image segmentation algorithm • DD function • Scene category can be a vector • Semantically-adaptive searching • Art & biomedical images

More Related