1 stat 231 a l yuille fall 2004
This presentation is the property of its rightful owner.
Sponsored Links
1 / 14

1. Stat 231. A.L. Yuille. Fall 2004 PowerPoint PPT Presentation


  • 75 Views
  • Uploaded on
  • Presentation posted in: General

1. Stat 231. A.L. Yuille. Fall 2004. Practical Issues with SVM. Handwritten Digits: US Post Office, MNIST Datasets. No Handout. For people seriously interested in this material see Learning with Kernels by B. Schoelkopf and A.J. Smola. MIT Press. 2002. 2. Practical SVM.

Download Presentation

1. Stat 231. A.L. Yuille. Fall 2004

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


1 stat 231 a l yuille fall 2004

1. Stat 231. A.L. Yuille. Fall 2004

  • Practical Issues with SVM.

  • Handwritten Digits: US Post Office, MNIST Datasets.

  • No Handout. For people seriously interested in this material see Learning with Kernels by B. Schoelkopf and A.J. Smola. MIT Press. 2002.

Lecture notes for Stat 231: Pattern Recognition and Machine Learning


2 practical svm

2. Practical SVM

  • Support Vector Machines first showed major success for

    the task of handwritten digit/character recognition.

  • US Post-Office database. MNIST database.

  • Issues with real problems:

    (1) Multiclassification – not just yes/no.

    (2) Large datasets – quadratic programming impractical.

    (3) Invariance in data. Prior Knowledge.

    (4) Which kernels? When are kernels generalizing?

Lecture notes for Stat 231: Pattern Recognition and Machine Learning


3 multiclassification

3. Multiclassification.

  • Two solutions for M classes.

    (A): One versus Rest. For each class, i =1,...,M construct a binary classifier

    where

    (n no. of data samples).

    Classify

    Comment: simple, but heuristic.

Lecture notes for Stat 231: Pattern Recognition and Machine Learning


4 multiclassification

4. Multiclassification

(B): Hyperplanes for each class label

Data and Slack Variables:

Quadratic Programming:

Lecture notes for Stat 231: Pattern Recognition and Machine Learning


5 multiclass and data size

5. Multiclass and Data Size

  • Empirically, methods A and B give similar quality results. Method (B) is more attractive. But the solution is more computationally intensive. This leads to issue:

  • (2) Large Datasets.

    The Quadratic Programming problem is most easily formulated in terms of the dual

    For large datasets n is enormous. Quadratic Programming is computationally expensive.

Lecture notes for Stat 231: Pattern Recognition and Machine Learning


6 large datasets

6. Large Datasets

  • Chunking is the favored solution. Observe that the

    will be non-zero only for the support vectors.

  • “Chunk” the training data into k sets of size n/k.

  • Train on these k sets and keep the support vectors for these sets.

  • Then train on the combined support vectors of the k sets.

  • Note: need to check the original data to make sure that it is correctly classified. If not, add more support vectors.

Lecture notes for Stat 231: Pattern Recognition and Machine Learning


7 large datasets

7. Large Datasets.

  • Chunking is successful and computationally efficient provided the number of support vectors is small.

  • This happens if there is a hyperplane/hypersurface with large margin separating the classes.

  • It is harder if data from the classes overlap – e.g. when there are a large number of data points which need non-zero slack variables (i.e. support vectors).

  • In either case, more support vectors are needed for the combined multiclass, case (B), than for the heuristic (A).

  • Note: other approximate methods for when chunking fails.

Lecture notes for Stat 231: Pattern Recognition and Machine Learning


8 invariances and priors

8. Invariances and Priors

  • (3) Invariances in the Data.

    Recognizing handwritten digits. The classifier should be insensitive to small changes to the data.

    For example, small rotations, and small translations.

Lecture notes for Stat 231: Pattern Recognition and Machine Learning


9 invariances and priors

9. Invariances and Priors

  • Virtual Support Vectors (VSV).

  • Strategy:

    (i) Train on the original dataset to get support vectors.

    (ii) Generate artificial examples by applying the transformations to the support vectors.

    (iii) Train again on the “virtual examples” generated by (ii).

Lecture notes for Stat 231: Pattern Recognition and Machine Learning


10 virtual support vectors

10. Virtual Support Vectors

Lecture notes for Stat 231: Pattern Recognition and Machine Learning


11 invariances and priors

11. Invariances and Priors

  • Other methods include:

    (i) Hand-designing features which are invariant to the problem.

    (ii) Training on virtual examples, before constructing support vectors. (Computationally expensive).

    (iii) Designing criteria allowing for data transformations.

    (iv) Learning features which are invariants (TPA.)

  • In general, it is best to select your input features using as much prior knowledge as you have about the problem.

Lecture notes for Stat 231: Pattern Recognition and Machine Learning


12 mnist results

12. MNIST Results

  • MNIST dataset of handwritten digits.

  • Summary of Results: page 341 S.S.

  • Best classifier uses a polynomial

  • 8 VSV means 8 invariance samples per data. (1 pixel translation, plus rotation).

  • MNIST Dataset has 600,000 handwritten digits.

  • LeNet is a multilayer network with special training plus boosting,

Lecture notes for Stat 231: Pattern Recognition and Machine Learning


13 mnist results

13. MNIST Results

Lecture notes for Stat 231: Pattern Recognition and Machine Learning


14 summary

14. Summary.

  • Applying SVM’s to real problems requires:

  • Multiclass: Method (A) One-versus-Rest, (B) Full solutions.

  • Computational practically – Chunking by dividing dataset into

    subsets, and using the support vectors from each set.

  • Invariance – generate new samples by apply translations to

    support vectors to generate virtual support vectors.

  • Very successful on the DNIST and US Postal Office datasets. Simpler than the LeNet approach (closest rival.)

Lecture notes for Stat 231: Pattern Recognition and Machine Learning


  • Login