1 / 32

The Expected Performance Curve Samy Bengio, Johnny Mariéthoz, Mikaela Keller

The Expected Performance Curve Samy Bengio, Johnny Mariéthoz, Mikaela Keller. MI – 25. oktober 2007 Kresten Toftgaard Andersen. Introduction to the paper. By Samy Bengio, Johnny Mariéthoz and Mikaela Keller, 2005 For machine learning community and researchers ect, who need to compare models.

marlon
Download Presentation

The Expected Performance Curve Samy Bengio, Johnny Mariéthoz, Mikaela Keller

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Expected Performance CurveSamy Bengio, Johnny Mariéthoz, Mikaela Keller MI – 25. oktober 2007 Kresten Toftgaard Andersen

  2. Introduction to the paper • By Samy Bengio, Johnny Mariéthoz and Mikaela Keller, 2005 • For machine learning community and researchers ect, who need to compare models. Content of the paper: • Introduces ROC curves very briefly. • Points out some risks when using ROC curves for comparing different classifying models. • Argues that ROC curves can be misleading by showing some results. • The authors contributes with a so called “Expected Performance Curve”, and argues why it is better for comparing models. • Extends EPC with confidence intervals and statistical difference tests. • Concludes the paper summarizing their contribution and by listing strenghts and weaknesses of ROC and EPC. • Acknowledgement and references

  3. Content • Motivation • Introduce terminology and notation, define problem. • Introduce ROC curves • Example: how to calculate a ROC • Present arguments of why ROC curves should be used with great care • Introduce EPC • Continue example showing how to calculate an EPC • Present arguments of why EPC might be better than ROC • Confidence interval • My opinion • Discussion

  4. Motivation ROC analysis is an important why to compare binary classifier models. Can be used to select optimal models and discard suboptimal models. Area of use: • Medicine (diagnostic testing, evaluate evidence-based medicine approaches) • Epidemiology (factors affecting health, evaluate optimal treatment approaches) • Radiology (radar signals, evaluate new radiology techniques ) • Psychology (signal detection, assess human detection of weak signals) • Machine Learning (evaluation of machine learning techniques) • …

  5. Definition of 2-class classifiers • Definition of 2-class classification problems: • Apply function and associated threshold on a seperate test data set (true class must be known) and count the outcome.

  6. Confusion matrix Given a 2 class classifier and an instance, there are four possible outcomes: • TP: instance is positive and is classified as positive • FN: instance is positive and is classified as negative • TN: instance is negative and is classified as negative • FN: instance is negative and is classified as positive

  7. Perfomance metrics • Selected measure is a pair which is generically called V1 and V2. • V1 and V2 can be calculated in many ways depending on the situation. All are simple combinations of TP, TN, FP and FN. • Exact calculation of V1 and V2 is not important in this paper.

  8. Perfomance metrics • An unique measure generically called V combines V1 and V2 • V can also be calculated in several ways depending on the situation (Half Total Error Rate)

  9. What is a ROC curve? ROC • Abbreviation for ”Receiver Operating Characteristics”. • Technique for visualizing, organizing and selecting classifiers based on their performance. • ROC can both be presented as a graph or a curve. Classifiers • Discrete classifiers (decision trees, rule sets ect.) • Probabilistic classifiers (Naive Bayes, neural network ect.) • Varying a threshold for a probabilistic classifier will trace a curve (ROC) Following example will show this.

  10. Example

  11. Example

  12. Example Threshold

  13. Example Threshold

  14. Example Threshold

  15. Example Threshold

  16. Example

  17. Example

  18. ROC curves • BEP = Breake Even Point • BEP corresponds to the threshold nearst to a solutions such that V1 = V2 • The selected threshold have a significant impact on the model. • The threshold represents the a trade-off between giving importance to V1 or V2.

  19. Potential risk of using ROC • Each point corresponds to a particular setting of the threshold. But in “real applications” the thresholds need to be decided before seeing the test set. • Normally the threshold is found by searching for the BEP using some equation. • Possibility of mismatch because training set is different from the test set. • Situations may occur where the optimal threshold found be using the training set, doesn’t correspond to the optimal threshold on the test set. • One parameter, the threshold, is tuned using the training set. Potential risk to expect that the training error reflects the general error. “Real applications often suffer from an additional mismatch between training and test conditions”. • Risk of a different trade-off (V1, V2) in test set. • ROC curves does not take the risk of a mismatch into account. • This probalility should be reflected in the procedure when calculating the performance curve.

  20. Potential risk of using ROC ROC’s of two real models for a Text-Independent Speaker Verifacation task. Looking at the curves only model B seems to be better than model A. Looking at the thresholds, A is actually the best model.

  21. Expected performance curve • EPC present a range of possible expected performance on the test set. • The calculation takes into account the possible mismatch while estimating the desired threshold. • A parameter alpha is used to estimate the possible missmatch of the threshold. Framework: Paremetric performance measure: C( V1(θ, D), V2(θ, D);  ) Depends on: The parameter , V1 and V2 computed on some data D for the threshold θ. Example: C( V1(θ, D), V2(θ, D);  ) = C( Precision(θ, D), Recall(θ, D) ;  ) = - ( Precision(θ, D) + (1 - )  Recall(θ, D)) Procedure: Vary  inside a reasonable range and for each  estimate θthat minimizes C(-,-;) on a development set and then use the obtained θ to compute V on the test set. At last plot V with respect to .

  22. EPC Algorithm

  23. Example

  24. Example

  25. Example

  26. Example

  27. Example

  28. Example of an typical EPC Alpha > 0,5 = more importance to false acceptance errors Alpha < 0,5 = more importance to false rejection errors

  29. EPC in real applications Expected Performance Curves for person authentication, where one wants to trade-off false acceptance rates with false rejection rates. Expected Performance Curves for text categorization, where one wants to trade-off precision and recall and print the F1 measure.

  30. Confidence Interval • Confidence intervals are used to indicate the reliability of an estimate

  31. My opinion • The authors got a point and the idea is good. • Good for comparing models… • …but hard to read much from EPC, ROC more informative. • Cumbersome to compute EPC. • Useful… maybe? • Apparently only used by the authors?

  32. End of Line • Questions • Discussion

More Related