1 / 24

ROC & AUC, LIFT

ROC & AUC, LIFT. ד"ר אבי רוזנפלד. Introduction to ROC curves. ROC = R eceiver O perating C haracteristic Started in electronic signal detection theory (1940s - 1950s) Has become very popular in biomedical applications, particularly radiology and imaging גם בשימוש בכריית מידע.

ohio
Download Presentation

ROC & AUC, LIFT

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. ROC & AUC, LIFT ד"ר אבי רוזנפלד

  2. Introduction to ROC curves • ROC = Receiver Operating Characteristic • Started in electronic signal detection theory (1940s - 1950s) • Has become very popular in biomedical applications, particularly radiology and imaging • גם בשימוש בכריית מידע

  3. False Positives / Negatives Confusion matrix 1 Confusion matrix 2 FN Actual Actual FP Predicted Predicted Precision (P) = 20 / 50 = 0.4 Recall (P) = 20 / 30 = 0.666 F-measure=2*.4*.666/1.0666=.5

  4. Different Cost Measures • The confusion matrix (easily generalize to multi-class) • Machine Learning methods usually minimize FP+FN • TPR (True Positive Rate): TP / (TP + FN) = Recall • FPR (False Positive Rate): FP / (TN + FP) = Precision

  5. Specific Example People without disease People with disease Test Result

  6. Call these patients “negative” Call these patients “positive” Threshold Test Result

  7. Call these patients “negative” Call these patients “positive” Some definitions ... True Positives Test Result without the disease with the disease

  8. Call these patients “negative” Call these patients “positive” False Positives Test Result without the disease with the disease

  9. Call these patients “negative” Call these patients “positive” True negatives Test Result without the disease with the disease

  10. Call these patients “negative” Call these patients “positive” False negatives Test Result without the disease with the disease

  11. Moving the Threshold: left ‘‘-’’ ‘‘+’’ Test Result without the disease with the disease

  12. ROC curve 100% True Positive Rate (Recall) 0% 100% 0% False Positive Rate (1-specificity)

  13. ההשפעה של שינוי הTHRESHOLD על הגרף

  14. Figure 5.2 A sample ROC curve.

  15. סוגים שונים של ROC גרפים

  16. Area under ROC curve (AUC) • מדד כללי • השטח מתחת לגרךROC • 0.50 הוא מחירה רנדומאלי, 1.0 הוא מושלם.

  17. AUC for ROC curves 100% 100% 100% 100% True Positive Rate True Positive Rate True Positive Rate True Positive Rate 0% 0% 0% 0% 100% 100% 100% 100% 0% 0% 0% 0% False Positive Rate False Positive Rate False Positive Rate False Positive Rate AUC = 100% AUC = 50% AUC = 90% AUC = 65%

  18. Lift Charts • X axis is sample size: (TP+FP) / N • Y axis is TP 80% of responses for 40% of cost Lift factor = 2 Model 40% of responses for 10% of cost Lift factor = 4 Random

  19. Lift factor Lift Value Sample Size

  20. הקשר בין המדדים

  21. לקראת התרגיל...

  22. לחצן ימני על מודל ואזCost / Benefit Analysis for Wood

  23. אפשר לשנות את הסף וגם לראות את הCONFUSION MATRIX

  24. אפשר לראות גם את הLift וגם השפעת מחיר

More Related