1 / 6

Classification and Prediction

Classification and Prediction. Additional Evaluation Metrics. Lift Curve. From test set, order instances according to their probability of being correctly predicted Rank rules (J48 , JRIP) Rank probabilities (Logistic regression) Plot true positive percentages

lot
Download Presentation

Classification and Prediction

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Classification and Prediction Additional Evaluation Metrics

  2. Lift Curve • From test set, order instances according to their probability of being correctly predicted • Rank rules (J48 , JRIP) • Rank probabilities (Logistic regression) • Plot true positive percentages • Compare against baseline (overall prediction accuracy for the class)

  3. ROC Curve • Receiver Operating Characteristic • Useful to find instances that have high proportion of positives • Plots counts of true positives versus false positives for a given class from test set. • The closer to upper left corner the better

  4. ROC Curve in WEKA Left click classifier results and choose Threshold Curve

  5. Cost Considerations • Cost / Profit Matrix • Bank example: decide whether to offer Personal Equity Plan (PEP) • True Positive: Payoff is $1000/customer • True Negative: $0 • False Positive: -$10/customer • False negative: -$1000/customer (opp. Cost)

  6. Bank Example • Run J48, JRIP and Logistic • Use Percentage Split 66/34 for test • Compute Total Expected Cost/Profit for each method. Assume 1000 customers in the consideration set. • Generate ROC curves for each method, for both “YES” and “NO” predictions.

More Related