1 / 42

Chapter 7: Model Assessment

7. Chapter 7: Model Assessment. 7. Chapter 7: Model Assessment. Assessment Types. The Model Comparison tool provides. C. KS. Summary statistics Statistical graphics. ASE. 7. Chapter 7: Model Assessment. Summary Statistics Summary. Prediction Type. Statistic.

omer
Download Presentation

Chapter 7: Model Assessment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 7 Chapter 7: Model Assessment

  2. 7 Chapter 7: Model Assessment

  3. Assessment Types The Model Comparison tool provides C KS Summary statistics Statistical graphics ASE

  4. 7 Chapter 7: Model Assessment

  5. Summary Statistics Summary Prediction Type Statistic Accuracy / Misclassification Profit / Loss KS-statistic Decisions ROC Index (concordance) Gini coefficient 1,2,3,… Rankings Average squared error SBC / Likelihood p≈E(Y) ^ Estimates

  6. Summary Statistics Summary Prediction Type Statistic Accuracy / Misclassification Profit / Loss KS-statistic Decisions ROC Index (concordance) Gini coefficient 1,2,3,… Rankings Average squared error SBC / Likelihood p≈E(Y) ^ Estimates

  7. Summary Statistics Summary Prediction Type Statistic Accuracy / Misclassification Profit / Loss KS-statistic Decisions ROC Index (concordance) Gini coefficient 1,2,3,… Rankings Average squared error SBC / Likelihood p≈E(Y) ^ Estimates

  8. Comparing Models with Summary Statistics • This demonstration illustrates the use of the Model Comparison tool, which collects assessment information from attached modeling nodes and enables you to easily compare model performance measures.

  9. 7 Chapter 7: Model Assessment

  10. Decisions Sensitivity charts Response rate charts Statistical Graphics Summary Prediction Type Statistic 1,2,3,… Rankings p≈E(Y) ^ Estimates ...

  11. Apply model to validation data. Statistical Graphics – Prediction Ranks validation data ...

  12. top 40% Prediction Ranks Select top n% cases. ...

  13. Sensitivity-Based Plots Count fraction of primary outcome cases in selection. top 40% 1.0 sensitivity 0.0 ...

  14. False Positive Fraction Count fraction of secondary outcome cases in selection. top 40% 1.0 sensitivity 0.0 0.0 1.0 false positive fraction (1-specificity) ...

  15. ROC Chart Repeat for all selection fractions. 1.0 sensitivity 0.0 0.0 1.0 false positive fraction (1-specificity) ...

  16. 1.0 sensitivity 0.0 0.0 1.0 false positive fraction (1-specificity) ROC Index ROC Index (c-statistic) ...

  17. 1.0 0.5 0% 100% 40% percent selected (decile) Response Rate Charts top 40% Select top n% cases. ...

  18. Cumulative Gain top 40% Count fraction of cases in selection with primary outcome. 1.0 cumulative gain 0.5 0% 100% 40% percent selected (decile) ...

  19. Cumulative Gains Chart Repeat for all selection fractions. 1.0 cumulative gain 0.5 0% 100% percent selected (decile) ...

  20. Comparing Models with Statistical Graphics • This demonstration illustrates the use of statistical graphics to compare models.

  21. Adjusting for Separate Sampling • This demonstration illustrates how to adjust for separate sampling in SAS Enterprise Miner.

  22. 7 Chapter 7: Model Assessment

  23. Outcome Overrepresentation The sample size is determined not by the total number of cases but by the number of cases in least common outcome (usually primary). ...

  24. Separate Sampling Cases are sampled separately from each outcome. Example: • sample all primary cases • match each primary case by one or more secondary cases ...

  25. Separate Sampling Benefit • Similar predictive power with smaller case count

  26. Separate Sampling Consequences • Must adjust assessment statistics and graphics • Must adjust prediction estimates for bias

  27. Adjusting for Separate Sampling (continued) • This demonstration illustrates how to adjust for separate sampling in SAS Enterprise Miner.

  28. Creating a Profit Matrix • This demonstration illustrates how to create a profit matrix.

  29. 7 Chapter 7: Model Assessment

  30. Profit Matrices solicit ignore 14.86 0 primary outcome -0.68 0 secondary outcome 0 profit distribution for solicit decision

  31. Profit Matrices solicit ignore 14.86 0 primary outcome -0.68 0 secondary outcome 0 profit distribution for solicit decision

  32. choose the larger ^ ^ Expected Profit Solicit = 14.86p1 – 0.68p0 Expected ProfitIgnore = 0 Decision Expected Profits solicit ignore 14.86 0 primary outcome -0.68 0 secondary outcome 0 ...

  33. Decision Threshold solicit ignore 14.86 0 primary outcome -0.68 0 secondary outcome 0 decision threshold ^ p1 ≥ 0.68 / 15.54  Solicit ^ p1 < 0.68 / 15.54  Ignore

  34. Average Profit solicit ignore 14.86 0 primary outcome -0.68 0 secondary outcome 0 average profit Average profit = (14.86NPS– 0.68 NSS ) / N NPS = # solicited primary outcome cases NSS = # solicited secondary outcome cases N= total number of assessment cases

  35. Evaluating Model Profit • This demonstration illustrates viewing the consequences of incorporating a profit matrix.

  36. Viewing Additional Assessments • This demonstration illustrates several other assessments of possible interest.

  37. Optimizing with Profit (Optional) • This demonstration illustrates optimizing your model strictly on profit.

  38. Exercise 1 • This exercise reinforces the concepts discussed previously.

  39. 7 Chapter 7: Model Assessment

  40. Assessment Tools Review Compare model summary statistics and statistical graphics. Model Comparison Create decision data; add prior probabilities and profit matrices. Data Source Tune models with average squared error or appropriate profit matrix. Modeling Tools

  41. Assessment Tools Review Obtain means and other statistics on data source variables. StatExplore

  42. 7 Chapter 7: Model Assessment

More Related