Download
1 / 42

Chapter 7: Model Assessment - PowerPoint PPT Presentation


  • 374 Views
  • Updated On :

7. Chapter 7: Model Assessment. 7. Chapter 7: Model Assessment. Assessment Types. The Model Comparison tool provides. C. KS. Summary statistics Statistical graphics. ASE. 7. Chapter 7: Model Assessment. Summary Statistics Summary. Prediction Type. Statistic.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Chapter 7: Model Assessment' - omer


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

7

Chapter 7: Model Assessment


7

Chapter 7: Model Assessment


Assessment types
Assessment Types

The Model Comparison tool provides

C

KS

Summary statistics

Statistical graphics

ASE


7

Chapter 7: Model Assessment


Summary statistics summary
Summary Statistics Summary

Prediction Type

Statistic

Accuracy / Misclassification

Profit / Loss

KS-statistic

Decisions

ROC Index (concordance)

Gini coefficient

1,2,3,…

Rankings

Average squared error

SBC / Likelihood

p≈E(Y)

^

Estimates


Summary statistics summary1
Summary Statistics Summary

Prediction Type

Statistic

Accuracy / Misclassification

Profit / Loss

KS-statistic

Decisions

ROC Index (concordance)

Gini coefficient

1,2,3,…

Rankings

Average squared error

SBC / Likelihood

p≈E(Y)

^

Estimates


Summary statistics summary2
Summary Statistics Summary

Prediction Type

Statistic

Accuracy / Misclassification

Profit / Loss

KS-statistic

Decisions

ROC Index (concordance)

Gini coefficient

1,2,3,…

Rankings

Average squared error

SBC / Likelihood

p≈E(Y)

^

Estimates


Comparing models with summary statistics
Comparing Models with Summary Statistics

  • This demonstration illustrates the use of the Model Comparison tool, which collects assessment information from attached modeling nodes and enables you to easily compare model performance measures.


7

Chapter 7: Model Assessment


Statistical graphics summary

Decisions

Sensitivity charts

Response rate charts

Statistical Graphics Summary

Prediction Type

Statistic

1,2,3,…

Rankings

p≈E(Y)

^

Estimates

...


Statistical graphics prediction ranks

Apply model to validation data.

Statistical Graphics – Prediction Ranks

validation data

...


Prediction ranks

top 40%

Prediction Ranks

Select top n% cases.

...


Sensitivity based plots
Sensitivity-Based Plots

Count fraction of primary outcome cases in selection.

top 40%

1.0

sensitivity

0.0

...


False positive fraction
False Positive Fraction

Count fraction of secondary outcome cases in selection.

top 40%

1.0

sensitivity

0.0

0.0

1.0

false positive fraction

(1-specificity)

...


Roc chart
ROC Chart

Repeat for all selection fractions.

1.0

sensitivity

0.0

0.0

1.0

false positive fraction

(1-specificity)

...


Roc index

1.0

sensitivity

0.0

0.0

1.0

false positive fraction

(1-specificity)

ROC Index

ROC Index

(c-statistic)

...


Response rate charts

1.0

0.5

0%

100%

40%

percent selected

(decile)

Response Rate Charts

top 40%

Select top n% cases.

...


Cumulative gain
Cumulative Gain

top 40%

Count fraction of cases in selection with primary outcome.

1.0

cumulative

gain

0.5

0%

100%

40%

percent selected

(decile)

...


Cumulative gains chart
Cumulative Gains Chart

Repeat for all selection fractions.

1.0

cumulative

gain

0.5

0%

100%

percent selected

(decile)

...


Comparing models with statistical graphics
Comparing Models with Statistical Graphics

  • This demonstration illustrates the use of statistical graphics to compare models.


Adjusting for separate sampling
Adjusting for Separate Sampling

  • This demonstration illustrates how to adjust for separate sampling in SAS Enterprise Miner.


7

Chapter 7: Model Assessment


Outcome overrepresentation
Outcome Overrepresentation

The sample size is determined not by the total number of cases but by the number of cases in least common outcome (usually primary).

...


Separate sampling
Separate Sampling

Cases are sampled separately from each outcome.

Example:

• sample all primary cases

• match each primary case

by one or more secondary

cases

...


Separate sampling benefit
Separate Sampling Benefit

• Similar predictive power

with smaller case count


Separate sampling consequences
Separate Sampling Consequences

• Must adjust assessment

statistics and graphics

• Must adjust prediction estimates for bias


Adjusting for separate sampling continued
Adjusting for Separate Sampling (continued)

  • This demonstration illustrates how to adjust for separate sampling in SAS Enterprise Miner.


Creating a profit matrix
Creating a Profit Matrix

  • This demonstration illustrates how to create a profit matrix.


7

Chapter 7: Model Assessment


Profit matrices
Profit Matrices

solicit

ignore

14.86

0

primary

outcome

-0.68

0

secondary

outcome

0

profit distribution

for solicit decision


Profit matrices1
Profit Matrices

solicit

ignore

14.86

0

primary

outcome

-0.68

0

secondary

outcome

0

profit distribution

for solicit decision


Decision expected profits

choose the larger

^

^

Expected Profit Solicit = 14.86p1 – 0.68p0

Expected ProfitIgnore = 0

Decision Expected Profits

solicit

ignore

14.86

0

primary

outcome

-0.68

0

secondary

outcome

0

...


Decision threshold
Decision Threshold

solicit

ignore

14.86

0

primary

outcome

-0.68

0

secondary

outcome

0

decision threshold

^

p1 ≥ 0.68 / 15.54  Solicit

^

p1 < 0.68 / 15.54  Ignore


Average profit
Average Profit

solicit

ignore

14.86

0

primary

outcome

-0.68

0

secondary

outcome

0

average profit

Average profit = (14.86NPS– 0.68 NSS ) / N

NPS = # solicited primary outcome cases

NSS = # solicited secondary outcome cases

N= total number of assessment cases


Evaluating model profit
Evaluating Model Profit

  • This demonstration illustrates viewing the consequences of incorporating a profit matrix.


Viewing additional assessments
Viewing Additional Assessments

  • This demonstration illustrates several other assessments of possible interest.


Optimizing with profit optional
Optimizing with Profit (Optional)

  • This demonstration illustrates optimizing your model strictly on profit.


Exercise 1
Exercise 1

  • This exercise reinforces the concepts discussed previously.


7

Chapter 7: Model Assessment


Assessment tools review
Assessment Tools Review

Compare model summary statistics and statistical graphics.

Model

Comparison

Create decision data; add prior probabilities and profit matrices.

Data Source

Tune models with average squared error or appropriate profit matrix.

Modeling

Tools


Assessment tools review1
Assessment Tools Review

Obtain means and other statistics on data source variables.

StatExplore


7

Chapter 7: Model Assessment