slide1
Download
Skip this Video
Download Presentation
Chapter 7: Model Assessment

Loading in 2 Seconds...

play fullscreen
1 / 42

Model Assessment - PowerPoint PPT Presentation


  • 374 Views
  • Uploaded on

7. Chapter 7: Model Assessment. 7. Chapter 7: Model Assessment. Assessment Types. The Model Comparison tool provides. C. KS. Summary statistics Statistical graphics. ASE. 7. Chapter 7: Model Assessment. Summary Statistics Summary. Prediction Type. Statistic.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Model Assessment' - omer


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

7

Chapter 7: Model Assessment

slide2

7

Chapter 7: Model Assessment

assessment types
Assessment Types

The Model Comparison tool provides

C

KS

Summary statistics

Statistical graphics

ASE

slide4

7

Chapter 7: Model Assessment

summary statistics summary
Summary Statistics Summary

Prediction Type

Statistic

Accuracy / Misclassification

Profit / Loss

KS-statistic

Decisions

ROC Index (concordance)

Gini coefficient

1,2,3,…

Rankings

Average squared error

SBC / Likelihood

p≈E(Y)

^

Estimates

summary statistics summary1
Summary Statistics Summary

Prediction Type

Statistic

Accuracy / Misclassification

Profit / Loss

KS-statistic

Decisions

ROC Index (concordance)

Gini coefficient

1,2,3,…

Rankings

Average squared error

SBC / Likelihood

p≈E(Y)

^

Estimates

summary statistics summary2
Summary Statistics Summary

Prediction Type

Statistic

Accuracy / Misclassification

Profit / Loss

KS-statistic

Decisions

ROC Index (concordance)

Gini coefficient

1,2,3,…

Rankings

Average squared error

SBC / Likelihood

p≈E(Y)

^

Estimates

comparing models with summary statistics
Comparing Models with Summary Statistics
  • This demonstration illustrates the use of the Model Comparison tool, which collects assessment information from attached modeling nodes and enables you to easily compare model performance measures.
slide9

7

Chapter 7: Model Assessment

statistical graphics summary

Decisions

Sensitivity charts

Response rate charts

Statistical Graphics Summary

Prediction Type

Statistic

1,2,3,…

Rankings

p≈E(Y)

^

Estimates

...

prediction ranks

top 40%

Prediction Ranks

Select top n% cases.

...

sensitivity based plots
Sensitivity-Based Plots

Count fraction of primary outcome cases in selection.

top 40%

1.0

sensitivity

0.0

...

false positive fraction
False Positive Fraction

Count fraction of secondary outcome cases in selection.

top 40%

1.0

sensitivity

0.0

0.0

1.0

false positive fraction

(1-specificity)

...

roc chart
ROC Chart

Repeat for all selection fractions.

1.0

sensitivity

0.0

0.0

1.0

false positive fraction

(1-specificity)

...

roc index

1.0

sensitivity

0.0

0.0

1.0

false positive fraction

(1-specificity)

ROC Index

ROC Index

(c-statistic)

...

response rate charts

1.0

0.5

0%

100%

40%

percent selected

(decile)

Response Rate Charts

top 40%

Select top n% cases.

...

cumulative gain
Cumulative Gain

top 40%

Count fraction of cases in selection with primary outcome.

1.0

cumulative

gain

0.5

0%

100%

40%

percent selected

(decile)

...

cumulative gains chart
Cumulative Gains Chart

Repeat for all selection fractions.

1.0

cumulative

gain

0.5

0%

100%

percent selected

(decile)

...

comparing models with statistical graphics
Comparing Models with Statistical Graphics
  • This demonstration illustrates the use of statistical graphics to compare models.
adjusting for separate sampling
Adjusting for Separate Sampling
  • This demonstration illustrates how to adjust for separate sampling in SAS Enterprise Miner.
slide22

7

Chapter 7: Model Assessment

outcome overrepresentation
Outcome Overrepresentation

The sample size is determined not by the total number of cases but by the number of cases in least common outcome (usually primary).

...

separate sampling
Separate Sampling

Cases are sampled separately from each outcome.

Example:

• sample all primary cases

• match each primary case

by one or more secondary

cases

...

separate sampling benefit
Separate Sampling Benefit

• Similar predictive power

with smaller case count

separate sampling consequences
Separate Sampling Consequences

• Must adjust assessment

statistics and graphics

• Must adjust prediction estimates for bias

adjusting for separate sampling continued
Adjusting for Separate Sampling (continued)
  • This demonstration illustrates how to adjust for separate sampling in SAS Enterprise Miner.
creating a profit matrix
Creating a Profit Matrix
  • This demonstration illustrates how to create a profit matrix.
slide29

7

Chapter 7: Model Assessment

profit matrices
Profit Matrices

solicit

ignore

14.86

0

primary

outcome

-0.68

0

secondary

outcome

0

profit distribution

for solicit decision

profit matrices1
Profit Matrices

solicit

ignore

14.86

0

primary

outcome

-0.68

0

secondary

outcome

0

profit distribution

for solicit decision

decision expected profits

choose the larger

^

^

Expected Profit Solicit = 14.86p1 – 0.68p0

Expected ProfitIgnore = 0

Decision Expected Profits

solicit

ignore

14.86

0

primary

outcome

-0.68

0

secondary

outcome

0

...

decision threshold
Decision Threshold

solicit

ignore

14.86

0

primary

outcome

-0.68

0

secondary

outcome

0

decision threshold

^

p1 ≥ 0.68 / 15.54  Solicit

^

p1 < 0.68 / 15.54  Ignore

average profit
Average Profit

solicit

ignore

14.86

0

primary

outcome

-0.68

0

secondary

outcome

0

average profit

Average profit = (14.86NPS– 0.68 NSS ) / N

NPS = # solicited primary outcome cases

NSS = # solicited secondary outcome cases

N= total number of assessment cases

evaluating model profit
Evaluating Model Profit
  • This demonstration illustrates viewing the consequences of incorporating a profit matrix.
viewing additional assessments
Viewing Additional Assessments
  • This demonstration illustrates several other assessments of possible interest.
optimizing with profit optional
Optimizing with Profit (Optional)
  • This demonstration illustrates optimizing your model strictly on profit.
exercise 1
Exercise 1
  • This exercise reinforces the concepts discussed previously.
slide39

7

Chapter 7: Model Assessment

assessment tools review
Assessment Tools Review

Compare model summary statistics and statistical graphics.

Model

Comparison

Create decision data; add prior probabilities and profit matrices.

Data Source

Tune models with average squared error or appropriate profit matrix.

Modeling

Tools

assessment tools review1
Assessment Tools Review

Obtain means and other statistics on data source variables.

StatExplore

slide42

7

Chapter 7: Model Assessment

ad