1 / 32

Model Evaluation and Selection via Prediction

Model Evaluation and Selection via Prediction. Real contributors. Lu Tian (Northwestern University) Tianxi Cai (Harvard University) Hajime Uno (Harvard University, DFCI). Outline . Background and motivation Developing and evaluating prediction rules based on a set of markers for

zeno
Download Presentation

Model Evaluation and Selection via Prediction

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Model Evaluation and Selection via Prediction

  2. Real contributors • Lu Tian (Northwestern University) • Tianxi Cai (Harvard University) • Hajime Uno (Harvard University, DFCI)

  3. Outline • Background and motivation • Developing and evaluating prediction rules based on a set of markers for • Non-censored outcomes • Censored event time outcomes • Evaluating the incremental value of a biomarker over • the entire population • various sub-populations • Incorporating the patient level precision of the prediction • Prediction intervals/sets • Remarks

  4. Regression modeling, Tree classification et al? • Association • Prediction

  5. Model checking? • Goodness of fit test (lack of fit test)? Is p-value a good metric for measuring lack of fit? • Quantitative approach? R-square? Likelihood ratio-type? Need heuristically interpretable distance function? (cost-benefit) • Every model is an approximation to the truth?

  6. Diagnosis Prognosis Treatment Background and Motivation • Personalized medicine: using information about a person’s biological and genetic make up to tailor strategies for the prevention, detection and treatment of disease • Important step: develop prediction rules that can accurately predict the disease outcome or treatment response

  7. Predictor Z Subject Characteristics Biomarkers Genetic Markers Outcome Y Disease status Time to event Treatment Response Background and Motivation • Accurate prediction of disease outcome and treatment response, however, are complex and difficult tasks. • Developing prediction rules involve • Identifying important predictors • Evaluating the accuracy of the prediction • Evaluating the incremental value of new markers

  8. Outcome Y ? Predictor Z CD4week 24 Age, CD4week 0, CD4week 8 RNAweek 0,RNAweek 8 Background and Motivation AIDS Clinical Trial : ACTG320 • Study objective: to compare • 3-drug regimen (n=579): Zidovudine + Lamivudine + Indinarvir • 2-drug regimen (n=577): Zidovudine + Lamivudine • Identify biomarkers for predicting treatment response • How well can we predict the treatment response? • Is RNA needed?

  9. Association Coefficients for RNA significant? Regression Analysis: CD4week 24 Background and Motivation Is RNA needed? Predictors

  10. Coefficient for RNAweek 8highly significant  RNA needed for a more precise prediction of responses?? Background and Motivation AIDS Clinical Trial Regression Coefficient

  11. prediction procedure Does adding RNA improve the prediction? • Prediction rule: based on regression models • The distance between and Y? Background and Motivation Is RNA needed? Y = CD4week 8 Z=Predictors

  12. Developing Prediction RulesBased on a Set of Markers • Regression approach to approximate Y | Z • Non-censored outcome:linear regression • Survival outcome: • Proportional Hazards model (Example: Framingham Risk Score) • Time-specific prediction models: • Regression modeling as a vehicle: • the procedure has to be valid when the imposed statistical model is not the true model!

  13. Y = 0 Y = 1 Developing and Evaluating Prediction Rules • Predict Y with Z based on the prediction model • Evaluate the performance of the prediction by the average “distance” between and Y • The utility or cost to predicting Y as is • The average “distance” is • Examples: • Absolute prediction error: • Total “Cost” of Risk Stratification:

  14. and Evaluating and Comparing Prediction Rules • The performance of the prediction model/rule with can be estimatedby • Prediction Model/Rule Comparison: • Prediction with E(Y | Z) = g1(a’Z) vs E(Y | W) = g2(b’W) • Compare two models/rules by comparing

  15. Variability in the Estimated Prediction Performance Measures • Variability in the prediction errors: • Estimate  = 50, SE = 1? SE = 50? • Inference about D and  = D1 – D2 • Confidence intervals based on large sample approximations to the distribution of

  16. and have the same limiting distribution Bias Correction • Bias issue in the apparent error type estimators • Bias correction via Cross-validation: • Data partition Tk, Vk • For each partition • Obtain based on observations in Tk • Obtain based on observations in Vk • Obtain cross-validated estimator

  17. Example: AIDS Clinical Trial • Objective: identify biomarkers to predict the treatment response • Outcome: Y = CD4week 24 • Predictors Z: Age, CD4week 0, CD4week 8, RNAweek 0,RNAweek 8 • Working Model: E(Y|Z) = ’Z

  18. Example: AIDS Clinical TrialIncremental Value of RNA Estimates 95% C.I. * : Std Error Estimates

  19. Incremental Value of RNA within Various Sub-populations

  20. ExampleBreast Cancer Gene Expression Study • Objective: construct a new classifier that can accurately predict future disease outcome • van’t Veer et al (2002) established a classifier based on a 70-gene profile • good- or poor-prognosis signature based on their correlation with the previously determined average profile in tumors from patients with good prognosis • Classify subjects as • Good prognosis if Gene score > cut-off • Poor prognosis if Gene score < cut-off • van de Vijver et al (2002) evaluated the accuracy of this classifier by using hazard ratios and signature specific Kaplan Meier curves

  21. ExampleBreast Cancer Gene Expression Study • Data consist of 295 Subjects • Outcome T: time to death • Predictors: Lymph-Node Status, Estrogen Receptor Status, gene score • We are interested in • Constructing prediction rules for identify subjects who would survive t-year, Y = I(Tt)=1. • Evaluating the incremental value of the Gene Score.

  22. Example: Breast Cancer DataPredicting 10-year Survival

  23. Evaluating the Prediction RuleBased on Various Accuracy Measures • For a future patient with T0 and Z0, we predict • Classification accuracy measures • Sensitivity • Specificity • Prediction accuracy measures

  24.  Naïve • o Clinical • Clinical + Gene van de Vijver Example: Breast Cancer DataPredicting 10-year Survival

  25. Example: Breast Cancer Data • To compare • Model II: g(a + Node + ER) • Model III: g(a + Node + ER + Gene) • Choosing cut-off values for each model to achieve SE = 69% which is an attainable value for Model II, then • Model II  SP = 0.45, PPV = 0.35, NPV = 0.77 • Model III  SP = 0.75, PPV = 0.54, NPV = 0.85 • 95% CI for the difference in • SP:[0.11, 0.45], PPV: [0.01, 0.24], NPV: [0.06, 0.19]

  26. Prediction IntervalAccounting for the Precision of the Prediction • Based on a prediction model • predict the response • summarize the corresponding population average accuracy • What if the population average accuracy of 70% is not satisfactory? How to achieve 90% accuracy? • What if can predict Y0 more precisely for certain Z0, while on the other hand fails to predict Y0 accurately? • Account for the precision of the prediction? Identify patients would need further assessment?

  27. Classic Rule: Risk of Death < 0.50  Survivor {Y=0} Risk of Death ≥ 0.50  Non-survivor {Y=1} Predicted Risk = 0.51 Predicted Risk = 0.04 {0} {1}

  28. Prediction Interval • To account for patient-levelprediction error, one may instead predict such that • The optimal interval for the population with Z0  is • : estimated conditional density function

  29. Example: Breast CancerStudy • Data: 295 patients • Response: 10 year survival • Predictors: Lymph-Node Status, Estrogen Receptor Status, Gene Score • Model • Possible prediction sets: {}, {0}, {1}, {0,1} • Classic prediction: considers {0}, {1} only.

  30. 90% Prediction Set: {0,1} 90% Prediction Set: {0} Predicted Risk = 0.04 Predicted Risk = 0.51

  31. (0%) 4% (63%) 39% (37%) 57% Example: Breast Cancer Study Prediction Sets Based on Clinical + Gene Score

  32. Remarks • Proper choice of the accuracy/cost measure • Classification accuracy vs predictive values • Utility function: what is the consequence of predicting a subject with outcome Y as • With an expensive or invasive marker • Should it be applied to the entire population? • Is it helpful for a certain sub-population? • Should the cost of the marker be considered when evaluating its value?

More Related