1 / 32

Time Series and Risk Adjusted Control Charts

Time Series and Risk Adjusted Control Charts. Michael E. Matheny, MD, MS, MPH TVHS Veteran’s Administration Division of General Internal Medicine Department of Biomedical Informatics Department of Biostatistics Vanderbilt University Medical Center Nashville , TN. Objectives.

elom
Download Presentation

Time Series and Risk Adjusted Control Charts

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Time Series and Risk AdjustedControl Charts Michael E. Matheny, MD, MS, MPH TVHS Veteran’s Administration Division of General Internal Medicine Department of Biomedical Informatics Department of Biostatistics Vanderbilt University Medical Center Nashville, TN

  2. Objectives • Industrial Statistical Process Control • Important statistical considerations for medical domain SPC use • Highlighted methods with examples • Risk Adjusted Statistical Process Control • Propensity Score Matched Proportional Difference • Risk Adjusted Sequential Probability Ratio Testing • Maximized Sequential Probability Ratio Testing • Conclusions

  3. Statistical Process ControlIndustrial Methods • Most well-defined sequential methods were in industrial processes • Shewhart charts • Exponentially weighted moving average charts • Cumulative sum charts • Sequential probability ratio testing

  4. Statistical Process ControlIndustrial Characteristics • Homogeneity of each observation is assumed • The observed data sets the expectations • This method looks for changes in proportion over time • Process consistency is monitored • Evaluating a medical product with a stable but “clinically unacceptable” adverse event rate would not generate an alarm. Source: Oakland, JS. Statistical Process Control, 5th Ed. Butterworth-Heinemann, 2003

  5. Statistical Process Controlperiodic p chart 3σ

  6. Statistical Process ControlMedical Domain Adoption Barriers • No comparison to pre-existing standard • New medical products are held to the adverse event standards of previous medical products • Process level heterogeneity • Changing standard of care over time • Introduction of new medical products • Patient level heterogeneity • Demographic factors • Chronic illness variation • Acute illness variation • Care tailored to individual patient

  7. Medical SPCCommon Reference Data Selections • Prospective concurrent controls • closely related medical product (same class or indication) • distantly related medical product (different class/indication) • Retrospective external controls • closely related medical product • distantly related medical product • Retrospective phase 3 clinical trial data • usually non-representative population • small sample size • limited outcomes

  8. Statistical Process ControlPeriodic -> Cumulative • The most basic p chart is only one time period, but this can be extended by analyzing all of the available data in each time period or sample size increment. • Strengths • Increasing sample size over time allows better detection of trends • Weaknesses • Repeated measurements inflate Type I error • Extended cumulative analysis can miss newly evolving changes from weighting of old data

  9. Statistical Process ControlRisk Adjustment • Risk adjustment is required to address observational heterogeneity • Two methods to address these requirements: • Apply prediction model to observed data developed from prior baseline or comparison data • Exposed-Unexposed (Case-control) Matching

  10. Statistical Process ControlRisk Adjustment • When a prediction model is used to perform risk adjusted SPC, the performance of the model drives the overall result. • Evaluation of discrimination and calibration, with emphasis on individual risk prediction performance • Awareness that individual prediction performance degrades quickly as the source data ages or when applied to a systematically different population

  11. Statistical Process ControlRandomized Controlled Trial Example • OPUS (TIMI-16) • Setting: 888 Hospitals in 27 Countries • Population: Control (3421), Intervention (6867) • Intervention: Oral 2b/3a Inhibitor vsPlacebo • Outcome: 30 day mortality • Trial stopped early • CLARITY (TIMI-28) • Setting: 313 Hospitals in 23 Countries • Population: Control (1739), Intervention (1751) • Intervention: Oral Anti-Platelet Agent vsPlacebo • Outcome: Major Bleeding • DSMB concerned, but trial did not stop early Source: Matheny et al. Medical Decision Making. 2009;29:247-56.

  12. Risk Adjusted SPCOPUS RCT LR-SPC p chart AUC 0.70 (0.62-0.78) χ2(8df)=8.38 (p=0.40)

  13. Risk Adjusted SPCCLARITY RCT LR-SPC p chart AUC 0.80 (0.74-0.87) χ2(8df)=10.5 (p=0.23)

  14. Risk Adjusted SPCSummary • More sensitive than trial analysis • Type I Error Inflation • Early detection methods are more tolerant of Type I error than Type II error • Not interpretable if no events in baseline

  15. Propensity Score MatchingCaliper Matching Method • ±(0.2 - 0.6)*SD of the estimated logit • ±(0.01 - 0.05) of estimated probability • Recent study simulation found that 0.2*SD and 0.02/0.03 were comparable and were optimal parameter selections • Time window limitation to limit process level changes between matched cases and controls Source: Austin, PC. Biometrical J. 2009;51:171-184.

  16. Propensity Score Matched SPCVascular Closure Device Surveillance Example • Setting • Brigham & Women’s Hospital (01/2002 – 12/2004) patients undergoing percutaneous coronary intervention (3947) • Exposure: vascular closure device • Outcome: Retroperitoneal Hemorrhage (25) • Baseline: Stanford University Data (2000 – 2004) • Propensity Score (62 variables) By calendar quarter with PS caliper of ±0.03 Source: Matheny et al. AMIA AnnuSymp Proc. 2007;518-522.

  17. Propensity Score Matched SPCVascular Closure Device Surveillance Example Matched 92.4% (1,144/1,238) by calendar quarter AUC 0.70

  18. Root Cause AnalysisConfounding By Indication Critical risk factor was location of access site, which was not captured in the routine ACC-NCDR (or Mass-DAC) datasets. Automated surveillance is hypothesis generating – all alerts REQUIRE detailed review and confirmation Source: Tiroch K et al. American Journal of Cardiology. 2008;102:1473-6.

  19. Propensity Score Matched SPCSummary • Risk adjustment method (again) drives overall analysis performance • Evaluate relative propensity score distribution • Report and evaluate matched and unmatched groups, can limit interpretation of results • Limited to concurrent analysis where baseline exposure data is coexisting • Requires sustained clinical care with both ‘exposed’ and ‘unexposed’ medical products • Type I error inflation remains

  20. Risk Adjusted SPCType I Error Adjustment Use of an alpha spending function to adjust per-period alerting boundary thresholds O’Brien-Fleming Lan-DeMets

  21. Risk Adjusted SPCTaxusExpress DES versus Other DES Surveillance Massachusetts PCI 2004-2007 Evaluation ofcumulative post-procedure myocardial infarction rate for new drug eluting stent as compared with propensity matched control DES. Propensity score matching resulted in 81.5% of 18,277 Taxus Express2 devices matched and analyzed. Source: Resnic et al. JAMA 2010;304(18):2019-2027.

  22. Another SPC PathwaySequential Probability Ratio Testing • Retrospective control data is more common that concurrent controls • Dominance of new medical product • National or regional external standard • Sequential framework that accounts for repeated measurements

  23. Sequential Probability Ratio TestRisk Adjusted Methods Summary Formal framework for incorporating ά and β error Specify Odds Ratio of event rate elevation detection desired Risk Adjustment using a binary outcome risk model to adjust the cumulative log odds Source: Spiegelhalter, et al. International Journal of Quality Healthcare 2003;15:7-13

  24. Setting: Brigham & Women’s Hospital (01/2002 – 10/2006) Exposure: All Operators (18) who performed PCI on patients (8750) Outcome: Inpatient Mortality (125) (1.4%) Baselines: National ACC-NCDR Data Local BWH Data Risk Adjusted SPRTInterventional Cardiology Operator Assessment Example Source: Matheny et al. American Heart Journal. 2008;155:114-20.

  25. Risk Adjusted SPRTOperator Outlier Detection

  26. Risk Adjusted SPRT Operator Root Cause Analysis Evaluated each fatal case for the operator in question, and noted a high compassionate use rate compared to other operators After excluding patients that were not considered candidates for CABG or were clearly documented as extremely high-risk compassionate care, operator did not exceed mortality rate expectations

  27. Risk-Adjusted SPRTMA CABG Mortality Institutional Outlier Detection Massachusetts CABG Mortality 2002-2007 2 known institutional outliers in 5 years of data (4 and 1) Both methods detected all true positives, SPRT had 1 false positive Source: Matheny et al. BMC Med. Inform. Dec. Mak. 2011;11:75.

  28. Maximized SPRTMethodology • Formal framework for incorporating ά and β error as well as repeated measurements • Composite hypothesis for event detection where any odds ratio > 1.0 • Poisson version uses prediction model from retrospective data for risk adjustment • Case-Control version uses propensity score matching for prospective concurrent analysis Source: Li, Kulldorff.Statist. Med. 2010;29:284-295

  29. Maximized SPRTH1N1 Vaccine Surveillance (Poisson) Source: Greene et al. Pharmacoepidemiol Drug Saf. 2011 Jun;20(6);583-90

  30. Conclusions • Detection of low frequency or long term adverse outcomes challenge traditional methods of statistical surveillance • (Near) Real time outcome surveillance can result in quickly detecting clinically meaningful efficacy or safety signals • The family of risk adjusted statistical process control methods have utility in continuous quality improvement initiatives such as institutional and provider performance assessment and medical product surveillance.

  31. The End Michael E. Matheny michael.matheny@vanderbilt.eduTVHS Veteran’s AdministrationGRECC, Room 4-B1101310 24th Ave. S.Nashville, TN  37212

  32. Acknowledgements Grant Funding • NIH NLM R-01-LM-0814204 (Resnic) • FDA SOL-08-00837A (Resnic) • VA HSR&D CDA-2 2008-020 (Matheny) • VA HSR&D IIR 292-1 (Matheny) • NIH AHRQ R-01-HS-019913 (Ohno-Machado) • NIH NHLBI U-54-HL-108460 (Ohno-Machado) • Boston • Richard Cope • UshaGovindarajulu • Sharon-Lise Normand • Frederic S. Resnic • Susan Robbins • VenkatesanVidi • Washington, DC • Thomas Gross • NilsaLoyo-Berrios • DanicaMarinac-Dabic • Nashville • Arijit Basu • Fern FitzHenry • Vincent Messina • LalitNookala • Theodore Speroff • San Diego • Lucila Ohno-Machado

More Related