Time series and risk adjusted control charts
1 / 32

Time Series and Risk Adjusted Control Charts - PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Time Series and Risk Adjusted Control Charts. Michael E. Matheny, MD, MS, MPH TVHS Veteran’s Administration Division of General Internal Medicine Department of Biomedical Informatics Department of Biostatistics Vanderbilt University Medical Center Nashville , TN. Objectives.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Time Series and Risk Adjusted Control Charts

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Time Series and Risk AdjustedControl Charts

Michael E. Matheny, MD, MS, MPH

TVHS Veteran’s Administration

Division of General Internal Medicine

Department of Biomedical Informatics

Department of Biostatistics

Vanderbilt University Medical Center

Nashville, TN


  • Industrial Statistical Process Control

  • Important statistical considerations for medical domain SPC use

  • Highlighted methods with examples

    • Risk Adjusted Statistical Process Control

    • Propensity Score Matched Proportional Difference

    • Risk Adjusted Sequential Probability Ratio Testing

    • Maximized Sequential Probability Ratio Testing

  • Conclusions

Statistical Process ControlIndustrial Methods

  • Most well-defined sequential methods were in industrial processes

    • Shewhart charts

    • Exponentially weighted moving average charts

    • Cumulative sum charts

    • Sequential probability ratio testing

Statistical Process ControlIndustrial Characteristics

  • Homogeneity of each observation is assumed

  • The observed data sets the expectations

    • This method looks for changes in proportion over time

  • Process consistency is monitored

  • Evaluating a medical product with a stable but “clinically unacceptable” adverse event rate would not generate an alarm.

Source: Oakland, JS. Statistical Process Control, 5th Ed. Butterworth-Heinemann, 2003

Statistical Process Controlperiodic p chart

Statistical Process ControlMedical Domain Adoption Barriers

  • No comparison to pre-existing standard

    • New medical products are held to the adverse event standards of previous medical products

  • Process level heterogeneity

    • Changing standard of care over time

    • Introduction of new medical products

  • Patient level heterogeneity

    • Demographic factors

    • Chronic illness variation

    • Acute illness variation

    • Care tailored to individual patient

Medical SPCCommon Reference Data Selections

  • Prospective concurrent controls

    • closely related medical product (same class or indication)

    • distantly related medical product (different class/indication)

  • Retrospective external controls

    • closely related medical product

    • distantly related medical product

  • Retrospective phase 3 clinical trial data

    • usually non-representative population

    • small sample size

    • limited outcomes

Statistical Process ControlPeriodic -> Cumulative

  • The most basic p chart is only one time period, but this can be extended by analyzing all of the available data in each time period or sample size increment.

  • Strengths

    • Increasing sample size over time allows better detection of trends

  • Weaknesses

    • Repeated measurements inflate Type I error

    • Extended cumulative analysis can miss newly evolving changes from weighting of old data

Statistical Process ControlRisk Adjustment

  • Risk adjustment is required to address observational heterogeneity

  • Two methods to address these requirements:

    • Apply prediction model to observed data developed from prior baseline or comparison data

    • Exposed-Unexposed (Case-control) Matching

Statistical Process ControlRisk Adjustment

  • When a prediction model is used to perform risk adjusted SPC, the performance of the model drives the overall result.

    • Evaluation of discrimination and calibration, with emphasis on individual risk prediction performance

    • Awareness that individual prediction performance degrades quickly as the source data ages or when applied to a systematically different population

Statistical Process ControlRandomized Controlled Trial Example

  • OPUS (TIMI-16)

    • Setting: 888 Hospitals in 27 Countries

    • Population: Control (3421), Intervention (6867)

    • Intervention: Oral 2b/3a Inhibitor vsPlacebo

    • Outcome: 30 day mortality

    • Trial stopped early


    • Setting: 313 Hospitals in 23 Countries

    • Population: Control (1739), Intervention (1751)

    • Intervention: Oral Anti-Platelet Agent vsPlacebo

    • Outcome: Major Bleeding

    • DSMB concerned, but trial did not stop early

Source: Matheny et al. Medical Decision Making. 2009;29:247-56.

Risk Adjusted SPCOPUS RCT LR-SPC p chart

AUC 0.70 (0.62-0.78)

χ2(8df)=8.38 (p=0.40)

Risk Adjusted SPCCLARITY RCT LR-SPC p chart

AUC 0.80 (0.74-0.87)

χ2(8df)=10.5 (p=0.23)

Risk Adjusted SPCSummary

  • More sensitive than trial analysis

    • Type I Error Inflation

    • Early detection methods are more tolerant of Type I error than Type II error

  • Not interpretable if no events in baseline

Propensity Score MatchingCaliper Matching Method

  • ±(0.2 - 0.6)*SD of the estimated logit

  • ±(0.01 - 0.05) of estimated probability

  • Recent study simulation found that 0.2*SD and 0.02/0.03 were comparable and were optimal parameter selections

  • Time window limitation to limit process level changes between matched cases and controls

Source: Austin, PC. Biometrical J. 2009;51:171-184.

Propensity Score Matched SPCVascular Closure Device Surveillance Example

  • Setting

    • Brigham & Women’s Hospital (01/2002 – 12/2004) patients undergoing percutaneous coronary intervention (3947)

  • Exposure: vascular closure device

  • Outcome: Retroperitoneal Hemorrhage (25)

  • Baseline: Stanford University Data (2000 – 2004)

  • Propensity Score (62 variables) By calendar quarter with PS caliper of ±0.03

Source: Matheny et al. AMIA AnnuSymp Proc. 2007;518-522.

Propensity Score Matched SPCVascular Closure Device Surveillance Example

Matched 92.4% (1,144/1,238) by calendar quarter

AUC 0.70

Root Cause AnalysisConfounding By Indication

Critical risk factor was location of access site, which was not captured in the routine ACC-NCDR (or Mass-DAC) datasets.

Automated surveillance is hypothesis generating – all alerts REQUIRE detailed review and confirmation

Source: Tiroch K et al. American Journal of Cardiology. 2008;102:1473-6.

Propensity Score Matched SPCSummary

  • Risk adjustment method (again) drives overall analysis performance

    • Evaluate relative propensity score distribution

    • Report and evaluate matched and unmatched groups, can limit interpretation of results

  • Limited to concurrent analysis where baseline exposure data is coexisting

    • Requires sustained clinical care with both ‘exposed’ and ‘unexposed’ medical products

  • Type I error inflation remains

Risk Adjusted SPCType I Error Adjustment

Use of an alpha spending function to adjust per-period alerting boundary thresholds



Risk Adjusted SPCTaxusExpress DES versus Other DES Surveillance

Massachusetts PCI 2004-2007

Evaluation ofcumulative post-procedure myocardial infarction rate for new drug eluting stent as compared with propensity matched control DES.

Propensity score matching resulted in 81.5% of 18,277 Taxus Express2 devices matched and analyzed.

Source: Resnic et al. JAMA 2010;304(18):2019-2027.

Another SPC PathwaySequential Probability Ratio Testing

  • Retrospective control data is more common that concurrent controls

    • Dominance of new medical product

    • National or regional external standard

  • Sequential framework that accounts for repeated measurements

Sequential Probability Ratio TestRisk Adjusted Methods Summary

Formal framework for incorporating ά and β error

Specify Odds Ratio of event rate elevation detection desired

Risk Adjustment using a binary outcome risk model to adjust the cumulative log odds

Source: Spiegelhalter, et al. International Journal of Quality Healthcare 2003;15:7-13


Brigham & Women’s Hospital (01/2002 – 10/2006)


All Operators (18) who performed PCI on patients (8750)


Inpatient Mortality (125) (1.4%)


National ACC-NCDR Data

Local BWH Data

Risk Adjusted SPRTInterventional Cardiology Operator Assessment Example

Source: Matheny et al. American Heart Journal. 2008;155:114-20.

Risk Adjusted SPRTOperator Outlier Detection

Risk Adjusted SPRT Operator Root Cause Analysis

Evaluated each fatal case for the operator in question, and noted a high compassionate use rate compared to other operators

After excluding patients that were not considered candidates for CABG or were clearly documented as extremely high-risk compassionate care, operator did not exceed mortality rate expectations

Risk-Adjusted SPRTMA CABG Mortality Institutional Outlier Detection

Massachusetts CABG Mortality 2002-2007

2 known institutional outliers in 5 years of data (4 and 1)

Both methods detected all true positives, SPRT had 1 false positive

Source: Matheny et al. BMC Med. Inform. Dec. Mak. 2011;11:75.

Maximized SPRTMethodology

  • Formal framework for incorporating ά and β error as well as repeated measurements

  • Composite hypothesis for event detection where any odds ratio > 1.0

  • Poisson version uses prediction model from retrospective data for risk adjustment

  • Case-Control version uses propensity score matching for prospective concurrent analysis

Source: Li, Kulldorff.Statist. Med. 2010;29:284-295

Maximized SPRTH1N1 Vaccine Surveillance (Poisson)

Source: Greene et al. Pharmacoepidemiol Drug Saf. 2011 Jun;20(6);583-90


  • Detection of low frequency or long term adverse outcomes challenge traditional methods of statistical surveillance

  • (Near) Real time outcome surveillance can result in quickly detecting clinically meaningful efficacy or safety signals

  • The family of risk adjusted statistical process control methods have utility in continuous quality improvement initiatives such as institutional and provider performance assessment and medical product surveillance.

The End

Michael E. Matheny michael.matheny@vanderbilt.eduTVHS Veteran’s AdministrationGRECC, Room 4-B1101310 24th Ave. S.Nashville, TN  37212


Grant Funding

  • NIH NLM R-01-LM-0814204 (Resnic)

  • FDA SOL-08-00837A (Resnic)

  • VA HSR&D CDA-2 2008-020 (Matheny)

  • VA HSR&D IIR 292-1 (Matheny)

  • NIH AHRQ R-01-HS-019913 (Ohno-Machado)

  • NIH NHLBI U-54-HL-108460 (Ohno-Machado)

  • Boston

    • Richard Cope

    • UshaGovindarajulu

    • Sharon-Lise Normand

    • Frederic S. Resnic

    • Susan Robbins

    • VenkatesanVidi

  • Washington, DC

    • Thomas Gross

    • NilsaLoyo-Berrios

    • DanicaMarinac-Dabic

  • Nashville

    • Arijit Basu

    • Fern FitzHenry

    • Vincent Messina

    • LalitNookala

    • Theodore Speroff

  • San Diego

    • Lucila Ohno-Machado

  • Login