time series and risk adjusted control charts n.
Skip this Video
Loading SlideShow in 5 Seconds..
Time Series and Risk Adjusted Control Charts PowerPoint Presentation
Download Presentation
Time Series and Risk Adjusted Control Charts

Loading in 2 Seconds...

play fullscreen
1 / 32

Time Series and Risk Adjusted Control Charts - PowerPoint PPT Presentation

  • Updated on

Time Series and Risk Adjusted Control Charts. Michael E. Matheny, MD, MS, MPH TVHS Veteran’s Administration Division of General Internal Medicine Department of Biomedical Informatics Department of Biostatistics Vanderbilt University Medical Center Nashville , TN. Objectives.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

Time Series and Risk Adjusted Control Charts

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
time series and risk adjusted control charts

Time Series and Risk AdjustedControl Charts

Michael E. Matheny, MD, MS, MPH

TVHS Veteran’s Administration

Division of General Internal Medicine

Department of Biomedical Informatics

Department of Biostatistics

Vanderbilt University Medical Center

Nashville, TN

  • Industrial Statistical Process Control
  • Important statistical considerations for medical domain SPC use
  • Highlighted methods with examples
    • Risk Adjusted Statistical Process Control
    • Propensity Score Matched Proportional Difference
    • Risk Adjusted Sequential Probability Ratio Testing
    • Maximized Sequential Probability Ratio Testing
  • Conclusions

Statistical Process ControlIndustrial Methods

  • Most well-defined sequential methods were in industrial processes
    • Shewhart charts
    • Exponentially weighted moving average charts
    • Cumulative sum charts
    • Sequential probability ratio testing
statistical process control industrial characteristics
Statistical Process ControlIndustrial Characteristics
  • Homogeneity of each observation is assumed
  • The observed data sets the expectations
    • This method looks for changes in proportion over time
  • Process consistency is monitored
  • Evaluating a medical product with a stable but “clinically unacceptable” adverse event rate would not generate an alarm.

Source: Oakland, JS. Statistical Process Control, 5th Ed. Butterworth-Heinemann, 2003

statistical process control medical domain adoption barriers
Statistical Process ControlMedical Domain Adoption Barriers
  • No comparison to pre-existing standard
    • New medical products are held to the adverse event standards of previous medical products
  • Process level heterogeneity
    • Changing standard of care over time
    • Introduction of new medical products
  • Patient level heterogeneity
    • Demographic factors
    • Chronic illness variation
    • Acute illness variation
    • Care tailored to individual patient
medical spc common reference data selections
Medical SPCCommon Reference Data Selections
  • Prospective concurrent controls
    • closely related medical product (same class or indication)
    • distantly related medical product (different class/indication)
  • Retrospective external controls
    • closely related medical product
    • distantly related medical product
  • Retrospective phase 3 clinical trial data
    • usually non-representative population
    • small sample size
    • limited outcomes
statistical process control periodic cumulative
Statistical Process ControlPeriodic -> Cumulative
  • The most basic p chart is only one time period, but this can be extended by analyzing all of the available data in each time period or sample size increment.
  • Strengths
    • Increasing sample size over time allows better detection of trends
  • Weaknesses
    • Repeated measurements inflate Type I error
    • Extended cumulative analysis can miss newly evolving changes from weighting of old data
statistical process control risk adjustment
Statistical Process ControlRisk Adjustment
  • Risk adjustment is required to address observational heterogeneity
  • Two methods to address these requirements:
    • Apply prediction model to observed data developed from prior baseline or comparison data
    • Exposed-Unexposed (Case-control) Matching
statistical process control risk adjustment1
Statistical Process ControlRisk Adjustment
  • When a prediction model is used to perform risk adjusted SPC, the performance of the model drives the overall result.
    • Evaluation of discrimination and calibration, with emphasis on individual risk prediction performance
    • Awareness that individual prediction performance degrades quickly as the source data ages or when applied to a systematically different population
statistical process control randomized controlled trial example
Statistical Process ControlRandomized Controlled Trial Example
  • OPUS (TIMI-16)
    • Setting: 888 Hospitals in 27 Countries
    • Population: Control (3421), Intervention (6867)
    • Intervention: Oral 2b/3a Inhibitor vsPlacebo
    • Outcome: 30 day mortality
    • Trial stopped early
    • Setting: 313 Hospitals in 23 Countries
    • Population: Control (1739), Intervention (1751)
    • Intervention: Oral Anti-Platelet Agent vsPlacebo
    • Outcome: Major Bleeding
    • DSMB concerned, but trial did not stop early

Source: Matheny et al. Medical Decision Making. 2009;29:247-56.

risk adjusted spc opus rct lr spc p chart
Risk Adjusted SPCOPUS RCT LR-SPC p chart

AUC 0.70 (0.62-0.78)

χ2(8df)=8.38 (p=0.40)

risk adjusted spc clarity rct lr spc p chart
Risk Adjusted SPCCLARITY RCT LR-SPC p chart

AUC 0.80 (0.74-0.87)

χ2(8df)=10.5 (p=0.23)

risk adjusted spc summary
Risk Adjusted SPCSummary
  • More sensitive than trial analysis
    • Type I Error Inflation
    • Early detection methods are more tolerant of Type I error than Type II error
  • Not interpretable if no events in baseline
propensity score matching caliper matching method
Propensity Score MatchingCaliper Matching Method
  • ±(0.2 - 0.6)*SD of the estimated logit
  • ±(0.01 - 0.05) of estimated probability
  • Recent study simulation found that 0.2*SD and 0.02/0.03 were comparable and were optimal parameter selections
  • Time window limitation to limit process level changes between matched cases and controls

Source: Austin, PC. Biometrical J. 2009;51:171-184.

propensity score matched spc vascular closure device surveillance example
Propensity Score Matched SPCVascular Closure Device Surveillance Example
  • Setting
    • Brigham & Women’s Hospital (01/2002 – 12/2004) patients undergoing percutaneous coronary intervention (3947)
  • Exposure: vascular closure device
  • Outcome: Retroperitoneal Hemorrhage (25)
  • Baseline: Stanford University Data (2000 – 2004)
  • Propensity Score (62 variables) By calendar quarter with PS caliper of ±0.03

Source: Matheny et al. AMIA AnnuSymp Proc. 2007;518-522.

propensity score matched spc vascular closure device surveillance example1
Propensity Score Matched SPCVascular Closure Device Surveillance Example

Matched 92.4% (1,144/1,238) by calendar quarter

AUC 0.70

root cause analysis confounding by indication
Root Cause AnalysisConfounding By Indication

Critical risk factor was location of access site, which was not captured in the routine ACC-NCDR (or Mass-DAC) datasets.

Automated surveillance is hypothesis generating – all alerts REQUIRE detailed review and confirmation

Source: Tiroch K et al. American Journal of Cardiology. 2008;102:1473-6.

propensity score matched spc summary
Propensity Score Matched SPCSummary
  • Risk adjustment method (again) drives overall analysis performance
    • Evaluate relative propensity score distribution
    • Report and evaluate matched and unmatched groups, can limit interpretation of results
  • Limited to concurrent analysis where baseline exposure data is coexisting
    • Requires sustained clinical care with both ‘exposed’ and ‘unexposed’ medical products
  • Type I error inflation remains
risk adjusted spc type i error adjustment
Risk Adjusted SPCType I Error Adjustment

Use of an alpha spending function to adjust per-period alerting boundary thresholds



risk adjusted spc taxus express des versus other des surveillance
Risk Adjusted SPCTaxusExpress DES versus Other DES Surveillance

Massachusetts PCI 2004-2007

Evaluation ofcumulative post-procedure myocardial infarction rate for new drug eluting stent as compared with propensity matched control DES.

Propensity score matching resulted in 81.5% of 18,277 Taxus Express2 devices matched and analyzed.

Source: Resnic et al. JAMA 2010;304(18):2019-2027.

another spc pathway sequential probability ratio testing
Another SPC PathwaySequential Probability Ratio Testing
  • Retrospective control data is more common that concurrent controls
    • Dominance of new medical product
    • National or regional external standard
  • Sequential framework that accounts for repeated measurements
sequential probability ratio test risk adjusted methods summary
Sequential Probability Ratio TestRisk Adjusted Methods Summary

Formal framework for incorporating ά and β error

Specify Odds Ratio of event rate elevation detection desired

Risk Adjustment using a binary outcome risk model to adjust the cumulative log odds

Source: Spiegelhalter, et al. International Journal of Quality Healthcare 2003;15:7-13


Brigham & Women’s Hospital (01/2002 – 10/2006)


All Operators (18) who performed PCI on patients (8750)


Inpatient Mortality (125) (1.4%)


National ACC-NCDR Data

Local BWH Data

Risk Adjusted SPRTInterventional Cardiology Operator Assessment Example

Source: Matheny et al. American Heart Journal. 2008;155:114-20.

risk adjusted sprt operator root cause analysis
Risk Adjusted SPRT Operator Root Cause Analysis

Evaluated each fatal case for the operator in question, and noted a high compassionate use rate compared to other operators

After excluding patients that were not considered candidates for CABG or were clearly documented as extremely high-risk compassionate care, operator did not exceed mortality rate expectations

risk adjusted sprt ma cabg mortality institutional outlier detection
Risk-Adjusted SPRTMA CABG Mortality Institutional Outlier Detection

Massachusetts CABG Mortality 2002-2007

2 known institutional outliers in 5 years of data (4 and 1)

Both methods detected all true positives, SPRT had 1 false positive

Source: Matheny et al. BMC Med. Inform. Dec. Mak. 2011;11:75.

maximized sprt methodology
Maximized SPRTMethodology
  • Formal framework for incorporating ά and β error as well as repeated measurements
  • Composite hypothesis for event detection where any odds ratio > 1.0
  • Poisson version uses prediction model from retrospective data for risk adjustment
  • Case-Control version uses propensity score matching for prospective concurrent analysis

Source: Li, Kulldorff.Statist. Med. 2010;29:284-295

maximized sprt h1n1 vaccine surveillance poisson
Maximized SPRTH1N1 Vaccine Surveillance (Poisson)

Source: Greene et al. Pharmacoepidemiol Drug Saf. 2011 Jun;20(6);583-90

  • Detection of low frequency or long term adverse outcomes challenge traditional methods of statistical surveillance
  • (Near) Real time outcome surveillance can result in quickly detecting clinically meaningful efficacy or safety signals
  • The family of risk adjusted statistical process control methods have utility in continuous quality improvement initiatives such as institutional and provider performance assessment and medical product surveillance.

The End

Michael E. Matheny michael.matheny@vanderbilt.eduTVHS Veteran’s AdministrationGRECC, Room 4-B1101310 24th Ave. S.Nashville, TN  37212


Grant Funding

  • NIH NLM R-01-LM-0814204 (Resnic)
  • FDA SOL-08-00837A (Resnic)
  • VA HSR&D CDA-2 2008-020 (Matheny)
  • VA HSR&D IIR 292-1 (Matheny)
  • NIH AHRQ R-01-HS-019913 (Ohno-Machado)
  • NIH NHLBI U-54-HL-108460 (Ohno-Machado)
  • Boston
    • Richard Cope
    • UshaGovindarajulu
    • Sharon-Lise Normand
    • Frederic S. Resnic
    • Susan Robbins
    • VenkatesanVidi
  • Washington, DC
    • Thomas Gross
    • NilsaLoyo-Berrios
    • DanicaMarinac-Dabic
  • Nashville
    • Arijit Basu
    • Fern FitzHenry
    • Vincent Messina
    • LalitNookala
    • Theodore Speroff
  • San Diego
    • Lucila Ohno-Machado