Time series and risk adjusted control charts
This presentation is the property of its rightful owner.
Sponsored Links
1 / 32

Time Series and Risk Adjusted Control Charts PowerPoint PPT Presentation


  • 78 Views
  • Uploaded on
  • Presentation posted in: General

Time Series and Risk Adjusted Control Charts. Michael E. Matheny, MD, MS, MPH TVHS Veteran’s Administration Division of General Internal Medicine Department of Biomedical Informatics Department of Biostatistics Vanderbilt University Medical Center Nashville , TN. Objectives.

Download Presentation

Time Series and Risk Adjusted Control Charts

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Time series and risk adjusted control charts

Time Series and Risk AdjustedControl Charts

Michael E. Matheny, MD, MS, MPH

TVHS Veteran’s Administration

Division of General Internal Medicine

Department of Biomedical Informatics

Department of Biostatistics

Vanderbilt University Medical Center

Nashville, TN


Objectives

Objectives

  • Industrial Statistical Process Control

  • Important statistical considerations for medical domain SPC use

  • Highlighted methods with examples

    • Risk Adjusted Statistical Process Control

    • Propensity Score Matched Proportional Difference

    • Risk Adjusted Sequential Probability Ratio Testing

    • Maximized Sequential Probability Ratio Testing

  • Conclusions


Time series and risk adjusted control charts

Statistical Process ControlIndustrial Methods

  • Most well-defined sequential methods were in industrial processes

    • Shewhart charts

    • Exponentially weighted moving average charts

    • Cumulative sum charts

    • Sequential probability ratio testing


Statistical process control industrial characteristics

Statistical Process ControlIndustrial Characteristics

  • Homogeneity of each observation is assumed

  • The observed data sets the expectations

    • This method looks for changes in proportion over time

  • Process consistency is monitored

  • Evaluating a medical product with a stable but “clinically unacceptable” adverse event rate would not generate an alarm.

Source: Oakland, JS. Statistical Process Control, 5th Ed. Butterworth-Heinemann, 2003


Statistical process control periodic p chart

Statistical Process Controlperiodic p chart


Statistical process control medical domain adoption barriers

Statistical Process ControlMedical Domain Adoption Barriers

  • No comparison to pre-existing standard

    • New medical products are held to the adverse event standards of previous medical products

  • Process level heterogeneity

    • Changing standard of care over time

    • Introduction of new medical products

  • Patient level heterogeneity

    • Demographic factors

    • Chronic illness variation

    • Acute illness variation

    • Care tailored to individual patient


Medical spc common reference data selections

Medical SPCCommon Reference Data Selections

  • Prospective concurrent controls

    • closely related medical product (same class or indication)

    • distantly related medical product (different class/indication)

  • Retrospective external controls

    • closely related medical product

    • distantly related medical product

  • Retrospective phase 3 clinical trial data

    • usually non-representative population

    • small sample size

    • limited outcomes


Statistical process control periodic cumulative

Statistical Process ControlPeriodic -> Cumulative

  • The most basic p chart is only one time period, but this can be extended by analyzing all of the available data in each time period or sample size increment.

  • Strengths

    • Increasing sample size over time allows better detection of trends

  • Weaknesses

    • Repeated measurements inflate Type I error

    • Extended cumulative analysis can miss newly evolving changes from weighting of old data


Statistical process control risk adjustment

Statistical Process ControlRisk Adjustment

  • Risk adjustment is required to address observational heterogeneity

  • Two methods to address these requirements:

    • Apply prediction model to observed data developed from prior baseline or comparison data

    • Exposed-Unexposed (Case-control) Matching


Statistical process control risk adjustment1

Statistical Process ControlRisk Adjustment

  • When a prediction model is used to perform risk adjusted SPC, the performance of the model drives the overall result.

    • Evaluation of discrimination and calibration, with emphasis on individual risk prediction performance

    • Awareness that individual prediction performance degrades quickly as the source data ages or when applied to a systematically different population


Statistical process control randomized controlled trial example

Statistical Process ControlRandomized Controlled Trial Example

  • OPUS (TIMI-16)

    • Setting: 888 Hospitals in 27 Countries

    • Population: Control (3421), Intervention (6867)

    • Intervention: Oral 2b/3a Inhibitor vsPlacebo

    • Outcome: 30 day mortality

    • Trial stopped early

  • CLARITY (TIMI-28)

    • Setting: 313 Hospitals in 23 Countries

    • Population: Control (1739), Intervention (1751)

    • Intervention: Oral Anti-Platelet Agent vsPlacebo

    • Outcome: Major Bleeding

    • DSMB concerned, but trial did not stop early

Source: Matheny et al. Medical Decision Making. 2009;29:247-56.


Risk adjusted spc opus rct lr spc p chart

Risk Adjusted SPCOPUS RCT LR-SPC p chart

AUC 0.70 (0.62-0.78)

χ2(8df)=8.38 (p=0.40)


Risk adjusted spc clarity rct lr spc p chart

Risk Adjusted SPCCLARITY RCT LR-SPC p chart

AUC 0.80 (0.74-0.87)

χ2(8df)=10.5 (p=0.23)


Risk adjusted spc summary

Risk Adjusted SPCSummary

  • More sensitive than trial analysis

    • Type I Error Inflation

    • Early detection methods are more tolerant of Type I error than Type II error

  • Not interpretable if no events in baseline


Propensity score matching caliper matching method

Propensity Score MatchingCaliper Matching Method

  • ±(0.2 - 0.6)*SD of the estimated logit

  • ±(0.01 - 0.05) of estimated probability

  • Recent study simulation found that 0.2*SD and 0.02/0.03 were comparable and were optimal parameter selections

  • Time window limitation to limit process level changes between matched cases and controls

Source: Austin, PC. Biometrical J. 2009;51:171-184.


Propensity score matched spc vascular closure device surveillance example

Propensity Score Matched SPCVascular Closure Device Surveillance Example

  • Setting

    • Brigham & Women’s Hospital (01/2002 – 12/2004) patients undergoing percutaneous coronary intervention (3947)

  • Exposure: vascular closure device

  • Outcome: Retroperitoneal Hemorrhage (25)

  • Baseline: Stanford University Data (2000 – 2004)

  • Propensity Score (62 variables) By calendar quarter with PS caliper of ±0.03

Source: Matheny et al. AMIA AnnuSymp Proc. 2007;518-522.


Propensity score matched spc vascular closure device surveillance example1

Propensity Score Matched SPCVascular Closure Device Surveillance Example

Matched 92.4% (1,144/1,238) by calendar quarter

AUC 0.70


Root cause analysis confounding by indication

Root Cause AnalysisConfounding By Indication

Critical risk factor was location of access site, which was not captured in the routine ACC-NCDR (or Mass-DAC) datasets.

Automated surveillance is hypothesis generating – all alerts REQUIRE detailed review and confirmation

Source: Tiroch K et al. American Journal of Cardiology. 2008;102:1473-6.


Propensity score matched spc summary

Propensity Score Matched SPCSummary

  • Risk adjustment method (again) drives overall analysis performance

    • Evaluate relative propensity score distribution

    • Report and evaluate matched and unmatched groups, can limit interpretation of results

  • Limited to concurrent analysis where baseline exposure data is coexisting

    • Requires sustained clinical care with both ‘exposed’ and ‘unexposed’ medical products

  • Type I error inflation remains


Risk adjusted spc type i error adjustment

Risk Adjusted SPCType I Error Adjustment

Use of an alpha spending function to adjust per-period alerting boundary thresholds

O’Brien-Fleming

Lan-DeMets


Risk adjusted spc taxus express des versus other des surveillance

Risk Adjusted SPCTaxusExpress DES versus Other DES Surveillance

Massachusetts PCI 2004-2007

Evaluation ofcumulative post-procedure myocardial infarction rate for new drug eluting stent as compared with propensity matched control DES.

Propensity score matching resulted in 81.5% of 18,277 Taxus Express2 devices matched and analyzed.

Source: Resnic et al. JAMA 2010;304(18):2019-2027.


Another spc pathway sequential probability ratio testing

Another SPC PathwaySequential Probability Ratio Testing

  • Retrospective control data is more common that concurrent controls

    • Dominance of new medical product

    • National or regional external standard

  • Sequential framework that accounts for repeated measurements


Sequential probability ratio test risk adjusted methods summary

Sequential Probability Ratio TestRisk Adjusted Methods Summary

Formal framework for incorporating ά and β error

Specify Odds Ratio of event rate elevation detection desired

Risk Adjustment using a binary outcome risk model to adjust the cumulative log odds

Source: Spiegelhalter, et al. International Journal of Quality Healthcare 2003;15:7-13


Time series and risk adjusted control charts

Setting:

Brigham & Women’s Hospital (01/2002 – 10/2006)

Exposure:

All Operators (18) who performed PCI on patients (8750)

Outcome:

Inpatient Mortality (125) (1.4%)

Baselines:

National ACC-NCDR Data

Local BWH Data

Risk Adjusted SPRTInterventional Cardiology Operator Assessment Example

Source: Matheny et al. American Heart Journal. 2008;155:114-20.


Risk adjusted sprt operator outlier detection

Risk Adjusted SPRTOperator Outlier Detection


Risk adjusted sprt operator root cause analysis

Risk Adjusted SPRT Operator Root Cause Analysis

Evaluated each fatal case for the operator in question, and noted a high compassionate use rate compared to other operators

After excluding patients that were not considered candidates for CABG or were clearly documented as extremely high-risk compassionate care, operator did not exceed mortality rate expectations


Risk adjusted sprt ma cabg mortality institutional outlier detection

Risk-Adjusted SPRTMA CABG Mortality Institutional Outlier Detection

Massachusetts CABG Mortality 2002-2007

2 known institutional outliers in 5 years of data (4 and 1)

Both methods detected all true positives, SPRT had 1 false positive

Source: Matheny et al. BMC Med. Inform. Dec. Mak. 2011;11:75.


Maximized sprt methodology

Maximized SPRTMethodology

  • Formal framework for incorporating ά and β error as well as repeated measurements

  • Composite hypothesis for event detection where any odds ratio > 1.0

  • Poisson version uses prediction model from retrospective data for risk adjustment

  • Case-Control version uses propensity score matching for prospective concurrent analysis

Source: Li, Kulldorff.Statist. Med. 2010;29:284-295


Maximized sprt h1n1 vaccine surveillance poisson

Maximized SPRTH1N1 Vaccine Surveillance (Poisson)

Source: Greene et al. Pharmacoepidemiol Drug Saf. 2011 Jun;20(6);583-90


Conclusions

Conclusions

  • Detection of low frequency or long term adverse outcomes challenge traditional methods of statistical surveillance

  • (Near) Real time outcome surveillance can result in quickly detecting clinically meaningful efficacy or safety signals

  • The family of risk adjusted statistical process control methods have utility in continuous quality improvement initiatives such as institutional and provider performance assessment and medical product surveillance.


Time series and risk adjusted control charts

The End

Michael E. Matheny [email protected] Veteran’s AdministrationGRECC, Room 4-B1101310 24th Ave. S.Nashville, TN  37212


Acknowledgements

Acknowledgements

Grant Funding

  • NIH NLM R-01-LM-0814204 (Resnic)

  • FDA SOL-08-00837A (Resnic)

  • VA HSR&D CDA-2 2008-020 (Matheny)

  • VA HSR&D IIR 292-1 (Matheny)

  • NIH AHRQ R-01-HS-019913 (Ohno-Machado)

  • NIH NHLBI U-54-HL-108460 (Ohno-Machado)

  • Boston

    • Richard Cope

    • UshaGovindarajulu

    • Sharon-Lise Normand

    • Frederic S. Resnic

    • Susan Robbins

    • VenkatesanVidi

  • Washington, DC

    • Thomas Gross

    • NilsaLoyo-Berrios

    • DanicaMarinac-Dabic

  • Nashville

    • Arijit Basu

    • Fern FitzHenry

    • Vincent Messina

    • LalitNookala

    • Theodore Speroff

  • San Diego

    • Lucila Ohno-Machado


  • Login