Dennis Ross-Degnan, ScD Harvard Medical School and Harvard Pilgrim Health Care Institute

1. Evaluation Methods in Healthcare Quality Improvement: Time Series Methods for Evaluating Quality Improvement Initiatives Dennis Ross-Degnan, ScD Harvard Medical School and Harvard Pilgrim Health Care Institute 2014 Academy for Healthcare Improvement Conference Baltimore, MD, May 30, 2014

2. Overview • Rationale • Study designs and inference • Interrupted time series (ITS) design • Segmented regression analysis of ITS • Data setup and models • Estimating effects • Discussion of examples • Strengthening ITS studies

3. RCTs: Gold Standard in Study Design but Rare in Natural or Quasi-Experiments Randomized Controlled Trial Time Intervention Group O1XO2 R O1 O2 Control Group X=policy intervention Ot=Measurement at time t

4. RaPP: Analyzing a Group-Randomized Quality Improvement Intervention • Tailored intervention to improve use of antihypertensive and cholesterol medicines for primary prevention of CVD in Norway • Educational outreach by pharmacist with audit+ feedback, EMR reminders (70 practices; 257 MDs) • Controls receive passive dissemination of evidence-based guidelines (69 practices; 244 MDs) • Outcomes measured monthly for eligible patients 1 year before & after intervention Fretheim A, Oxman AD, Håvelsrud K et al. Rational Prescribing in Primary Care (RaPP): A cluster randomized trial of a tailored intervention. PLoS Medicine 2006. Fretheim A, Soumerai SB, Zhang F, et al. Interrupted time series analysis yielded an effect estimate concordant with the cluster-randomized controlled trial result. Journal of Clinical Epidemiology 2013

5. Traditional Difference in Difference Analysis of RAPP RCT RCT DiD: +9.0% (4.9%, 13.1 %)

6. Time Intervention Group O1XO2 Non-random Comparison Group Design R O1 O2 Comparison Group “Quasi-Experimental” Design: Non-random Comparison Group X=policy intervention Ot=Measurement at time t

7. Strongest Quasi-Experimental Design:Interrupted Time Series (ITS) Time Experimental Group O1 O2 O3X O4 O5O6 Stronger if includes Comparison Group O1 O2 O3O4 O5O6 X=policy intervention Ot=Measurement at time t Time series: multiple measures of a single characteristic at equidistant time intervals

8. ITS Analysis of RaPP Study: Intervention Group Only

9. ITS Logic and Parameters Estimated by Segmented Linear Regression Outcome Intervention Overall change at given time Immediate level change Baseline trend Change from baseline trend Baseline level before intervention after intervention Time Assumption: Baseline trend correctly reflects what would have happened without intervention Adapted from Schneeweiss et al, Health Policy 2001

10. ITS Analysis of RaPP Study: Intervention Group Only Assumptions: Linearity Normality Autocorrelation structure Yt = ß0 + ß1*time + ß2*intervention + ß3*time after intervention RCT DiD: +9.0% (4.9%, 13.1 %) ITS: +11.5% (9.5%, 13.5%) β2 β0 β3 β1

11. Does Adding RaPP Control Group Change Interpretation of Intervention Effects? RCT DiD: +9.0% (4.9%, 13.1 %) ITS: +11.5% (9.5%, 13.5%) ITS+comparison: +14.0% (8.6%, 19.4%)

12. Multiple-Drug Recipients (n=860) All Other Patients (n=8002) Effects of Reimbursement Caps Followed by Copayments in NH Medicaid 3 Rx per month cap begins Cap replaced by $1 Copay Adapted from Soumerai et al, N Engl J Med 1987

13. Multiple-Drug Recipients (n=860) All Other Patients (n=8002) Effects of Reimbursement Caps Followed by Copayments in NH Medicaid 3 Rx per month cap begins Cap replaced by $1 Copay Back Adapted from Soumerai et al, N Engl J Med 1987

14. Threats to Validity of ITS Design • Confounding: co-occurring intervention • Selection: pre-intervention factors affect inclusion (e.g., volunteers) • Statistical regression: group(s) selected because of baseline use • Instrumentation: change in measurement (ascertainment) • History or maturation: external event or natural process explains effect

15. Threats to Reliability of ITS Estimates • Data quality • Short segments (few time points) • Unstable data (high variability) • Missing data or wild data points • Nature of population or process • Changing denominators • Low frequency (e.g., deaths) • Near boundary (e.g., 0% or 100%) • Non-linear trends

16. Summary • Advantages of ITS • Intuitive visual display • Direct estimate of effects • Controls common threats to validity • Limitations of ITS • Requires reasonably stable data • Boundary problems • Ideally 10+ data points per segment • Sensitive to points near end of segment

17. Strengthening ITS Studies • Check data quality: Outliers, missing data, implausible data • Contrast multiple outcomes or groups: High-risk subgroups, different intensity • Account for intervention phase-in: Anticipatory effects, implementation time • Match on baseline values: Standardize or propensity match comparison groups • Test model assumptions: Normality of errors, linearity of segments

18. Discussion • Questions • Additional topics • Data quality checking • Sequential interventions • Comparison groups • Interpreting effects • Selecting and matching study groups

19. Selected References • Lagarde M. How to do (or not to do) …Assessing the impact of a policy change with routine longitudinal data. Health Policy and Planning 2012;27:76-83. • Linden A, Adams JL. Applying a propensity score-based weighting model to interrupted time series data: improving causal inference in programme evaluation. J Evaluation in Clinical Practice 2010;1-8. • Schneeweiss S, Maclure M, Walker AM, Grootendorst P, Soumerai SB. On the evaluation of drug benefits policy changes with longitudinal claims data: the policy maker’s versus the clinician’s perspective. Health Policy 2001;55:97-109. • Serumag`a B, Ross-Degnan D, Avery A, Elliott RA, Majumdar SR, Zhang F, Soumerai SB. Effect of pay for performance on the management and outcomes of hypertension in the United KingdomL interrupted time series study. BMJ 2011; • Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. Belmont, CA: Wadsworth, 2002. • Soumerai SB, Avorn J, Ross-Degnan D, Gortmaker S. Payment restrictions for prescription drugs under Medicaid. Effects on therapy, cost, and equity. N Engl J Med 1987;317:550-556. • Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D. Segmented regression analysis of interrupted time series studies in medication use research. Journal of Clinical Pharmacy and Therapeutics 2002;27:299-309. • Wagner AK, Zhang F, Soumerai SB, Walker AM, Gurwitz JH, Glynn RJ, Ross-Degnan D. Benzodiazepine use and hip fractures in the elderly: Who is at greatest risk? Archives of Internal Medicine 2004;164:1567-1572. • Zhang F, Wagner AK, Soumerai SB, Ross-Degnan D. Methods for estimating confidence intervals in interrupted time series analyses of health interventions. J ClinEpidemiol 2009;62:143-148. • Zhang F, Wagner AK, Ross-Degnan D. Simulation-based power calculation for designing interrupted time series analyses of health policy interventions. J ClinEpidemiol. 2011; 64: 1252-61.

20. Good Internet Resources • STATA, SAS, R, SPSS • http://www.ats.ucla.edu/stat/ • SAS procautoreg and Stataarima • http://www.stata.com/statalist/archive/2009-02/msg00140.html • Correcting for Autocorrelation using Stata • http://www.polsci.wvu.edu/duval/ps602/Notes/STATA/cocran-orcutt.htm • Google!