Evaluation Methods in Healthcare Quality
This presentation is the property of its rightful owner.
Sponsored Links
1 / 20

Dennis Ross-Degnan, ScD Harvard Medical School and Harvard Pilgrim Health Care Institute PowerPoint PPT Presentation


  • 66 Views
  • Uploaded on
  • Presentation posted in: General

Evaluation Methods in Healthcare Quality Improvement: Time Series Methods for Evaluating Quality Improvement Initiatives. Dennis Ross-Degnan, ScD Harvard Medical School and Harvard Pilgrim Health Care Institute 2014 Academy for Healthcare Improvement Conference Baltimore, MD, May 30, 2014.

Download Presentation

Dennis Ross-Degnan, ScD Harvard Medical School and Harvard Pilgrim Health Care Institute

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Dennis ross degnan scd harvard medical school and harvard pilgrim health care institute

Evaluation Methods in Healthcare Quality Improvement: Time Series Methods for Evaluating Quality Improvement Initiatives

Dennis Ross-Degnan, ScD

Harvard Medical School and

Harvard Pilgrim Health Care Institute

2014 Academy for Healthcare Improvement Conference

Baltimore, MD, May 30, 2014


Overview

Overview

  • Rationale

    • Study designs and inference

    • Interrupted time series (ITS) design

  • Segmented regression analysis of ITS

    • Data setup and models

    • Estimating effects

  • Discussion of examples

  • Strengthening ITS studies


Rcts gold standard in study design but rare in natural or quasi experiments

RCTs: Gold Standard in Study Design but Rare in Natural or Quasi-Experiments

Randomized Controlled Trial

Time

Intervention Group

O1XO2

R

O1

O2

Control Group

X=policy intervention Ot=Measurement at time t


Rapp analyzing a group randomized quality improvement intervention

RaPP: Analyzing a Group-Randomized Quality Improvement Intervention

  • Tailored intervention to improve use of antihypertensive and cholesterol medicines for primary prevention of CVD in Norway

    • Educational outreach by pharmacist with audit+ feedback, EMR reminders (70 practices; 257 MDs)

    • Controls receive passive dissemination of evidence-based guidelines (69 practices; 244 MDs)

  • Outcomes measured monthly for eligible patients 1 year before & after intervention

    Fretheim A, Oxman AD, Håvelsrud K et al. Rational Prescribing in Primary Care (RaPP): A cluster randomized trial of a tailored intervention. PLoS Medicine 2006.

    Fretheim A, Soumerai SB, Zhang F, et al. Interrupted time series analysis yielded an effect estimate concordant with the cluster-randomized controlled trial result. Journal of Clinical Epidemiology 2013


Traditional difference in difference analysis of rapp rct

Traditional Difference in Difference Analysis of RAPP RCT

RCT DiD: +9.0% (4.9%, 13.1 %)


Quasi experimental design non random comparison group

Time

Intervention Group

O1XO2

Non-random Comparison Group Design

R

O1

O2

Comparison Group

“Quasi-Experimental” Design: Non-random Comparison Group

X=policy intervention Ot=Measurement at time t


Strongest quasi experimental design interrupted time series its

Strongest Quasi-Experimental Design:Interrupted Time Series (ITS)

Time

Experimental Group

O1 O2 O3X O4 O5O6

Stronger if includes

Comparison Group

O1 O2 O3O4 O5O6

X=policy intervention Ot=Measurement at time t

Time series: multiple measures of a single characteristic at equidistant time intervals


Its analysis of rapp study intervention group only

ITS Analysis of RaPP Study: Intervention Group Only


Its logic and parameters estimated by segmented linear regression

ITS Logic and Parameters Estimated by Segmented Linear Regression

Outcome

Intervention

Overall change at given time

Immediate level change

Baseline trend

Change from baseline trend

Baseline level

before intervention after intervention

Time

Assumption: Baseline trend correctly reflects what would have happened without intervention

Adapted from Schneeweiss et al, Health Policy 2001


Its analysis of rapp study intervention group only1

ITS Analysis of RaPP Study: Intervention Group Only

Assumptions:

Linearity

Normality

Autocorrelation structure

Yt = ß0 + ß1*time + ß2*intervention + ß3*time after intervention

RCT DiD: +9.0% (4.9%, 13.1 %)

ITS: +11.5% (9.5%, 13.5%)

β2

β0

β3

β1


Does adding rapp control group change interpretation of intervention effects

Does Adding RaPP Control Group Change Interpretation of Intervention Effects?

RCT DiD: +9.0% (4.9%, 13.1 %)

ITS: +11.5% (9.5%, 13.5%)

ITS+comparison: +14.0% (8.6%, 19.4%)


Effects of reimbursement caps followed by copayments in nh medicaid

Multiple-Drug Recipients (n=860)

All Other Patients (n=8002)

Effects of Reimbursement Caps Followed by Copayments in NH Medicaid

3 Rx per month cap begins

Cap replaced by $1 Copay

Adapted from Soumerai et al, N Engl J Med 1987


Effects of reimbursement caps followed by copayments in nh medicaid1

Multiple-Drug Recipients (n=860)

All Other Patients (n=8002)

Effects of Reimbursement Caps Followed by Copayments in NH Medicaid

3 Rx per month cap begins

Cap replaced by $1 Copay

Back

Adapted from Soumerai et al, N Engl J Med 1987


Threats to validity of its design

Threats to Validity of ITS Design

  • Confounding: co-occurring intervention

  • Selection: pre-intervention factors affect inclusion (e.g., volunteers)

  • Statistical regression: group(s) selected because of baseline use

  • Instrumentation: change in measurement (ascertainment)

  • History or maturation: external event or natural process explains effect


Threats to reliability of its estimates

Threats to Reliability of ITS Estimates

  • Data quality

    • Short segments (few time points)

    • Unstable data (high variability)

    • Missing data or wild data points

  • Nature of population or process

    • Changing denominators

    • Low frequency (e.g., deaths)

    • Near boundary (e.g., 0% or 100%)

    • Non-linear trends


Summary

Summary

  • Advantages of ITS

    • Intuitive visual display

    • Direct estimate of effects

    • Controls common threats to validity

  • Limitations of ITS

    • Requires reasonably stable data

    • Boundary problems

    • Ideally 10+ data points per segment

    • Sensitive to points near end of segment


Strengthening its studies

Strengthening ITS Studies

  • Check data quality: Outliers, missing data, implausible data

  • Contrast multiple outcomes or groups: High-risk subgroups, different intensity

  • Account for intervention phase-in: Anticipatory effects, implementation time

  • Match on baseline values: Standardize or propensity match comparison groups

  • Test model assumptions: Normality of errors, linearity of segments


Discussion

Discussion

  • Questions

  • Additional topics

    • Data quality checking

    • Sequential interventions

    • Comparison groups

    • Interpreting effects

    • Selecting and matching study groups


Selected references

Selected References

  • Lagarde M. How to do (or not to do) …Assessing the impact of a policy change with routine longitudinal data. Health Policy and Planning 2012;27:76-83.

  • Linden A, Adams JL. Applying a propensity score-based weighting model to interrupted time series data: improving causal inference in programme evaluation. J Evaluation in Clinical Practice 2010;1-8.

  • Schneeweiss S, Maclure M, Walker AM, Grootendorst P, Soumerai SB. On the evaluation of drug benefits policy changes with longitudinal claims data: the policy maker’s versus the clinician’s perspective. Health Policy 2001;55:97-109.

  • Serumag`a B, Ross-Degnan D, Avery A, Elliott RA, Majumdar SR, Zhang F, Soumerai SB. Effect of pay for performance on the management and outcomes of hypertension in the United KingdomL interrupted time series study. BMJ 2011;

  • Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. Belmont, CA: Wadsworth, 2002.

  • Soumerai SB, Avorn J, Ross-Degnan D, Gortmaker S. Payment restrictions for prescription drugs under Medicaid. Effects on therapy, cost, and equity. N Engl J Med 1987;317:550-556.

  • Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D. Segmented regression analysis of interrupted time series studies in medication use research. Journal of Clinical Pharmacy and Therapeutics 2002;27:299-309.

  • Wagner AK, Zhang F, Soumerai SB, Walker AM, Gurwitz JH, Glynn RJ, Ross-Degnan D. Benzodiazepine use and hip fractures in the elderly: Who is at greatest risk? Archives of Internal Medicine 2004;164:1567-1572.

  • Zhang F, Wagner AK, Soumerai SB, Ross-Degnan D. Methods for estimating confidence intervals in interrupted time series analyses of health interventions. J ClinEpidemiol 2009;62:143-148.

  • Zhang F, Wagner AK, Ross-Degnan D. Simulation-based power calculation for designing interrupted time series analyses of health policy interventions. J ClinEpidemiol. 2011; 64: 1252-61.


Good internet resources

Good Internet Resources

  • STATA, SAS, R, SPSS

    • http://www.ats.ucla.edu/stat/

  • SAS procautoreg and Stataarima

    • http://www.stata.com/statalist/archive/2009-02/msg00140.html

  • Correcting for Autocorrelation using Stata

    • http://www.polsci.wvu.edu/duval/ps602/Notes/STATA/cocran-orcutt.htm

  • Google!


  • Login