Evaluation Methods in Healthcare Quality
Download
1 / 20

Dennis Ross-Degnan, ScD Harvard Medical School and Harvard Pilgrim Health Care Institute - PowerPoint PPT Presentation


  • 129 Views
  • Uploaded on

Evaluation Methods in Healthcare Quality Improvement: Time Series Methods for Evaluating Quality Improvement Initiatives. Dennis Ross-Degnan, ScD Harvard Medical School and Harvard Pilgrim Health Care Institute 2014 Academy for Healthcare Improvement Conference Baltimore, MD, May 30, 2014.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Dennis Ross-Degnan, ScD Harvard Medical School and Harvard Pilgrim Health Care Institute' - nicodemus-illias


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Evaluation Methods in Healthcare Quality Improvement: Time Series Methods for Evaluating Quality Improvement Initiatives

Dennis Ross-Degnan, ScD

Harvard Medical School and

Harvard Pilgrim Health Care Institute

2014 Academy for Healthcare Improvement Conference

Baltimore, MD, May 30, 2014


Overview
Overview

  • Rationale

    • Study designs and inference

    • Interrupted time series (ITS) design

  • Segmented regression analysis of ITS

    • Data setup and models

    • Estimating effects

  • Discussion of examples

  • Strengthening ITS studies


Rcts gold standard in study design but rare in natural or quasi experiments
RCTs: Gold Standard in Study Design but Rare in Natural or Quasi-Experiments

Randomized Controlled Trial

Time

Intervention Group

O1XO2

R

O1

O2

Control Group

X=policy intervention Ot=Measurement at time t


Rapp analyzing a group randomized quality improvement intervention
RaPP Quasi-Experiments: Analyzing a Group-Randomized Quality Improvement Intervention

  • Tailored intervention to improve use of antihypertensive and cholesterol medicines for primary prevention of CVD in Norway

    • Educational outreach by pharmacist with audit+ feedback, EMR reminders (70 practices; 257 MDs)

    • Controls receive passive dissemination of evidence-based guidelines (69 practices; 244 MDs)

  • Outcomes measured monthly for eligible patients 1 year before & after intervention

    Fretheim A, Oxman AD, Håvelsrud K et al. Rational Prescribing in Primary Care (RaPP): A cluster randomized trial of a tailored intervention. PLoS Medicine 2006.

    Fretheim A, Soumerai SB, Zhang F, et al. Interrupted time series analysis yielded an effect estimate concordant with the cluster-randomized controlled trial result. Journal of Clinical Epidemiology 2013


Traditional difference in difference analysis of rapp rct
Traditional Difference in Difference Analysis of RAPP RCT Quasi-Experiments

RCT DiD: +9.0% (4.9%, 13.1 %)


Quasi experimental design non random comparison group

Time Quasi-Experiments

Intervention Group

O1XO2

Non-random Comparison Group Design

R

O1

O2

Comparison Group

“Quasi-Experimental” Design: Non-random Comparison Group

X=policy intervention Ot=Measurement at time t


Strongest quasi experimental design interrupted time series its
Strongest Quasi-Experimental Design: Quasi-ExperimentsInterrupted Time Series (ITS)

Time

Experimental Group

O1 O2 O3X O4 O5O6

Stronger if includes

Comparison Group

O1 O2 O3O4 O5O6

X=policy intervention Ot=Measurement at time t

Time series: multiple measures of a single characteristic at equidistant time intervals


Its analysis of rapp study intervention group only
ITS Analysis of Quasi-ExperimentsRaPP Study: Intervention Group Only


Its logic and parameters estimated by segmented linear regression
ITS Logic and Parameters Estimated by Segmented Linear Regression

Outcome

Intervention

Overall change at given time

Immediate level change

Baseline trend

Change from baseline trend

Baseline level

before intervention after intervention

Time

Assumption: Baseline trend correctly reflects what would have happened without intervention

Adapted from Schneeweiss et al, Health Policy 2001


Its analysis of rapp study intervention group only1
ITS Analysis of RegressionRaPP Study: Intervention Group Only

Assumptions:

Linearity

Normality

Autocorrelation structure

Yt = ß0 + ß1*time + ß2*intervention + ß3*time after intervention

RCT DiD: +9.0% (4.9%, 13.1 %)

ITS: +11.5% (9.5%, 13.5%)

β2

β0

β3

β1


Does adding rapp control group change interpretation of intervention effects
Does Adding RegressionRaPP Control Group Change Interpretation of Intervention Effects?

RCT DiD: +9.0% (4.9%, 13.1 %)

ITS: +11.5% (9.5%, 13.5%)

ITS+comparison: +14.0% (8.6%, 19.4%)


Effects of reimbursement caps followed by copayments in nh medicaid

Multiple-Drug Recipients (n=860) Regression

All Other Patients (n=8002)

Effects of Reimbursement Caps Followed by Copayments in NH Medicaid

3 Rx per month cap begins

Cap replaced by $1 Copay

Adapted from Soumerai et al, N Engl J Med 1987


Effects of reimbursement caps followed by copayments in nh medicaid1

Multiple-Drug Recipients (n=860) Regression

All Other Patients (n=8002)

Effects of Reimbursement Caps Followed by Copayments in NH Medicaid

3 Rx per month cap begins

Cap replaced by $1 Copay

Back

Adapted from Soumerai et al, N Engl J Med 1987


Threats to validity of its design
Threats to Validity of ITS Design Regression

  • Confounding: co-occurring intervention

  • Selection: pre-intervention factors affect inclusion (e.g., volunteers)

  • Statistical regression: group(s) selected because of baseline use

  • Instrumentation: change in measurement (ascertainment)

  • History or maturation: external event or natural process explains effect


Threats to reliability of its estimates
Threats to Reliability of ITS Estimates Regression

  • Data quality

    • Short segments (few time points)

    • Unstable data (high variability)

    • Missing data or wild data points

  • Nature of population or process

    • Changing denominators

    • Low frequency (e.g., deaths)

    • Near boundary (e.g., 0% or 100%)

    • Non-linear trends


Summary
Summary Regression

  • Advantages of ITS

    • Intuitive visual display

    • Direct estimate of effects

    • Controls common threats to validity

  • Limitations of ITS

    • Requires reasonably stable data

    • Boundary problems

    • Ideally 10+ data points per segment

    • Sensitive to points near end of segment


Strengthening its studies
Strengthening ITS Studies Regression

  • Check data quality: Outliers, missing data, implausible data

  • Contrast multiple outcomes or groups: High-risk subgroups, different intensity

  • Account for intervention phase-in: Anticipatory effects, implementation time

  • Match on baseline values: Standardize or propensity match comparison groups

  • Test model assumptions: Normality of errors, linearity of segments


Discussion
Discussion Regression

  • Questions

  • Additional topics

    • Data quality checking

    • Sequential interventions

    • Comparison groups

    • Interpreting effects

    • Selecting and matching study groups


Selected references
Selected References Regression

  • Lagarde M. How to do (or not to do) …Assessing the impact of a policy change with routine longitudinal data. Health Policy and Planning 2012;27:76-83.

  • Linden A, Adams JL. Applying a propensity score-based weighting model to interrupted time series data: improving causal inference in programme evaluation. J Evaluation in Clinical Practice 2010;1-8.

  • Schneeweiss S, Maclure M, Walker AM, Grootendorst P, Soumerai SB. On the evaluation of drug benefits policy changes with longitudinal claims data: the policy maker’s versus the clinician’s perspective. Health Policy 2001;55:97-109.

  • Serumag`a B, Ross-Degnan D, Avery A, Elliott RA, Majumdar SR, Zhang F, Soumerai SB. Effect of pay for performance on the management and outcomes of hypertension in the United KingdomL interrupted time series study. BMJ 2011;

  • Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. Belmont, CA: Wadsworth, 2002.

  • Soumerai SB, Avorn J, Ross-Degnan D, Gortmaker S. Payment restrictions for prescription drugs under Medicaid. Effects on therapy, cost, and equity. N Engl J Med 1987;317:550-556.

  • Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D. Segmented regression analysis of interrupted time series studies in medication use research. Journal of Clinical Pharmacy and Therapeutics 2002;27:299-309.

  • Wagner AK, Zhang F, Soumerai SB, Walker AM, Gurwitz JH, Glynn RJ, Ross-Degnan D. Benzodiazepine use and hip fractures in the elderly: Who is at greatest risk? Archives of Internal Medicine 2004;164:1567-1572.

  • Zhang F, Wagner AK, Soumerai SB, Ross-Degnan D. Methods for estimating confidence intervals in interrupted time series analyses of health interventions. J ClinEpidemiol 2009;62:143-148.

  • Zhang F, Wagner AK, Ross-Degnan D. Simulation-based power calculation for designing interrupted time series analyses of health policy interventions. J ClinEpidemiol. 2011; 64: 1252-61.


Good internet resources
Good Internet Resources Regression

  • STATA, SAS, R, SPSS

    • http://www.ats.ucla.edu/stat/

  • SAS procautoreg and Stataarima

    • http://www.stata.com/statalist/archive/2009-02/msg00140.html

  • Correcting for Autocorrelation using Stata

    • http://www.polsci.wvu.edu/duval/ps602/Notes/STATA/cocran-orcutt.htm

  • Google!


ad