Econometrics with Observational Data: Research Design

Econometrics with Observational Data: Research Design Todd Wagner

Research Design • Goal: evaluate behaviors and identify causation • Policy X caused effect Y • Medication A resulted in B hospitalizations • Unit of analysis can be individual or organizational

Research Methods Random assignment? Yes Intent to Treat?

Research Methods Random assignment? Yes Intent to Treat? No Yes On Treatment Basic RCT Analysis

On Treatment • RCT comparing drug A to drug B • Adherence for drugs • A is 70% • B is 40% • What does a comparison of A versus B tell us?

Research Methods Random assignment? No Yes Intent to Treat? Is there a control group?

Research Methods Is there random assignment Randomized Trial Is there a control group Quasi-experimental Design Descriptive Study

Quasi-Experimental Designs • Difference-in-differences • Regression discontinuity • Switching replications • Non-equivalent dependent variables Most common In health

Difference-in-Differences • AKA: DD, D in D, or Diff in Diff • Differences across time and arms • Usually two arms: treatments, controls • In theory can be used with 3+ arms

Methods for Identifying Controls • Inherent matching: Find similar individuals not getting treatment to serve as controls (e.g., twins) • Statistical: use statistical techniques to identify best comparison groups • Location: use other geographic sites, states or regions as controls

Unit of Analysis • D in D works for different units of analysis • Person–people followed over time • Site– sites followed over time • State– states followed over time • May need to make some analytical changes depending on unit of analysis

Diff in Diff example • Gruber, Adams and Newhouse (1997) • Tennessee increased Medicaid fees for primary care services (goal encourage office care; decrease hospital-based ambulatory care) • What is the effect of this policy change?

Research Designs • Difference-in-differences • Regression discontinuity • Switching replications • Nonequivalent dependent variables

Regression Discontinuity • Participants are assigned to program or comparison groups solely on the basis of an observed measure (education test or means test) • Appropriate when we wish to target a program or treatment to those who most need or deserve it

Regression Discontinuity • Partial coverage (not everyone gets the treatment) • Requires the selection mechanism to be fully known • Selection mechanism must be consistently applied to all persons

RD Design Graphically Test for significance Source: Urban Institute Threshold MUST be known and consistently applied

Switching Replications • Has two groups and three waves of measurement • AKA waitlist control group • This design is sometimes used in randomized trials

Example from Pap Smear Study 100% treat 90% 80% 70% Immediate treatment 60% 50% Cumulative % Followed Up 40% 30% 20% delayed treatment 10% 0% 1 2 3 4 5 6 7 8 9 10 11 12 > 12 Months since Initial Pap Intervention Control

Non-Equivalent DVs • Analyze dependent variable that should not be affected by the intervention • Example: Intervention is designed to affect quality of diabetes care, but could also see if intervention affected quality of asthma care

Notes on the Analysisof DD data

Analytical Methods • Plot or graph unadjusted data • Graduate to more complex models • Address, if possible, model limitations

DD Raw Data Baseline 1-Year Follow-Up Exp. Control Exp Control DD+ ----------------------------------------------------------------------------------------- Utilization Entry (% yes) 84.5% 86.1% 88.9% 86.8% 3.7 (36.2) (34.6) (31.4) (33.9) No. of visits (0-16) 3.69 3.84 3.73 3.67 0.21 (4.28) (4.36) (4.00) (4.07) ------------------------------------------------------------------------------------------ Standard deviations in parentheses +DD = (Expfollowup- Expbaseline)-(Controlfollowup- Controlbaseline) † unadjusted estimates

Diff n Diff Model Y = a + b1G + b2T + b3GT+ gX + e Y=outcome G = group (0=control, 1=treatment) T= time (0=baseline, 1=follow-up) X = characteristics of person, place, etc. e = error term

Program Effect • If b3 = 0 then the program has no effect • Limited statistical power. Testing interactions increases risk of type 2 error. Outcome = a + b1G + b2T + b3GT + gX + e

Organizing the Dataset +------------------------------+ avgcost sta3n exp yr_d year -------------------------------- . 358 0 0 93 . 358 0 1 94 318.2305 402 1 0 93 323.2815 402 1 1 94 472.0291 405 1 0 93 480.1368 405 1 1 94 364.0456 436 0 0 93 398.9824 436 0 1 94 369.9669 437 0 0 93 346.4565 437 0 1 94 270.0007 438 0 0 93 322.2588 438 0 1 94 292.7632 442 1 0 93 . 442 1 1 94 475.6746 452 1 0 93 494.9601 452 1 1 94 Note: Data Listed in Stata

Identification • How do you obtain an unbiased estimate of b3? • For an unbiased estimate of GT, G must not be correlated with e; that is, G must be exogenous Outcome = a + b1G + b2T + b3GT + gX + e

Identification • G may be endogenous • Selection bias • Selection bias is type of endogeneity • Caused by non-random assignment • Outcome and G (group) affect each other -- causality runs both ways • Impact: b3 is biased Outcome = a + b1G + b2T + b3GT + gX + e

Example: VA Residential Treatment Wagner, T. H., & Chen, S. (2005). An economic evaluation of inpatient residential treatment programs in the department of veterans affairs. Med Care Res Rev, 62(2), 187-204.

Residential Treatment Programs • RTPs provide mental health and substance use treatment • RTPs were designed to • treat eligible veterans in a less-intensive and more self-reliant setting. • to provide cost-effective care that “promotes independence and fosters responsibility.”

Objectives • Did the RTPs save money? • Were savings a “one-time” event or do they continue to accrue?

Design Choice • Selection mechanism is not observed– can’t use regression discontinuity • We know who adopted RTP and when– DD is feasible

Methods • Built a longitudinal dataset for 1993-1999 for all VA medical centers • Tracked approved RTP programs (N=43) • We merged data from the PTF and CDR to track • Total MH inpatient days (PTF) and dollars (CDR) • Total SA inpatient days (PTF) and dollars (CDR)

Outcomes • Department-level costs • Average cost per MH day • Average cost per SA day • Total MH/SA department costs • Sensitivity analysis • Outpatient MH/SA costs • FTE

Multivariate models • Fixed-effects models1 • DV: Department-level costs • Controlled for medical center size • Inflation adjusted to 1999 using CPI • Year dummies • Wage index 1 Random effects were similar; Hausman tests were not significant. Fixed effects were more conservative.

Results: Mental Health • Average cost savings of $81 per day (p<0.01). • Savings do not appear to be increasing over time.

Mental Health Costs

Results: Substance Abuse • Average cost savings of $112 per day (p<0.01). • Savings do not appear to be increasing over time.

Mental Health Costs

Sensitivity Analysis • RTPs were associated with a slight decrease in the costs of outpatient psychiatry. • RTPs were associated with a decrease in FTE

Limitations • Not clear if RTPs could be better– are they treating the right patient? • Endogeneity of RTPs • 1 and 2 year lags (medical centers with RTPs in 1994 and 1995) are not associated with costs • There does not appear to be self-selection in RTPs.

Any Questions?

Design References Trochim, W. Research Methods Knowledge Database http://www.socialresearchmethods.net/kb/ Rossi, PH, and HE Freeman. Evaluation: A systematic approach. 5th ed. New York: Sage, 1993.

Regression References • Wm. Greene. Econometric Analysis. • J Wooldridge. Econometric Analysis of Cross Section and Panel Data.

You’ve Almost Made It • June11th Mark Smith, Endogeneity • TBA: Todd Wagner: Using Stata

Econometrics with Observational Data: Research Design