H ealth E ffectiveness R esearch Ce nter

1. Estimating Treatment Effects with Observational Data using Instrumental Variable Estimation: The Extent of Inference John M Brooks. Ph.D. Health Effectiveness Research Center (HERCe) Colleges of Pharmacy and Public Health University of Iowa June 8, 2004 HealthEffectivenessResearchCenter

2. Research Goal: Estimate casual relationships between "treatment" and “outcome” in healthcare... • treatment on outcome • behavior on outcome • system change on outcome

3. The best estimation method to make inferences about these relationships is a function of: • 1. the manner in which the researcher collects data; and • 2. the approach used to control for “confounding factors” • confounding factors: factors related to both the • treatment and outcome.

5. Sources of Treatment Variation in Health Care • 1. Randomized Controlled Trials: study of patients with • a given medical condition in which treatment is • randomly assigned. • • Why randomly assign treatment to patients? • To help ensure that estimated treatment affects are • attributable to the treatment and not unmeasured • confounders. • The Gold Standard

6. • Why don’t we do more Randomized Controlled Trials between approved treatments? → ethical problems → expensive and time-consuming → little motivation → inability to generalize

7. 2. Observational Healthcare Databases • • Database Types: • → Claims: medical service treatment claims from • individuals with health insurance • → Provider-Specific: databases describing the • utilization of a set of providers. • → Health Care Surveys: surveys of patients or • providers detailing health • care utilization.

8. • Strengths: → plenty of variation in treatment choice; → potentially enhanced ability to generalize – reveals variation in treatment choice across a variety of clinical scenarios; → can assess treatments in practice – estimate “effectiveness”; → unobtrusively collected; → the power of large numbers and time.

9. • Weaknesses: • → data usually not collected for researcher purposes; • → missing information; • - care not covered is not observed • - care not claimed is not observed • - claim form limitations • - nuances of illness, treatment, and patient that can’t • be recorded on claims forms • → patient enrollment variation; • → confounding information may be unobserved.

10. Is the Main Source of Weakness with Observational Data • Unmeasured Confoundersor Treatment Selection Bias? • 1. Unmeasured Confounders • • Unmeasured Confounders argument: • → homogenous treatment effect; • → unmeasured factors related to both treatment • and outcome is the source of bias.

11. • Assume true outcome relationship is: Y = ao + a1•T + a2•L + e where: Y = measure of outcome (e.g. 1 if survive to a certain time period, 0 otherwise); T = 1 if receive treatment, 0 otherwise; and L = additional factor (e.g. severity, other treatments). Goal is to estimate a1 – the effect of treatment on outcome.

12. • For Estimation Suppose: • → L is not measured and the estimation model is: • Y = ao + a1•T + u where: • u = (a2•L + e) • → L is related to Y (a2≠ 0); and • → T and L are related (Cov(T,L) ≠ 0). • Cov(T,L) – covariance of T & L. Cov(T,L) ≠ 0 essentially means that T & L move together.

13. • Define the ordinary least squares (ANOVA) estimate of • a1 as . • → It can be shown that under these assumptions is • a biased estimate of a1 through its expected value: • = a1 + Cov(T,L)•a2 • → Also note that will equal a1 if either: • -- Cov(T,L) = 0; or • -- a2 = 0.

14. • Suppose theory about the unmeasured variable “L” suggests: → “a2 < 0” (patients with higher severity have lower cure rate). → Cov(T,L) > 0 (treated patients are generally more severe). • Plug in “signs” into our expected value formula to find: < a1.

15. • Problem with the Unmeasured Confounders argument to describe bias in observational data: → It does not provide a theoretical foundation to link treatments to unmeasured factors.... Why is Cov(T,L) ≠ 0? → In the case we just described, if treatment effect (a1) is the same for all patients, why would Cov(T,L) > 0? Perhaps patients getting treatments: -- live in areas with high/low poverty; -- live in areas with more pollution; or -- also tend to get other unmeasured treatments.

16. 2. Treatment Selection Bias (the gestalt underlying most • negative reviewer’s comments) • • Treatment Selection Bias argument: • → heterogeneous treatment effect -- Cov(T,L) • reflects the decision-maker’s beliefs about the • differences in treatment effectiveness across • patients; and • → bias comes from unmeasured factors (severity, • other treatments) related to the treatment’s • expected effectiveness that affects both • treatment choice and outcome.

17. • Assume true outcome relationship is: Y = bo + (b1 + b2•L) •T + b3•L + e where: Y = measure of outcome (e.g. 1 if survive to a certain time period, 0 otherwise); T = 1 if receive treatment, 0 otherwise; L = unmeasured factor (e.g. severity, other treatment); b3 = the direct effect of L on Y; and (b1 + b2•L) = effect of T on Y that depends on L.

18. → L is now related to T through theory linking "treatment choice" to the decision-makers expectations of treatment benefits across patients with different “L”. T = co + c1•L + c2•W + v where: T = 1 if receive treatment, 0 otherwise; L = unmeasured factor (e.g. severity, other treatment) affecting treatment choice through expected treatment effectiveness; and W = other factors affecting treatment choice. If decision makers use L in treatment decisions, c1≠ 0 and Cov(T,L) ≠ 0.

19. • Ultimate goal should be to estimate (b1 + b2•L) – the effect of treatment T on outcome Y across levels of L. • For estimation suppose: → L is not measured and it is wrongly assumed by the researcher that the effect of T is homogenous, the estimation model is: Y = ao + a1•T + u where: u = (b2•L•T + b3•L + e)

20. • Define the ordinary least squares (ANOVA) estimate of a1 as . → It can be shown that the expected value of is: → If c1 = 0 (no selection based on L), then becomes: Yields an average estimate that depends on the mix of “L” in the population (e.g. RCT using a broad population).

21. • How does c1 • (b2 + b3) affect this estimate? → Assume that L is unmeasured illness severity and that higher L means more severe illness. → Higher L lowers survival which implies b3 < 0. → If treatment benefit is less for more severe cases (e.g. surgery for heart attacks) then: benefit falls less treatment with higher in more severity severe cases Estimate of average population treatment benefit will be biased high.

22. → If treatment benefit is greater for more severe cases (e.g. antibiotics for otitis media) then: benefit increases more treatment with higher in more severity severe cases Estimate of average population treatment affect is biased but sign can not be determined.

23. • So what do we have here? → Observational data contains enormous treatment variation. → Treatment choice may be related to the selection or sorting of patients using unmeasured (to the researcher) characteristics that are related to expected outcomes. → Under “selection”, standard statistical techniques yield biased estimates that don’t apply to anyone anyway. Do we have any alternatives?

24. Instrumental Variables (IV) Estimation and “Subset B” • IV estimation offers consistent estimates for a subset of patients (McClellan, Newhouse 1993): Marginal Patients: patients whose treatment choices vary with measured factors called instruments that do not directly affect outcomes. • McClellan and Newhouse argue that estimates of treatment effects for Marginal Patientsare useful: → They are estimates for patients for whom the benefits of treatment are the least certain – patients least like those in RCTs. → Estimates may be more suitable than RCT estimates to address the question of whether existing treatment rates should change.

26. • Patients in Subset B are interesting because: → the “best” treatment choice (treat or don’t treat) is least certain; → treatment or no-treatment for a patient in this subset is not considered bad medicine – the “art” of medice; → the possibility of gaining new RCT evidence for patients in this subset is remote (ethics, motivation); → McClellan et al. 1994 argue that policy interventions affect mainly the treatment choices for patients in this subset; and → Non-clinical factors (e.g. provider access, market pressures) affect mainly the treatment choices of patients in this subset.

29. • IV estimation involves: 1. Finding measured variables or “instruments” (Z) that: a. are related to the possibility of a patient receiving treatment (cov(T,Z) ≠ 0); and b. are assumed (through theory) unrelated directly to Y or to unmeasured confounding variables (cov(Z,L) = 0). The theoretical basis for “Z” variables should come from a model of treatment choice – the “W” variables in: T = co + c1•L + c2•W + v where: W = other factors affecting treatment choice.

30. • IV estimation involves con’t: 2. Grouping patients using values of the “instrument”. 3. Estimate treatment effects for marginal patients by exploiting treatment variation rate differences across patient groups. Local Average Treatment Effect -- (Imbens & Angrist 1994)

31. • For example, if an instrument divides patients into two groups, a simple IV estimate can be found by calculating: 1. the overall treatment rate in each group (ti = treatment rate in group “i”); and 2. the overall outcome rate in each group (yi = outcome rate in group “i”); and estimate: where: = average treatment effect for the “marginal patients” specific to the instrument used in the analysis – only those patients whose treatment choices were affected by the instrument who must have come from Subset B.

33. • We have treatment rates for each group: Closer Group Treatment Rate: .60 Further Group Treatment Rate: .50 Suppose we also measured “cure” rates in both groups: Closer Group Cure Rate: .40 Further Group Cure Rate: .38 • Four numbers lead to the following IV estimate:

34. • Strict Interpretation: → If the treatment rate in the Further Group was increased .01 percentage point (e.g. .50 to .51) by increasing treatment for the M patients in the Further Group, the Cure rate in the Further Group would increase .002 (.01 • .2) – from .38 to .382. • Stretched “Policy-Relevant” Interpretation (McClellan et al. 1994) → A behavioral intervention that increases the overall treatment rate by .01 percentage point (e.g. .55 to .56) would lead to an increase in the cure rate of .002 (.01 • .2).

35. • Stretched interpretation assumes that the treatment effect for patients in Subset B is fairly homogenous and an IV estimate from a single instrument can be generalized to all patients in Subset B. This allows one to say: • Stretched interpretation is not perfectly accurate if treatment effects are heterogeneous within Subset B and different instruments affect treatment choices from different patients within Subset B. → Results from a single instrument may still be more appropriate than assuming RCT results apply to Subset B. → Ability to generalize results may increase if more than one instrument is used in an IV analysis.

36. • IV qualifiers to remember: • → second property of IV variables (cov(Z,L) = 0) is • forever an assumption (unless more data are • obtained); and • → unmeasured but correlated treatments may still bias • estimated treatment benefits. • Researchers should fully qualify their IV estimates – don't oversell.

37. Hypothetical Example to Demonstrate “4-Number” Result Suppose: • 2100 children with Acute Otitis Media (AOM) in a population. • Two treatment possibilities: 1. antibiotics; 2. watchful waiting. • The patients in our sample are in one of three severity types “low”, “medium”, and “high” • Severity type is observed by the provider/patient but is not observed by the researcher.

38. • The 2100 patients are distributed across severity type in the following manner: severity type High Medium Low number of patients 800 800 500 • The actual underlying cure rates for each severity type by treatment are: severity type treatment High Medium Low antibiotics .95 .97 .98 watchful waiting .80 .90 .98

39. → Higher severity means a lower the cure rate in general (b3 < 0). → Antibiotics have a higher curative effect in more severe patients and offer no advantage to the less severe (b2 > 0). ASSUMPTION: Treatment effects are heterogenous. → All providers have inclination that antibiotics work well in the "high" severity patients; have little effect on the "low" severity patients; but the effect in the "medium" type is unknown to providers. Leads to selection bias...the more severe kids are treated (c1 > 0).

40. Potential Methods to analyze: 1. Randomize Patients Into Treatments -- ANOVA 2. Providers Assign Treatments -- ANOVA 3. Instrumental Variable Grouping

41. 1. Randomize Patients Across Population – ANOVA. Patient Treatment Assignments After Randomization by Severity Type severity type patient groups High Medium Low antibiotics 400 400 250 watchful waiting 400 400 250

42. Expected average cure rates for each group: • Unbiased average antibiotic treatment rate for the entire population (.965-.881 = .084), but • To whom does it apply? A patient randomly chosen from an urn? Are patients chosen from urns?

43. 2. Providers Assign Treatments -- ANOVA If providers follow “inclinations”, we may end up with something like: Number of Patients Assigned by Providers to Each Treatment Group by Severity Type severity type patient group High Medium Low antibiotics 800 400 0 watchful waiting 0 400 500

44. Expected average cure rates for each group: • For this population the average treatment effect is (≈.084). We find a biased low estimate of the antibiotic treatment effect for the average patient (.957 - .944 = .013 < .084). • To which patients does this estimate apply?

45. 3. Instrumental Variable Grouping -- Further: • a. Assume information is available to approximate • distances from patients to providers • • address of patient • • supply of providers in area around patients • b. Evidence suggests that patients in areas with more • physicians per capita have a higher probability of being • treated with antibiotics for their AOM than patients in • areas with fewer physicians per capita.

46. If “b” is true, divide 2100 patients into two groups based on the physicians per capita in the area around their home: Group 1: the group of patients living in areas with a higher number of physicians per capita; Group 2: the group of patients living in areas with a lower number of physicians per capita;

47. Using our assumptions, does this grouping qualify as an instrument? 1. Doc supply related to treatment? Yes, if patients tend to go to the closest provider for treatment. If true, and providers follow inclinations we may see treatment patterns something like: Patient Treatment Assignments by Severity Type patient severity type group High Medium Low Group 1 100% antibiotics 80% antibiotics 100% W.W. 20% W.W. Group 2 100% antibiotics 30% antibiotics 100% W.W. 70% W.W.

48. 2. Is grouping related to unmeasured confounding variables (e.g. severity)? Related to severity only if parents chose residences in expectation of the severity of a future acute condition. If not related to severity, we assume equivalent severity distributions across groups: Number of Patients in Each Group by Severity Type severity type patient group High Medium Low Group 1 400 400 250 Group 2 400 400 250

49. Expected average estimated cure rates for these groups: Well, (.959428 - .946092) = .013336 doesn't appear to reveal much of anything…

50. Now look at the antibiotic treatment rate in each group: 720/1050 = .68571 in Group 1 520/1050 = .4952381 in Group 2 These differences also don't look very informative…. The IV change in the cure rates resulting from a one unit increase in the drug treatment rate equals: • This estimate is the average difference in the antibiotic cure rate for the marginal or in this example the “Medium” severity patients.