Is the Propensity Score an Inferior Version of Instrumental Variables?

1. Is the Propensity Score an Inferior Version of Instrumental Variables? International Health Economics Association

2. Contributors • William H. Crown, PhD Medstat • Onur Baser, PhD Medstat • Ernst R. Berndt, PhD Massachusetts Institute of Technology, Sloan School of Management, and National Bureau of Economic Research

3. Overview • What do we mean by bias? • Propensity score • Instrumental variables • The Identification issue • Empirical example • Conclusions

4. Estimating Treatment Effects • For any given individual, an outcome can be measured as: Y=DY1+(1-D)Y0 Where D=1 if treated D=0 if not treated And Y1=outcome for treated Y0=outcome for not treated • Treatment effects are measured as: • E(Y1-Y0)

5. Econometric Approach Evaluation problem leads to switching regime model of Quandt (1972): Y1=g1(x)+u1 Y0=g0(x)+u0 Where: E(u1|x)=0 E(u0|x)=0

6. The Usual Estimate E(Y1-Y0|X, D=1) =g1(x)-g0(x)+E(u1-u0|x,D=1) This quantity is the “mean effect of treatment on the treated”

7. Sources of bias in treatment effects estimated from observational data • Treated patients and controls can have different distributions on observed variables • Outcomes and explanatory variables may not be measured in the same way for both groups • Treated patients and controls may be in different environments that affect their behavior • Treated patients and controls may have different distributions on unobserved variables

8. What matters most? • Controlling for observable variables (1-3) is most important empirically • Nevertheless, bias from unobservables is still large relative to randomized experiments Heckman, Ichimura, Todd. Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme. Review of Economic Studies, 64:605-654, 1997.

9. Propensity Score Methods: An Overview

10. Three main methods • Matching • Stratification • Regression D’Agostino. 1998. Tutorial in biostatistics: Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Statistics in Medicine, 17: 2265-2281

11. Empirical Implementation: Matching • Model probability of treatment as a function of observable variables • Match participants with controls • Nearest neighbor • Mahalanobis metric matching • Nearest Mahalanobis metric matching with calipers

12. Empirical Implementation: Stratification • Model probability of treatment as a function of observable variables • Propensity scores are grouped into strata • Treated patients and controls are compared within strata Five strata generally sufficient to remove 90% or more of bias from differences in distributions of observable variables (Rosenbaum and Rubin , 1984)

13. Empirical Implementation: Regression • Model probability of treatment as a function of observable variables • Include estimated propensity score in outcome equation as an additional covariate • Sometimes broader set of variables used for propensity score and narrower set for outcome equation

14. Simulations of Alternative Propensity Score Methods

15. Background • Overall Prescription Drug Use • Rising prescription drug expenditures • Health plans have attempted to control rising prescription drug costs by changing cost sharing provisions • Asthma-Specific Prescription Drug Use • Asthma “controller” medications underutilized • Increases in controller-to-reliever ratio are beneficial • Co-payments also used to create incentives for appropriate healthcare use

16. Overview of Asthma Medications • Controller Medications • taken daily on a long-term basis to achieve and maintain control of persistent asthma • Daily: Long-Term Control • Corticosteroids (inhaled and systemic) • Cromolyn/nedocromil • Long-acting beta2-agonists • Methylxanthines • Leukotriene modifiers Source: National Asthma Education and Prevention Program, NHLBI, (1997)

17. Overview of Asthma Medications • Reliever or acute rescue medications • quick-relief medications taken to provide prompt reversal of acute airflow obstruction and relief of accompanying bronchoconstriction • As-needed: Quick Relief • Short-acting beta2-agonists • Anticholinergics • Systemic corticosteroids Source: National Asthma Education and Prevention Program, NHLBI, (1997)

18. Methods • Data source: • 1995–2000 MarketScan files (a large database of private sector inpatient, outpatient, and prescription drug medical claims) • Area Resource File for county-level race and income information

19. Study Sample • Enrolled in a participating health plan during at least two of the years 1995-2000 • Patients with evidence of asthma • Identified by diagnosis for asthma on patient claims or prescriptions

20. Measures • Plan type • Fee-for-service (FFS) versus non-FFS plans • Co-payments • Out of pocket payments for asthma drugs by therapeutic class for each plan • Out of pocket payments for emergency room visits, outpatient visits, and hospitalizations for each plan • Co-morbidities • Counts of unique ICD-9 diagnosis codes, flags for specific comorbidities commonly found with asthma • Utilization • claims and encounters over the study period

21. Analytic Approach • Bootstrap estimates of alternative propensity score approaches • Matching • Stratification • Regression • Standard errors based on 100 replications

22. Propensity Score Matching Results

23. Propensity Score Stratification Results

24. Propensity Score Regression Results

25. Instrumental Variables: Preferable to Propensity Scores?

26. The Problem Y=B0+B1X1+B2X2+…+BkXk+Bk+1Xk+1+u E(u)=0 Cov(Xj,u)=0 for j=1,…k But Cov(Xk+1,u) is not equal to 0 Therefore Xk+1 is “endogenous”

27. Sources of endogeneity • Omitted variables • Measurement error • Simultaneity

28. Sample selection bias • Sample selection bias is a special case of missing variables. • If an unobserved variable Xk+1 is correlated with the treatment variable Xk then Xk is endogenous to Xk+1

29. The IV approach • We need to find an observable variable Z not in the outcome equation that satisfies two conditions: • Cov(Z,u)=0 •  is not =0 Where  is the partial correlation between Z and Xk From the reduced form equation: Xk=C0+C1X1+C2X2+…Ck-1Xk-1+ Z+ê

30. IV Estimation to Control for Selection Bias • Estimate the probability of treatment as a function of all exogenous variables—including those that are associated only with treatment and not with outcomes. • Substitute this reduced form (predicted probability of treatment) into structural model for outcomes. • This becomes reduced form for outcomes.

31. Identification • If there is exactly one instrumental variable for treatment, it is possible to solve for the structural parameter of the effect of treatment on outcomes. This is known as exact identification. • If there is more than one instrumental variable for treatment, the outcome equation is over-identified.

32. IV versus Propensity Score • Both methods start with estimating probability of treatment as a function of exogenous variables. • IV adds explicit criteria of identification • IV also adds standard error adjustment in outcomes equation to recognize that IV is an estimated variable.

33. Some Possible Instruments • Copays—stratification by year, then plan, then controller and reliever variables. Calculate mean copays for each class of drug by plan. Calculate ratio. • Prescribing %--stratification by year, then tax ID, then patients getting controller, reliever, and combination therapy. Calculate % distribution of asthma prescriptions within tax ID. Sums to 100% so leave one out of model.

34. Results of Comparisons of IV and Propensity Score Methods

35. Where Does This Leave Us? • Some of the concepts from IV can be usefully combined with propensity score methods—particularly the concept of exclusionary variables • Regardless of approach used, propensity score methods adjust for observable variables only. • IV compares favorably to Mahalanobis matching with calipers.

36. Combination methods • Heckman et al (1997) argue for the use of propensity score matching, coupled with differences in differences methods to correct for selection bias. • This method is attractive when the outcome variable is observed at baseline and at the end of the study period for both study participants and controls.

37. Summary (1) • Propensity score methods can be effective in controlling bias from observable variables. This is often the lion’s share of the bias (i.e., 80% or more). • If distributions of propensity scores are not comparable for treated and untreated groups, however, inferences must be confined to range where they are comparable.

38. Summary (2) • IV methods appear to generate results similar to Mahalanobis matching with calipers and control for both observable and unobservable variables. Also provide standard error correction on estimated variable. • However, it is valuable to match even if only to understand the overlap of distributions in comparison groups • Application of techniques like differences in differences to matched observations are a non-parametric alternative to IV for removing bias from unobservable variables as well as sources of bias from observable variables