Sukyeong Pi Larry Featherston Employment and Disability Institute Cornell University Feb. 21, 2009

2. Agenda Randomized Controlled Trial Observational Studies Propensity Score Matching Example Limitations of PSM

3. Randomized Controlled Trial (RCT) A research study in which the participants are randomly assigned groups to objectively compare different interventions RCT is recognized as a sound scientific method: Gold Standard for making causal inferences and making policy decisions Control for subject selection bias: Minimize subject differences between groups

4. Limitations of RCT Philosophical/Ethical Issue: Against the obligation to offer each student optimal treatment Strategic Issues: Requires time and specialized expertise, Generalizability Issue Tactical Issues: Issues of treatment fidelity and integrity Logistical Issues: Challenges finding adequate numbers of subjects, Expensive requiring substantial resources

5. Advantages of Observational Studies Address chief criticism of RCTs: Genealizability Availability, Cost, Time Serve as a rich source of descriptive information Examine exposure in real life ? Policy decisions possible Large sizes permit investigation of exposures with smaller effect sizes

6. Observational Studies Selection Bias: No control for group assignment (Ignorability of treatment assignment) Baseline characteristics of comparison groups are different in ways that affect the outcome due to observed or unobserved confounders. One approach to remove the bias in nonrandomized experiments is propensity score matching.

7. Propensity Score Matching Definition: The conditional probability (0 to 1) of receiving a given exposure (treatment) given a vector of measured or observed covariates. Assumption of RCT: the probability to be assigned to treatment group is 0.5 PS reduces baseline information to a single composite summary of the covariates, thus minimizing differences and improving comparability between two groups in observational research

8. Procedures of Propensity Score Analysis Estimate propensity for treatment given covariates using Logistic Regression method: Save predicted value e (x) = �0 + �1?X1i + �2?X2i +� + �n?Xni + ei Propensity Score = e(x) / {1+e(x)} Balance check Compare propensity scores between Tx and Ctl groups 3. Estimate effect of treatment on outcome using PS a. Regression Model b. Stratification c. Matching

9. EXAMPLE Research Question: What is the effect of VR services? (LR found top three services related to successful VR outcome: On the Job Support, Rh Tech, Job Placement) Data Source: 2006 RSA 911 data (including consumers closed after IPE developed; N=352,138) IVs: Gender, Race/Ethnicity, Level of Education, Work Status at Application, Primary Source of Support, SSI/DI, Type of Disability Intervention (tx): Types of Services Outcome: Type of Closure

10. Step 0: Data Set-up Variable Selection by crosstabulation of covariates and type of closure (outcome) Covariates for this example (dummy var.) - Gender (2) - Race/Ethnicity (3): White no Hispanic, African, others - Education (3): <12 yr, 12 yrs (incl. SE cert), >12 yrs - Work Status at App (3): Emp wo sup, Other emp, No emp - Source of Support (2): Personal Income, Others - SSI/DI (2): Y/N - Disability (5): Sensory, All Mental with SA, LD/ADHD, MR/Autism, Others

11. Step 1: Propensity Scores Goal: to include all variables that play a role in the selection process, including interactions and other nonlinear terms and variables that show weak relations to outcome (e.g., p<.10 or p<.25) (Rosenbaum & Rubin,1984) �Unless a variable can be excluded because there is a consensus that it is unrelated to outcome or is not a proper covariate, it is advisable to include it in the propensity score model even if it is not statistically significant.� (Rubin & Thomas,1984) In the example, all variables were included for PS computation

12. Step 1: PS by Stepwise LR

13. Step 2: Balance Check Compare two groups in their distributions using descriptive statistics and t-tests Box plot graph illustrates some overlaps (similar characteristic band of propensity scores) between two groups No overlap indicates that the differences in outcome was drawn from group differences (Selection Bias), not from the service effect (e.g., rehab tech services)???

14. Step 2: Check Distribution/Balance

15. Step 2: Check Distribution/Balance

16. Step 2: Balance Check

17. Step 2: Check Distribution/Balance

18. Step 2: Check Distribution/Balance

19. Step 3: Analysis with PS Three techniques are commonly used to reduce selection bias and increase precision with PS - Regression (covariance) adjustment - Stratification - Matching

20. Step 3: Analysis I - Regression Treat the PS as an additional covariate in multivariable regression model As a composite of confounders, PS can reduce bias in the estimate of the treatment effect by adjusting for the pattern of observed confounders. Treatment effect appears more efficient when using PS as a covariate after stratification within the strata

21. Step 3: Analysis II - Stratification Solution for the problem of dimensionality to make two groups comparable (2k subclasses needed for k covariates) PS as a scalar summary of all the observed background covariates, stratification can balance the distributions of the covariates Five strata based on the PS will remove over 90% of the bias in each of the covariates (Cochran, 1968)

22. Step 3: Analysis II - Stratification

23. Step 3: Analysis II - Stratification

24. Step 3: Analysis � Stratification (26 closures)

25. Step 3: Analysis III - Matching

26. Step 3: Analysis III - Matching

27. Interpretation What do you think? Do you think PS gives better ideas to make a causal inference?

28. Limitations of PSM With only observed covariates; No control for unobserved (e.g., age for this example) Inspection of the overlap between conditions before matching or other techniques: Group overlap must be substantial (e.g., rehab tech svcs) Best with large samples