Experimental Evaluations

1. Experimental Evaluations Methods of Economic Investigation Lecture 4

2. Why are we doing this? • Experiments are becoming more popular in economics • Development Economics, “Field Experiments” • Behavioral Economics, “Laboratory Experiments” • Sometimes experiments don’t go as planned • Need some econometrics to prove no problems • Need some econometrics to fix the problems • Good baseline for understanding attribution of estimated differences • Can compare other forms of evaluation to the “experimental ideal”

3. Some Basic Terminology • Start with example where X is binary (though simple to generalize): • X=0 is control group • X=1 is treatment group • Causal effect sometimes called treatment effect • Randomization implies everyone has same probability of treatment • We can change this a bit with weights…

4. Why is Randomization Good? • If X allocated at random then know that X is independent of all pre-treatment variables in whole wide world • If you don’t have a random sample, then this will apply to internal comparisons but will make external comparisons more difficult • Implies there cannot be a problem of omitted variables, reverse causality etc • On average, only reason for difference between treatment and control group is different receipt of treatment

5. Pre-treatment characteristics must be independent of randomized treatment • Define the joint distribution of X and W as f(X,W) • Can decompose this into: f(X,W)=fX│W(X│W)fW(W) • Now random assignment means fX│W(X│W)=fX (X) • This implies: f(X,W)=fX (X)fW(W) • This implies X and W independent

6. What are we estimating • In words: We want to know if, on average, there is a difference in the outcome of interest between the control group and the treatment groups • In Math: we want an estimate of:

7. Estimating Treatment Effects • Take mean of outcome variable in treatment group • Take mean of outcome variable in control group • Take difference between the two • This is how you learned it in theory and it’s right BUT: • Does not generalize to where X is not binary • Does not directly compute standard errors

8. Estimating Treatment Effects: A Regression Approach • Run regression: yi=β0+β1Xi+εi • The OLS estimator of β1 is an unbiased estimator of the causal effect of X on y: • To see why: • Recall that the OLS estimates E(y|X) • E(y|X=0)= β0 so OLS estimate of intercept is consistent estimate of E(y│X=0) • E(y|X=1)= β0+β1so β1 is consistent estimate of E(y│X=1) -E(y│X=0) • Hence can read off estimate of treatment effect from coefficient on X • Approach easily generalizes to where X is not binary • Also gives estimate of standard error

9. Computing Standard Errors • Unless told otherwise regression package will compute standard errors assuming errors are homoskedastic i.e. • Even if only interested in effect of treatment on mean X may affect other aspects of distribution e.g. variance • This will cause heteroskedasticity • This is a second order issue: your coefficient estimates are “right” (consistent) • HOWEVER you can’t do any inference because your OLS standard errors are inconsistent

10. ‘Robust’ Standard Errors • Also called: • Huber-White standard errors • Heteroskedastic-consistent standard errors • Statistics Approach • Get variance of estimate of mean of treatment and control group • Sum to give estimate of variance of difference in means

11. A Regression-Based Approach • Can estimate this by using sample equivalents • Note that this is same as OLS standard errors if X and ε are independent

12. A Regression-Based Approach • Have to interpret residual variance differentyl – not common to all individuals but the mean across individuals • With one regressor can write robust standard error as: • Simple to use in practice e.g. in STATA: . reg y x, robust

13. Summary So Far • Econometrics very easy if all data comes from randomized controlled experiment and everything went as planned • Just need to collect data on treatment/control and outcome variables • Compare means of outcomes of treatment and control groups • Everything doesn’t always go as planned • Treatment effects are small • Randomization fails • Non-compliance to treatment

14. How to avoid Experimental problems? • Get info on other regressors • Can get consistent estimate of treatment effect without worrying about other variables • But there are reasons to include other regressors: • Improved efficiency • Check for randomization • Improve randomization • Control for conditional randomization • Heterogeneity in treatment effects

15. The Uses of Other Regressors I: Improved Efficiency • Don’t just want consistent estimate of causal effect – also want low standard error (or high precision or efficiency). • Especially important if treatment effects are small • Standard formula for standard error of OLS estimate of βis σ2(X’X)-1 • σ2 comes from variance of residual in regression – (1-R2)* Var(y)

16. The asymptotic variance of βˆ is lower when W is included • Proof: (Will only do case where X and W are one-dimensional) • When W is included variance of the estimate of the treatment effect will by first diagonal element of:

17. Proof (continued) • Now: • Using trick from end of notes on causal effects we can write this as:

18. Proof (continued) • Inverting leads to • By randomization X and W are independent so: • The only difference is in the error variance – this must be smaller when W is included as R2 rises

19. The Uses of Other Regressors II: Check for Randomization • Randomization can go wrong • Poor implementation of research design • Bad luck • If randomization done well then W should be independent of X – this is testable: • Test for differences in W in treatment/control groups • Probit model for X on W

20. The Uses of Other Regressors III:Improve Randomization • Can also use W at stage of assigning treatment • Can guarantee that in your sample X and W are independent instead of it being just probabilistic

21. The Uses of Other Regressors:Adjust for Conditional Randomization • Conditional randomization is where probability of treatment is different for people with different values of W, but random conditional on W • Why have conditional randomization? • May have no choice • May want to do it (c.f. stratification) • MUST include W to get consistent estimates of treatment effects

22. Controlling for Conditional Randomization • What we know about our treatment • X is by construction correlated with W • But, conditional on W, X independent of other factors • But must get functional form of relationship between y and W correct – matching procedures • This is not the case with (unconditional) randomization

23. The Uses of Other Regressors: Heterogeneity in Treatment Effects • So far have assumed causal (treatment) effect the same for everyone but there’s no reason to believe this • May use other variables to test if treatment effect is different between two groups

24. Estimating Heterogeneous Treatment Effecs • Start with case of no other regressors, suppose that there are different beta’s for everyone, so that: yi=β0+β1iXi+εi • Random assignment implies X independent of β1i • Sometimes called random coefficients model

25. What treatment effect to estimate? • Would like to estimate causal effect for everyone – this is not possible because we only have one realization of the treatment effect of each person • Instead, we estimate some average this is called the Average Treatment Effect (ATE) • We can estimate ATE with OLS because:

26. Proposition 2.5OLS estimates ATE • Proof for single regressor:

27. Observable Heterogeneity • Full outcomes notation: • Outcome if in control group: y0i=γ0’Wi+u0i • Outcome if in treatment group: y1i=γ1’Wi+u1i • Treatment effect is (y1i-y0i) and can be written as: (y1i-y0i )=(γ1- γ0 )’Wi+u1i-u0i • Note treatment effect has observable and unobservable component • Can estimate as: • Two separate equations • One single equation

28. Combining treatment and control groups into single regression • We can write: • Combining outcomes equations leads to: • Regression includes W and interactions of W with X – these are observable part of treatment effect • Note: error likely to be heteroskedastic

29. Units of Measurement • Causal effect measured in units of ‘experiment’ – not very helpful • Often want to convert causal effects to more meaningful units • Health Example last week: How much does an extra $ in income get you in better health? • How to interpret improvements in X percent or X standard deviations

30. Simple estimator of this would be: • where S is the factor we change in the treatment • Takes the treatment effect on outcome variable and divides by treatment effect • Not hard to compute but how to get standard error? • We can do it in a regression • Can use your X as an instrument (more on this when we do IV)

31. Uses of Extra Regressors: Partial Compliance • So far: • in control group implies no treatment • in treatment group implies get treatment • Often things are not as clean as this • Treatment is an opportunity—not everyone takes it up • Close substitutes available to those in control group • Implementation not perfect so some people get into or out of treatment despite RA

32. Some Terminology: ITT • Z denotes whether in control or treatment group – Intention To Treat (ITT) • X denotes whether actually get treatment • With perfect compliance: • Pr(X=1│Z=1)=1 • Pr(X=1│Z=0)=0 • With imperfect compliance: 1>Pr(X=1│Z=1)>Pr(X=1│Z=0)> 1>Pr(X=0│Z=0)>Pr(X=0│Z=1)>0

33. What Do We Want to Estimate? • ‘Intention-to-Treat’: ITT=E(y|Z=1)-E(y|Z=0) • This can be estimated in usual way in the regression • Pro’s • Don’t worry about selection in compliance • Get an average effect of your intervention • Cons • Don’t know what the effect of the actual treatment is—combined effect of treatment and non-compliance

34. More Terminology: TOT • What if we only looked at those who took up the treatment then we would estimate Treatment Effect on Treated (TOT) • Pros • Looks directly at treatment • Cons • May be biased by the selection of individuals into treatment and control groups

35. Estimating TOT • Can’t use simple regression of y on Z • But should recognize TOT as Wald estimator • Can estimated by regressing y on X using Z as instrument • Relationship between TOT and ITT:

36. Next Steps… • What to do when we can’t do experiments because • They didn’t work out like planned • We couldn’t do one on this particular issue • Natural Experiments • Use variation in the world • Several different methods over the next few classes