1 / 30

Non-Experimental Evaluations

Non-Experimental Evaluations. Methods of Economic Investigation Lecture 5. Today’s Lecture. Review the Evaluation Problem Review the Experimental Approach Issues of Compliance Natural Experiments. Review: The Evaluation Problem. In words:

mgardiner
Download Presentation

Non-Experimental Evaluations

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Non-Experimental Evaluations Methods of Economic Investigation Lecture 5

  2. Today’s Lecture • Review the Evaluation Problem • Review the Experimental Approach • Issues of Compliance • Natural Experiments

  3. Review: The Evaluation Problem • In words: • We want to evaluated the effect of a treatment on an outcome • Worried that some omitted/unobservable factor affects both who is “treated” and the outcome. So let’s do this very simply: there’s some treatment T it occurs at some time k

  4. Homogeneous Treatment Effects • Before the treatment (so at time t<k) • After the treatment (so at time t≥ k) Relationship between characteristics X and the Outcome Y Treatment Effect

  5. The Fundamental Identification Problem • Worry that assignment to treatment is not random • Assignment process leads to non-zero correlation between treatment (T) and the error (U) • Probably because participation decision is based on characteristics that affect outcome (Y) too • If you can’t observe EVERYTHING that affects both treatment probability and outcome then your OLS will not be a valid approach

  6. Experimental Approach • Randomization implies that T is independent of X • Don’t have to worry about what factors generated selection into treatment because it is by design • May still include some Xs to increase efficiency • Can use these X’s to show that X and d are uncorrelated

  7. Heterogeneous Treatment Effects • For simplicity for the moment ignore the other regressors (X) • Define the population mean of the treatment effect • Last time, we showed that in this situation when the heterogeneity does not affect probability of treatment then OLS estimates ATE consistently ( ) Treatment Effect for individual i

  8. Defining TOT • Let’s parameterize this a bit more so that an individual’s treatment effect is the mean treatment effect and some individual specific component • Then, we can define the mean impact of the treatment on the treated (TOT) as Mean deviation of the treatment effect among the treatment group population

  9. OLS with Heterogeneous Treatment Effects • The OLS estimate will be: • Our treatment effect is now: • Errors are heteroskedastic (that’s ok)

  10. Heterogeneous TE and ATE • If E(εidi)≠0 then the OLS estimator identifies: • Worry that individuals factors that are unobservable might be correlated with treatment =0 because of Random Assignment TOT

  11. Without Randomization we’re back to the basic problem in evaluation • We want to evaluate: E(Y| T=1) – E(Y | T=0) • We have some treatment effect: Yi = a + bi*Ti + ei • Then we know that • E(Y| T=0) = a + E(ei| T=0) • E(Y | T= 1) = a + b + E(ei| T=1) • Our Estimate will then be: • E(bi) + [E(ei| T=1) – E(ei| T=0)] Treatment Effect Selection Effect

  12. In general there are 4 kinds of Non-Experimental Evaluations 1. Controls, controls, controls… 2. “Natural Experiments” 3. Matching 4. Selection Models 5. Structural Models “Reduced Form” Replicate experiment with an ‘engineered’ control group “Parametric/ Structural” Model the form of the selection effect or bias and apply a statistical correction

  13. Controls method • Basic goal—conditional independence so that it’s like our conditional/stratified randomization • Focus of next week’s class: • If we don’t omit anything, then we’re set • Trade-off between omitted variables and “over-controlling” • Even if we can’t measure everything, maybe be able to control with “fixed effects” and “interactions” (both are error component models)

  14. Pros and Cons • Pros • Transparent to others • Can be convincing with robust controls and well-defined outcomes • Cons • VERY data intensive • Can be hard to measure everything • Not very convincing in complicated settings

  15. “Natural Experiment” • Use an explicit source of variation • Don’t model (or at least don’t use a model) of underlying behavior based on behavior/preferences • ‘Natural’ or ‘Quasi’ Experiments • Used to refer to situation that is not experimental but is ‘as if’ it was • Not a precise definition – saying your data is a ‘natural experiment’ makes it sound better • Refers to case where variation in X is ‘good variation’ (directly or indirectly via instrument)

  16. A Famous Example: London, 1854

  17. The Case of the Broad Street Pump • Regular cholera epidemics in 19th century London • Two Theories on Why: • Widely believed to be caused by ‘bad air’ • John Snow thought ‘bad water’ was cause • An ideal experimental design: • give some people good water and some bad water • Ethical Problems with this

  18. Soho Outbreak: August/September 1854 • Observation 1: People closest to Broad Street Pump most likely to die • Does this distinguish between the two theories? • NO—breathe same air so does not resolve air vs. water hypothesis

  19. Distinguishing between theories • Observation 2: Some places near the well had few deaths • Nearby workhouse few deaths (own well • Nearby brewery had own no deaths (workers all drank beer & own well) • Observation 3: Some people far away died • Woman died in Hampstead • Had the habit of having water from pump deliver by her niece in Islington

  20. Why is this a Natural experiment? • Good variation: • water supply ‘as if’ it had been randomly assigned (existed before outbreak) • other factors (‘air’) held constant • Can estimate treatment effect • difference in means • run regression of death on water source distance to pump, other factors

  21. What’s that got to do with it? • Strongly suggests water the cause • Certainly relative to the air counter-theory • Investigation of well found contamination by sewer • This is non-experimental data but analysed in a way that makes a very powerful case

  22. Elements of Non-experimental Evaluation • Good source of variation (we’re going to call this “EXOGENOUS VARIATION”) • Good measure of treatment and control group outcomes • Good measure of treatment and control group control variables

  23. Methods for Analysing Data from Natural Experiments • If data is ‘as if’ it were experimental then can use all techniques described for experimental data and then some: • OLS • Simple conditional means difference as in experimental data • Difference-in-differences (really just like OLS but with a pre-treatment regression too) • Instrumental Variables

  24. ‘Natural Experiments’ Pros and Cons • Advantages: • source of variation is clear • “model free” • Disadvantages: • Identifying assumption may be questionable or hard to believe (internal validity) • may tell you about this experiment but “local” in that it does not tell you about preference parameters (External validity) • using it to make simulations as to other policy changes may be bad

  25. Matching Methods • Non-parametric (no functional form assumption) approach • Take controls and use combined variation to reconstruct experimental conditions • The goal: on everything one can observe for a treatment group individual find a ‘nearest neighbor’

  26. Matching Pros and Cons • Pros • Very flexible in determining “propensity” for treatment • Can use all the variation across all variables (relative to simple control methods) • Cons • Need a lot of common support for you control and treatment groups (i.e. need good matches) • Heavy data requirements • Heterogeneous treatments hard to measure

  27. Selection Correction • Parametrically model the way selection occurs • Famous Example: Female Labor Force Participation • A special case of “structural estimation” but can be used in combination with any methods when you’re worried about a specific form of selection bias

  28. A quick word on Structural Estimation • In some cases, we can use theory to model the endogeneity. • Often this is based on utility or objective function (e.g. profit) maximization theory • In the original structural literature, there was little attention to identification and the results may be identified by nonlinearities and parametrization. • This is no longer the norm, with more attention being placed on identification. • Can also use this more loosely to constrain your estimation to realm of the likely or feasible

  29. Structural Estimation Pros and Cons • Advantages: • once you recover the parameters of the utility function (or other pref), you can use those parameters to simulate what will happen if policy changes. • Disadvantages: • have to implement possibly untestable assumptions about economic and statistical model • often generate wide range of estimates • Can be very sensitive to specification and assumptions—not great external validity

  30. Conclusion • Natural experiments useful source of knowledge • Often requires use of IV • Instrument exogeneity and relevance need justification • Weak instruments potentially serious • Good practice to present first-stage regression • Finding more robust alternative to IV an active research area

More Related