1 / 21

EVAL 6970: Experimental and Quasi-Experimental Designs

EVAL 6970: Experimental and Quasi-Experimental Designs. Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012. Agenda. Regression discontinuity designs. Questions to Consider. Throughout today’s discussion, consider the following questions

kyle-dennis
Download Presentation

EVAL 6970: Experimental and Quasi-Experimental Designs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. EVAL 6970:Experimental and Quasi-Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012

  2. Agenda Regression discontinuity designs

  3. Questions to Consider • Throughout today’s discussion, consider the following questions • What characteristics of regression discontinuity designs make them more amenable to causal interpretation versus those discussed previously (i.e., quasi-experimental designs with and without pretests and control groups and interrupted time-series)? • What are the associated limitations or drawbacks of regression discontinuity designs, if any? • How could you apply regression discontinuity designs to your own work?

  4. Note • The principles of regression discontinuity can be confusing, so… • …ASK QUESTIONS!!!

  5. Basic Structure of Regression Discontinuity Designs

  6. Basic Structure

  7. Theory of Regression Discontinuity

  8. Some Assumptions Most randomized experiments compare posttest means Regression discontinuity designs compare regression lines for treatment and control groups Rather than making the assumption that pretest means are equivalent for both groups, regression discontinuity looks for a change between the functional form of the regression line (e.g. slope or intercept) for the treatment group and control

  9. Some Assumptions In (most) other designs for quasi-experiments, the selection process is never fully known (selection is determined by a large system of variables beyond the researcher’s control) In sharp contrast, there are no unknowns in the assignment process for regression discontinuity In fact, the selection process is completely known and perfectly measured (even though there will be some error associated with the assignment variable) Assignment variables only measure how participants got into conditions, and when assignment is based only on that score, error is effectively zero

  10. Some Assumptions • Additionally, most experiments and quasi-experiments try to equate treatment and control • Regression discontinuity designs explicitly acknowledge—and, in fact, base assignment on—pre-existing differences between treatment and control groups • Units are assigned to conditions based on a cutoff score (i.e., cut score) on an assignment variable • The assignment variable must occur prior to treatment • Units on one side of the cut score are assigned to one condition, and units on the other side to another condition

  11. Implementing Regression Discontinuity Designs

  12. Implementation • Like randomized experiments, regression discontinuity designs yield unbiased estimates of treatment effects (as long as assumptions are met) • Assignment to treatment must be based only on the cutoff score • The assignment variable cannot be caused by the treatment, and must be continuous • Dichotomous variables should not be used as an assignment variable, because this makes it impossible to estimate a regression function • The assignment variable is often a pretest score, but it can be almost anything

  13. Implementation • The cut score should be near the mean of the assignment variable, because extreme values can cause problems • Modeling the regression function can become more difficult and/or error-prone in smaller samples • Statistical power depends on sample size, and also “prefers” samples with equal or nearly equal sizes • Researchers can use a composite variable for assignment, to include the effect of multiple influences • Avoiding selection bias requires that • Assignment to conditions is strictly controlled • All units could have received treatment

  14. Forms of Effects

  15. Types of Effects • First, note that “discontinuity” means • A treatment effect (if present) causes an upward or downward displacement in the regression function • A discontinuity can be a change in either the intercept or the slope • The discontinuity should occur at exactly the cut score

  16. Threats to Validity

  17. Validity Threats • Few internal validity threats are plausible with a well planned and correctly executed regression discontinuity design • A plausible threat would have to cause a discontinuity in the regression line that corresponds precisely with the cut score • Except in rare cases, selection, history, and maturation threats are drastically reduced • However, attrition can be a major concern, especially if attrition rates are correlated with the assignment variable • Therefore, validity concerns mainly focus on statistical conclusion validity • In particular, the regression lines must be good models of functional form (e.g., nonlinear functions, interaction terms)

More Related