Quasi-Experimental Design

1. Quasi-Experimental Design Jung Eun (Jessie) Hong Feb. 23, 2009

2. Outlines • Experimental Design • Definition • Process • A Key Point • Types of Experimental Designs • Quasi-Experimental Design • Designs • Strengths • Weaknesses • Examples in Geography • Ongoing Debate

3. Definition of Experimental Design • A structured, organized method • To determine whether some program or treatment causes some outcome or outcomes to occur. • If X, then Y • Because there may be lots of reasons, other than the program, for why you observed the outcome, • If not X, then not Y needs to be addressed, too

4. Process of Experimental Design • To show that there is a casual relationship, • Two “equivalent” groups • The program or treatment group gets the program • The comparison or control group does not • The groups are treated the same in all other respects • Differences in outcomes between two groups must be due to “the program”

5. A Key Point of Experimental Design • How do we create two groups that are “equivalent”? • Assign people randomly from a common pool of people into the two groups • The experiment relies on the idea of “random assignment” to obtain two similar groups. • A key to the success of the experiment • Assume that two groups are “probabilistically equivalent”

6. Types of Designs Is random assignment used? No Yes Randomized or True experiment Is there a control group or multiple measures? No Yes Non-experiment Quasi-experiment

7. Quasi-Experimental Design • Similar to the experimental design, but lacks the key ingredient, “random assignment” • Easily and more frequently implemented • Extensively used in the social sciences • A useful method for measuring social variables • Two classic quasi-experimental designs • The Nonequivalent Groups Design • The Regression-Discontinuity Design

8. The Nonequivalent Groups Design • The most frequently used in social research • Try to select groups that are as similar as possible to compare the treated one with the comparison one • e.g. two comparable classrooms or schools • Cannot be sure whether the groups are comparable • The groups may be different prior to the study • Any prior differences between the groups may affect the outcome of the study • Require a pretest and posttest

10. The Regression-Discontinuity Design • A useful method for determining whether a program of treatment is effective • Participants are assigned to program or comparison groups based on a cutoff score on a pretest • e.g. Evaluating new learning method to children who obtained low scores at the previous test. • Cutoff score = 50 • The treatment group: children who obtained 0 to 50 • The comparison group: children who obtained 51 to 100 • The program (treatment) can be given to those most in need

11. The Regression-Discontinuity Design With no treatment effect With Ten point treatment effect

12. The Regression-Discontinuity Design Discontinuity

13. Strengths of Quasi-Experimental Design • Useful in generating results for general trends in social sciences • Difficult pre-selection and randomization of groups • Easily integrated with individual case studies • Generated results can reinforce the findings in a case study • Allow statistical analysis to take place • Enable to reduce the time and resources required for experimentation • Not required extensive pre-screening and randomization

14. Weaknesses of Quasi-Experimental Design • Without proper randomization, statistical tests can be meaningless • Do not explain any pre-existing factors and influences outside of the experiment • The researcher needs to control additional factors that may have affected the results • Some form of pre-testing or random selection may be necessary to explain statistical results thoroughly

15. Quasi-experiments vs. Non-experiments to address similar questions • Both designs are applicable when the subjects are not able to be randomized • Some variables cannot ethically be randomized • e.g. Studying the effect of maternal alcohol use when the mother is pregnant

16. Example of Quasi-Experimental Design in Geography • Baker and White (2003) • The Effects of GIS on Students’ Attitudes, Self-efficacy, and Achievement in Middle School Science Classrooms • Conducted the Nonequivalent quasi-experimental design • Two eighth grade teachers, across ten classrooms • Total 192 eighth grade students participated • Treatment group: used a Web-based GIS application • Control group: used paper maps

17. Example of Quasi-Experimental Design in Geography • Impossible to randomly assign each student to a GIS or paper mapping conditions • Randomly assigned whole classes to two conditions • Different instructors affected the results differently • Instructor effect played a substantial role in student attitudes and self-efficacy

18. Ongoing Debate • Whether true experiments or quasi-experiments represents the superior design • Supporters of true experiments • Difficult to isolate the program effects using quasi-experiments • Quasi-experimental results are biased and sensitive to minor changes • Not sure about whether quasi-experimental designs can adequately control selection bias • Hard to determine better design • True experiments are impossible and impractical in some cases

19. Any questions???