Download Presentation
## Quasi-Experimental Designs

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**Quasi-Experimental Designs**Manipulated Treatment VariablebutGroups Not Equated**Pretest-Posttest Nonequivalent Groups Design**N O X O N O O • Cannot assume that the populations are equivalent prior to treatment. • Selection and Selection Interactions are threats to internal validity. • Can try to select subjects or intact groups in ways that make it likely that the groups are similar, but what about unknown variables on which the groups may differ.**Double-Pretest Nonequivalent Groups Design**N O O X O N O O O • Some control for Selection x Maturation. • If groups are maturing at different rates, that may be shown in the two pretests.**Regression-Discontinuity Design**C O X O C O O • ‘C’ indicates subjects are assigned to groups based on score on covariate. • Groups are deliberately nonequivalent. • I shall illustrate with a hypothetical example**Evaluating an Online Tutorial**• IV = Student completes online tutorial or not. • DV = Student’s score on statistics course exam. • Pretest/Covariate = Student’s score on a test of statistics aptitude. • How do I assign students to groups?**Evaluating an Online Tutorial**Let the students self-select into groups. • This gives me a pretest-posttest nonequivalent groups design. • I use pretest scores as covariate in ANCOV. • This does not, however, remove all possible confounds. • How might the groups have differed other than on statistics aptitude???**Evaluating an Online Tutorial**Randomly assign students to groups. • This would be an experimentally sound, randomized pretest-posttest control group design. • And you would live to regret trying it. • Students complain. • Their parents complain. • The Chair of the Department intervenes. • The IRB revokes your authority to do research.**Evaluating an Online Tutorial**Try a switching-replications design. • Those who have to wait until the second half of the class would be disadvantaged • if you don’t learn the beginning material well, the later material will very hard to learn. • Those who have it taken away at mid-semester will complain.**Evaluating an Online Tutorial**Apply the treatment only to those most in need of it, those lowest in statistics aptitude. • Those not selected may complain that they could benefit from it too. • Tough, there is an American tradition of favoring the underdog.**Evaluating an Online Tutorial**• Those selected may complain about having to do extra work. • You can’t please everybody every time. Convince them they need to do it. • May be cases when you want to give the tutorial only to those with highest aptitude • purpose of tutorial is to allow brightest students to finish course early, allowing prof more time to spend in class with others.**Evaluating an Online Tutorial**• Suppose I pick a cutoff on the covariate so that ½ get the treatment, ½ don’t. • Simulated data are in file RegD0.txt . • C = control group, T = Treatment group. • 2nd score is posttest score. • 3rd score is pretest score. • I defined the treatment effect to be zero in the population.**Evaluating an Online Tutorial**• In the population, Post = 7 + 1.35Pre + error, = .9, • In the sample, ignoring group, Post = 7.58 + 1.27Pre + error, r = .85, and MSE = 2.13 • Look at this plot of the data:**Evaluating an Online Tutorial**• Now I compute two separate regressions, one for each group. • T: Post = 8.09 + 1.17Pre + error, r = .62, and MSE = 2.13. • C: Post = 6.33 + 1.43Pre + error, r = .72, and MSE = 2.29. • The plot shows how little the two lines differ:**Evaluating an Online Tutorial**• I re-simulated the data, with a 3 point treatment effect built in. • The data are at RegD1.txt. • T: Post = 11.27 + 1.07Pre + error, r = .82, and MSE = 1.35. • C: Post = 7.90 + 1.18Pre + error, r = .82, and MSE = 1.25 • The plot shows a clear regression discontinuity:**Evaluating an Online Tutorial**• The dotted line shows the expected regression for the treatment group if there were no treatment effect. • Hard to imagine how any threat to internal validity would create the observed regression discontinuity. • Caution: This analysis assumes the regression is linear, not curvilinear.**Proxy-Pretest Design**N O1 X O2 N O1 O2 • You have a nonequivalent groups posttest only control group design. • The treatment has already been administered. • Now you decide you want a pretest too. • Can’t warp time, can find an archival proxy pretest.**PSYC 2210 and Understanding Stats**• Mid-semester, I ask myself “does taking 2210 improve students understanding of stats?” • I’ll compare students in current 2210 class with those in another class (excluding any who have already taken 2210). • I want a pretest too, but the treatment is already in progress.**PSYC 2210 and Understanding Stats**• I use, as a proxy pretest, students’ final averages from PSYC 2101. • Conduct an ANCOV • IV = took 2210 or not • DV = end of course stats achievement test • COV = the proxy pretest**Pretest subjects different than posttest subjects.**I want to evaluate online tutorial in stats. Both I and my friend Linda taught stats this semester and last semester. Both of us gave our students a end of course standardized exam. N O N X O N O N O Separate Pre-Post Samples Design**Row 1: My students last semester, no tutorial.**Row 2: My students this semester, with tutorial. Row 3: Linda;s students last semester, no tutorial Row 4: Linda’s students last semester, no tutorial. Selection problems likely. N O N X O N O N O Separate Pre-Post Samples Design**Nonequivalent Groups Switching Replications Design**N O X O O N O O X O • I am teaching two sections of stats. • I make the experimental tutorial available the first half semester to one class • and the second half semester to the other. • Might reduce complaints, until students from the two classes meet each other.**Nonequivalent Dependent Variables Design**• Only one group of subject, but two DVs. • One DV you expect to be affected by X. • The other you expect not to be affected by X. • The second DV serves as a control variable. • Should be similar enough to 1st DV that it will be effected in same way by history, maturation, etc.**Nonequivalent Dependent Variables Design**• I want to evaluate effect of stats remedial tutorial given to all PSYC 2210 students. • DV1 = stats given at start and end of semester. • DV2 = Vocabulary test given at start and end of semester. • More impressive if have multiple control variables and an a priori prediction of extent to which each will be effected.**Nonequivalent Dependent Variables Design**• Stats Knowledge (DV1) – most affected • Logical Thinking – next most • Verbal Reasoning – same as LT • Arithmetic Skills – next most • Vocabulary – next most • Artistic Expression – least affected by treatment**Regression Point Displacement Design**N(n = 1) O X O N O O • Only one subject in the treatment group • Several or many in the control group. • X = novel economic development plan. • Treatment unit = your hometown, in which the plan was just initiated.**Regression Point Displacement Design**• You consult state economic database. • Pick 20 cities comparable to your city, these serve as the control group. • Pretest = value of criterion variable (such as unemployment rate) last year. • Posttest = value of same variable two years later.**Regression Point Displacement Design**• Plot Post x Pre for the Control Group. • Draw in regression for predicting Post from Pre. • Plot experimental unit data point. • If it is displaced well away from regression line, you have evidence of a treatment effect.