Experimental Design

Experimental Design Ch. 8 – Let’s not kid ourselves, this is going to hurt

Experimental Design • How on Earth can you ensure that 2 groups of different people are equal (in all respects, not just on the measure of choice) at the beginning of an experiment? • You can’t • But you can make it more probable (and to experimenters, good enough) Remember though, even if you achieve this, groups can still grow different after they have been formed

Experimental Design • Searching for group equivalence • What we do: • Random assignment • Does it work? • Maybe! • Sample size, power &c.

Experimental Design • If random assignment is the solution, and increased internal validity is the benefit, is there a cost? • Undoubtedly • Sample size big enough? • Control of social threats, & mortality • Its unreal, so improved internal validity comes at the cost of external validity

Experimental Design • 2-group experimental designs Two-group, post-test only randomized experimental design

Experimental Design • More on probabilistic equivalence • Random assignment will distribute folk to groups such that their scores on any measure will be distributed randomly (duh)…this means they will probably be different, but that it is statistically improbable that this will be a significant difference

Experimental Design • More on probabilistic equivalence

Experimental Design Random selection  Random assignment External validity control Internal validity control

Experimental Design • Classifying experimental designs • Signal enhancing vs. noise reducing • The signal vs. noise idea: Strong treatment enhances signal Good measurement reduces noise

Experimental Design • Classifying experimental designs • Signal enhancing vs. noise reducing • Designs differ in their strengths ~ • Factorial designs focus on isolating aspects or combinations of treatments that seem to affect the measurement most (signal enhancer) • Covariance/blockingdesigns focus on lessening the effects of known sources of noise (noise reducers)

Experimental Design • Factorial designs • Imagine an educational program… • You are interested in (IV’s) • Time of instruction (1 hour vs. 4 hr) • Setting (in-class or pulled out of class) • You measure via study scores (DV) Note – we are now dealing with 2 independent variables for the first time

Experimental Design: Factorial

Experimental Design: Factorial • A silly example - The marshmallow peeps study • Factor 1: Alcohol (presence/absence) • Factor 2: Smoking (yes/no)

Experimental Design: Factorial • Does alcohol have an effect? • Imbibed liberally • Moderate headache • Nausea • No permanent damage

Does tobacco have an effect? • No marketing to young chicks • Peep grabs a ciggie… • …lights up… • …begins smoking… • …& continues ‘til satiated Experimental Design: Factorial • It can give up any time it wants to…no effect

Experimental Design: Factorial • So, alcohol & nicotine are benign? • Wait..what if you combined them? • Sum of the parts? • More than the sum of the parts?

Experimental Design: Factorial • Is there an interaction? • Combine the elements • Faint flame…blackening • …smell of caramel… • Metamorphosis • “ball of charred goo…” • “less sweet” • “crunchier” • “gross”

Experimental Design: Factorial • Variations – i. 2 x 3

Experimental Design: Factorial • Variations – ii. 2 x 2 x 3 (3 factor)

Experimental Design: Factorial • Variations – iii. 2 x 3 + control

Experimental Design: Blocking • Reducing noise – Randomized block designs • Key point – unexplained variation in a sample reduces power • The solution is to reduce the variation within the sample by splitting the sample up • You split across some factor that you know causes the sample to differ with respect to the measure of interest (making multiple blocks) • You do not include this as a factor in the experiment, because it is not of interest • Each block will have less variability on the measure, and therefore more power

Experimental Design: Blocking • Reducing noise – Randomized block designs Here is the design notation for what was described on the last slide

Experimental Design: Blocking • Reducing noise – Randomized block designs “+’s” show scores for all treatment group members (average of all “+” gives treatment group score – average on x-axis is for pretest, and on y-axis is for posttest “o’s” show scores for all control group members (average of all “o” gives control group score – average on x-axis is for pretest, and on y-axis is for posttest

Experimental Design: Blocking • Reducing noise – Randomized block designs Note that, regardless of the block, the spread of scores on the post-test is less within the block than across the entire measure

Experimental Design: Covariates • Reducing noise – Covariance designs • Design can vary, but basic is this – • Lingo – “controlling for”, “removing the effect of” • Both terms imply use of covariates

Experimental Design: Covariates • Reducing noise – Covariance designs

Experimental Design: Hybrids • Solomon 4 group To examine & control testing effects in pre-post arrangements

Experimental Design: Hybrids • Switched replication design To examine & control social interaction threats

Experimental Design: Hybrids • Reducingsocial interaction threats • (other than via switched replication) • Blind & double blind set ups • Placebos • Isolation of groups

Experimental Design