Experimental Design & Analysis

1. Experimental Design & Analysis Random and Fixed Factors; Fractional Factorials April 10, 2007 DOCTORAL SEMINAR, SPRING SEMESTER 2007

2. Outline • Fixed and random factors • Fixed factor model • Random factor model • Mixed factor model • Examples • SAS • Fractional factorial designs

3. Fixed Factors • Most ANOVA designs are fixed effects models, meaning that data are collected on all relevant categories of the independent variables • Relevance of levels are dictated by some theoretical underpinning • E.g. comparing dosages of a drug (5 mg, 10 mg, 20 mg) are of theoretical interest, perhaps representing low, medium, high levels • Comparing sexes with respect to some response variable, the factor sex is fully represented with 2 levels

4. Random Factors • In random effects models, in contrast, data are collected only for a sample of categories • For instance, a researcher may study the effect of item order in a questionnaire. Six items could be ordered 720 ways. However, the researcher may limit herself to the study of a sample of six of these 720 ways • The random effect model in this case would test the null hypothesis that the effects of ordering = 0 • Studies examining whether facial symmetry is important in mate selection may include a variety of symmetric faces • Within-subjects designs include a random effect

5. Fixed and Random Factors • Random effects are factors which meet 2 criteria: • Replaceability: The levels (categories) of the factor (independent variable) are randomly or arbitrarily selected, and could be replaced by other, equally acceptable levels. • Generalization: The researcher wishes to generalize findings beyond the particular, randomly or arbitrarily selected levels in the study.

6. Mixed Effects Model • Treatment by replication design is a common mixed effects model • A fixed factor, such as male and female faces • A random factor, or replication factor, representing variety of bilateral symmetry, such as distance of facial features

7. Fixed and Random Factors • Effects shown in the ANOVA table for a random factor design are interpreted a bit differently from standard, within-groups designs • Main effect of the fixed treatment variable is the average effect of the treatment across the randomly sampled or arbitrarily selected categories of the random variable • The effect of the fixed by random (e.g. treatment by replication) interaction indicates the variance of the treatment effect across the categories of the random variable • Main effect of the random effect variable (e.g. the replication effect) is of no theoretical interest as its levels are arbitrary cases from a large population of equally acceptable cases

8. Random and Fixed Factors • For one-way ANOVA, computation of F is the same for fixed and random effects, but computation differs when there are two or more independent variables • Resulting ANOVA table still gives similar sums of squares and F-ratios for the main and interaction effects, and is read similarly • Assumptions: normality, homogeneity of variances, and sphericity, but robust to violations of these assumptions

9. Testing for Significance

10. Testing for Significance • What is impact of assuming a random factor is a fixed factor • Type I error? • Type II error?

11. Random and Fixed Factors • Treating a random factor as a fixed factor will inflate Type I error • If a random factor is treated as a fixed factor, research findings may pertain only to the particular arbitrary cases studied and findings and inferences may be different with alternative cases

12. SAS Procedures • For random effects models or mixed effects models PROC GLM DATA = mydata; CLASS factor1 factor2; MODEL depvar = factor1|factor2; RANDOM factor1 / TEST; RUN;

13. Fractional Factorials • Even if the number of factors, k, in a design is small, the 2k runs specified for a full factorial can quickly become very large • E.g. 26 = 64 runs is for a two-level, full factorial design with six factors

14. Fractional Factorials • The solution to this problem is to use only a fraction of the runs specified by the full factorial design • Which runs to make and which to leave out is the subject of interest here • We use various strategies that ensure an appropriate choice of runs • Properly chosen fractional factorial designs for 2-level experiments have the desirable properties of being both balanced and orthogonal • Assume that higher-order interactions are not of interest

15. Fractional Factorials • Imagine testing the effects of attention variables on processing of negation • Duration of exposure (10 sec vs. 20 sec vs. 40 sec) • Cognitive load (7 digit vs. 3 digit) • Self relevance (self vs. other)

16. Fractional Factorials • 3 x 2 x 2 design = 12 conditions! • Are all cells and interactions of comparable value? • Consider running fractional factorial • Yoke cells

17. Fractional Factorials • 4 conditions!

18. Fractional Factorials • Taguchi methods