Analysis of Variance

1. Analysis of Variance Wrap-Up Notes 46-511 Fall, 2007

2. Learning Objectives • Understand a couple of useful extensions of ANOVA • Repeated Measures Designs • Intraclass Correlation • Latin Squares Design • Random Effects Designs • Hierarchical Designs

3. Intraclass Correlation Coefficient (ICC) • Based on within-subjects design • Columns = Judges / Raters • Rows = Stimulus • Coefficient reveals degree of rater agreement or reliability Three experts watch job candidates in an assessment center and provide ratings of performance:

4. Use RM ANOVA • Assume experts are random sample

5. RM ANOVA & Nuissance Effects • Carryover effects • Fatigue/boredom effects • Methods of addressing • Averaging Out • Randomization/Counterbalancing • Partitioning • Latin Squares/Counterbalancing

6. Randomization • Advantages • Disadvantages

7. Latin Squares Designs • Objective: to partition out variance due to nuissance factor(s) • Example: we want to know the effects of four computer screen layouts on a vigilance task • Thus, our design is…

8. Latin Squares Design (cont’d) If we use (Latin) letters to denote the different treatments: The Latin Squares Solution becomes:

9. Leads to Two Effects

10. Main Effects from Design

11. Between Subjects Latin Squares Designs • Objective: to accommodate multiple between subjects factors while reducing sample size needs / balanced fractional replication. • Example: we want to know the effects of… • A: Display-color (4 colors), • B: Display-size (4 sizes), and • C: Layout of information (4 levels) on computer operators. • On accuracy in a vigilance task • A 3-way factorial design requires 4 x 4 x 4 = 64 conditions. • Latin squares design requires 16 conditions

12. The Latin Squares Solution • A & C main effects derived in the usual way • B main effects derived off diagonal (B1, etc.).

13. Issues with Latin Squares designs • Strict Assumption • No interaction effects • Calculating F • By hand, fairly straight forward • In SPSS… • Planning these designs quickly becomes complicated • Algorithms for generating cell arrangements • Extensive tables (Fisher & Yates, 1953; Cochran & Cox, 1957)

14. Random Effects Model • What are random effects? • One-Way • N-Way • Repeated Measures • Characteristics • Power • Interpretation • Calculation

15. 1-Way Example • A computer screen must contain 20 different pieces of information • Does the placement of the 20 items make a difference in perception & processing by operators? • There are 20! Permutations (2.43 x 1018) • What to do?

16. Random Effects Solution • Randomly select a subset of categories to see if there is a main effect • Generalize findings to entire “population” of layouts Five layouts are chosen randomly (Factor/IV), number of operator errors recorded (DV)

17. Results • For one-way designs • No difference in ANOVA calculations • Main difference = ability to calculate variance components… Variance Component =

18. ICC as an Effect Size Measure

19. Two-Way Random Effects Model… • For example… • Assume our 2-way example (anxiety by task difficulty) was a random effects model • That we had chosen our levels of difficulty & anxiety at random • Why would we have done this?

20. Results • Error Term • Drawbacks of this design • Caveat on SPSS

21. Hierarchical/Nested Models • Sometimes interested in the effects of more than one factor, but are unable to fully cross the factors. • May be due to… • Experimental content – different levels of B need to be associated with different levels of A • Constraints in how data are collected (e.g., organizational structure) • Can occur with between, within, or mixed designs

22. Example 1: Experimental Manipulation • Inducing False Memories • Inducing mundane false memories (e.g., getting lost in a grocery store) • More extraordinary memories (house fire when a child)

23. Characteristics of this design • B is nested under A • B is treated as a random factor • No interaction effects • Other possibilities • Completely between • Completely within • Mixed (A between, B within)

24. Example 2: Structural Constraints • Wish to test two different types of employee interventions to reduce turnover • Type of intervention (Factor A): • Level 1: intervention involving greater employee involvement in decision making • Level 2: intervention involving different array of benefits, compensation & training • Location (Factor B): • Each intervention must be implemented at entire work locations, thus 10 work locations are selected for A1 and 10 different ones for A2.

25. Example 3: Example 1 made more complicated • Inducing False Memories • Inducing mundane false memories (e.g., getting lost in a grocery store) • More extraordinary memories (house fire when a child) • Wish to cross with gender

26. Issues with Nested Designs • Cannot obtain information about interactions • Most software (e.g., SPSS, SAS) will allow for analysis, but not straight forward or easy • Other, newer regression based methods for hierarchical designs available, and may be more appropriate

27. Final Thoughts • Experimental Methods / ANOVA can be very flexible • Strike a balance between complexity and elegance • Important sources on ANOVA; • Howell, D. C. (2002). Statistical methods for psychology (5th ed.). Pacific Grove, CA: Duxbury.* • Keppel, G. & Wickens, T. D. (2004). Design and analysis: A researcher’s handbook (4th ed.). Englewood Cliffs, N. J.: Prentice-Hall • Kirk, R. E. (1995). Experimental design: Procedures for the behavioral sciences (3rd ed.). Monterey, CA: Brooks/Cole • Tabachnick, B. & Fidell, (2001). Computer-assisted research design and analysis. Pearson/Allyn & Bacon.* • Winer, B. J., Brown, D. R., & Michels, K. M. (1991). Statistical principles in experimental design, 3rd ed. New York: McGraw-Hill * Newer editions available