Experimental Design & Analysis

1. Experimental Design & Analysis Hypothesis Testing and Analysis of Variance January 30, 2007 DOCTORAL SEMINAR, SPRING SEMESTER 2007

2. Outline • Statistical inferences • Null hypothesis testing • Sources of variance • Treatment variance vs. error variance • Sums of squares

3. Means The mean is a measure of central tendency μ or x Sum of squares The sum of squares refers to the sum of the squared deviations of a set of scores or values from their mean (1) Subtract the mean from each score (2) Square each resulting difference (3) Add up the squared differences Mean square, or variance Mean square of a sample is a measure of variability Divide sum of squares by the degrees of freedom σ2 or s2 Standard deviation σ or s Statistical Inference

4. Statistical Inference • Student's t-test is used to compare two groups when sample sizes are small (N < 30), but with larger samples the normal curve z test is used • Let E be the experimental condition and let C be the control condition. Let m be the means, s the standard deviations, and n be the sample size t = (mE - mC) / √[(s2E + s2C) / n ] • The critical value is the value found in t-tables

5. Statistical Inference • Hypothesis testing • Inferences about the population based on parameters estimated from a sample • μ1 = μ2 = μ3 = etc. suggests that there are no treatment effects • Observing an effect requires falsifying the null hypothesis H0 • How different are two mean scores?

6. Differences due to treatment Systematic source of difference by virtue of experimental condition Differences due to error Random source of difference because participants are randomly assigned to groups Statistical Inference

7. treatment effects + experimental error experimental error = 1 ? = 1 experimental error experimental error Statistical Inferences • Evaluation of null hypothesis differences between experimental groups of subjects differences among subjects within same groups

8. Null Hypothesis Significance Testing • 5 steps of NHST • State H0, H1, and  level • Determine rejection region and state rejection rule • Compute test statistic • Make a decision (reject H0 or fail to reject H0) • Conclude in terms of the problem scenario

9. Statistical Inference • When H0 is true, F value = 1, although random variation can result in it being > 1 or < 1 • When H0 is false, F expected >1 • Since random variation can account for why F > 1, the problem we face is determining how much >1 it must be in order for us to conclude, on the basis of improbability, that its size indicates a real difference in the treatments • The F distribution enables us to determine how improbable an experimental outcome is under the assumption our null hypothesis is true. If the value for F is so large that it is improbable (p < .05 or p < .01), given that our null hypothesis is true, we will conclude our null hypothesis is false and that the hypothetical population means are not equal

10. Sources of Variance • Partitioning variance • In order to distinguish systematic variability (treatment effects) from random error, we must partition variance by calculating component deviation scores • Total deviation • Between-groups deviation • Within-groups deviation

11. Y5,2 Y2 YT 2 4 6 8 10 0 Y5,2 - Y2 Y2 - YT Between-group deviation Within-group deviation Y5,2 - Y2 Total deviation Component Deviations From Keppel & Wickens, p. 23

12. Sources of Variance • What is the grand mean, YT? • What are the group means? Treatment A2 Treatment A3 Treatment A1 Participant 1 16 Participant 2 18 Participant 3 10 Participant 4 12 Participant 5 19 Participant 1 2 Participant 2 10 Participant 3 9 Participant 4 13 Participant 5 11 Participant 1 4 Participant 2 7 Participant 3 8 Participant 4 10 Participant 5 1 A2 = ? A3 = ? T = ? A1 = ? Y2 = ? Y3 = ? YT = ? Y1 = ? From Keppel & Wickens, p. 24

13. Calculating Sums of Squares • Identify bracket terms* • Using sums [Y] = ΣYij2 = 162 + 182 + … + 132 + 112 = 1,890 [A] = ΣAj2 /n = (752 + 302 + 452)/5 = 1,710 [T] = ΣT2/an = 1502/15 = 22,500/15 = 1,500 • Using means [Y] = ΣYij2 = 162 + 182 + … + 132 + 112 = 1,890 [A] = nΣYj2 = (5)(152 + 62 + 92) = 1,710 [T] = anΣYT2 = (3)(5)102 = 15 (100) = 1,500 See authors’ note on bracket terms, Keppel and Wickens, p. 31.

14. Sources of Variance • Calculate sums of squares • Total sums of squares = Σ(Yij – YT)2 • Between sums of squares = Σ(Yj – YT)2 • Within sums of squares =Σ(Yij – Yj)2 Sums of SquaresTotal = Sums of SquaresBetween + Sums of SquaresWithin

15. Calculating Sums of Squares • Write sums of squares using bracket terms • SST = [Y] – [T] = 1,890 – 1,500 = 390 • SSA = [A] – [T] = 1,710 – 1,500 = 210 • SSS/A = [Y] – [A] = 1,890 – 1,710 = 180

16. Analysis of Variance • Sums of squares provide the building blocks for analysis of variance and significance testing • Analysis of variance involves 2 steps • Calculate variance estimates, known as mean squares, by dividing component sums of squares by degrees of freedom • Ratio of between-group mean square and within-group mean square provides F statistic

17. Analysis of Variance SourcedfMean SquareF A a-1 SSA/dfA MSA/ MSS/A S/A a (n-1) SSS/A/dfS/A Total an-1 a = # of levels of factor A n = sample size of a group df = degrees of freedom Check F value in table of critical values organized by degrees of freedom to determine significance level Numerator degrees of freedom = a-1 Denominator degrees of freedom = a(n-1)