CEP 933: Planned and Post hoc Contrasts

1. CEP 933: Planned and Post hoc Contrasts To make more explicit statements about the means that we have analyzed in ANOVA we must use contrasts, also known as planned and post hoc comparisons of means. From the names you can see that the difference here is whether you know what means you want to compare ahead of time (i.e., you have planned them), or whether you “go hunting” for mean differences after you’ve found a significant F test in ANOVA (then you are doing post hoc tests).

2. CEP 933: Planned and Post hoc Contrasts We’ll see below that planned (a priori) tests are more powerful than post hoc tests, because post hoc tests penalize you for just hunting around to figure out which means differ. In statistics it is always better to know what to expect than to just look around for something potentially interesting.

3. CEP 933: Planned and Post hoc Contrasts One other point is important here. If you have planned some comparisons of means ahead of time because you expect specific means to differ, then you do not really need to do the F test to see if you can reject the omnibus null hypothesis that all means are equal. Our book says that we don’t even really need to do the F test for post hoc tests, but that is an unusual opinion.

4. CEP 933: Planned and Post hoc Contrasts Planned comparisons are more specific than the omnibus test, so they can be done whether or not the overall test is significant. Also usually they are more targeted – we may only make a few of all possible comparisons when doing planned comparisons. Often researchers who have planned some comparisons do the omnibus test anyway, but it is not necessary.

5. CEP 933: Planned and Post hoc Contrasts When we use contrasts, the tests we use are built to protect us from having overly large chances of making a Type I error (i.e., we are protected from saying H0 is false when it is actually true). To do this the tests consider Type I error rates in two different ways, by examining the rate “per comparison” (PC or “per-contrast”) or the so-called “familywise” (FW) rate, which pertains to a set of comparisons.

6. CEP 933: Planned and Post hoc Contrasts If we are making several comparisons, the familywise rate may apply. The FW rate tells us what the chance is of making “at least one Type I error” in the set of comparisons. The comparison procedures differ because some of them limit the per-contrast error rate and the others limit the familywise rate. Suppose  is the error rate of one comparison. Then Per-contrast (PC) error rate =  Familywise (FW) error rate = 1 - (1 - )c for c independent comparisons

7. CEP 933: Planned and Post hoc Contrasts Suppose  =.05 is the error rate of one comparison. Also let’s say we are making 3 comparisons. Here are the values: Per-contrast (PC) error rate  = .05 Familywise (FW) error rate 1 - (1 - )c = 1- (.95)3 =1-.857 = .142 If we just add the rates of the three comparisons we get: c  = 3  = 3 * .05 = .15 In general, PC < FW < c  ; FW is usually close to c 

8. CEP 933: Planned and Post hoc Contrasts Also here there is one additional concern – when we look at the data before doing comparisons we are increasing our chances of making a Type I error, because we may decide to only test the differences among means that look big. This is why most post-hoc comparisons examine all possible comparisons, and also why post-hoc tests are not as powerful as planned tests.

9. CEP 933:What’s a contrast anyway? A contrast is just a linear combination of means. Usually such a combination takes the form of a difference between two means, or a difference between averages of two sets of means. For instance, L1 = - is a contrast. Also another contrast is L2 = ( + )/2 -

10. CEP 933: What’s a contrast? The example L1 = - is a pairwise comparison. Here we have taken the difference between two means. Some of the tests we’ll encounter examine ALL pairwise comparisons (i.e., they look at all k(k-1)/2 differences for all possible pairs in a set of k means).

11. CEP 933: What’s a contrast? The contrast L2 = ( + )/2 - is a more complex contrast. For the example L2, we have averaged the means for groups 1 and 2 and we compare them to the mean for group 3. This is the same as writing 1/2 + 1/2 + (-1)

12. CEP 933: What’s a contrast? All comparisons are tests of specific hypotheses. For instance if we are comparing all pairs of means we are testing H0:mj = mj for j and j = 1, 2, … k where j j Or suppose we are testing L2 above. We will be testing H0: (m1 + m2 )/2 = m3 or H0: (m1 + m2 )/2 - m3 = 0

13. CEP 933: Contrasts The contrast in the population is often named using the Greek letter psi Y = S cj mj So for L1 above we have Y1 = S cj mj = (m1 - m2 ) and for L2 we have Y2 = S cj mj = (m1 + m2 )/2 - m3

14. CEP 933: Contrasts Our book calls the estimate of the contrast L, for linear combination, and L = S cj . (The book also uses aj for the coefficients, but cj is more common notation.) The values of the "coefficients" cj are determined by your ideas about the subsets of means you want to compare. Above the cj values were 1/2, 1/2 and -1. The examples below should help to clarify the nature of the coefficients. Later we will learn how to use contrasts, but for now we’ll see a list of the many possible comparison procedures that use contrasts.

15. CEP 933: List of comparison tests TEST USE ERROR, POWER ----------------------------------------------------------------------------- POC k-1 planned per-contrast a Planned independent (Most folks use a’ as orthogonal contrasts rate for each test, but we comparisons can use the Bonferroni approach to reduce error) most powerful contrast tests available Trend k-1 independent same as POC trend tests

16. CEP 933: List of comparison tests TEST USE ERROR, POWER ----------------------------------------------------------------------------- Dunn or any # (c) of planned familywise a Bonferroni contrasts (use per-contrast level of (use if contrasts a/c if you desire are not orthogonal) familywise rate of a = a) Dunnett* paired contrasts of 1 familywise a mean with (k-1) other means (e.g., one control vs other treatments)

17. CEP 933: List of comparison tests TEST USE ERROR, POWER ---------------------------------------------------------------------------- Fisher’s LSD* post-hoc, familywise a all pairs of means Tukey’s HSD post-hoc, familywise a all pairs of means same CV is used for all pairs Newman-Keuls post-hoc, mystery a all pairs of means (fw rate = ka/2 CV varies as means for even k) get closer power is higher than Tukey test

18. CEP 933: List of comparison tests TEST USE ERROR, POWER ----------------------------------------------------------------------------- Ryan* post-hoc, controls rate using all pairs of means different levels for each pair of means Scheffe any # of post hoc familywise a contrasts based on large set of contrasts, low power

19. CEP 933: Rules for coefficients There are not too many “rules” for figuring out what the coefficients in a contrast should be. The coefficients are the numbers we multiply times our means. Here are a few “rules”: 1. The coefficients MUST sum to zero within a contrast. 2. The coefficients should make sense – they often produce averages that are subtracted from each other. 3. It’s better (easier) to use integers (whole numbers).

20. CEP 933: Rules for coefficients 1. The coefficients MUST sum to zero within a contrast. Consider our contrast above: L2= ( + )/2 – We said this is the same as The s show L2= ½ * + ½ * –1 * the coefficients. The sum of the coefficients is ½ + ½ + -1 = 0

21. CEP 933: Rules for coefficients 2. The coefficients should make sense – they often produce averages that are subtracted from each other. Suppose we have 5 means. If we want to compare 3 means to 2 others we can compute This is like computing two new means: and Then we compute

22. CEP 933: Rules for coefficients 3.It’s better (easier) to use integers (whole numbers). This is true because eventually we need to get standard errors for the contrasts, and the SEs involve squaring the coefficients. So for the contrast we’ll need to square all the fractions in this:

23. CEP 933: Rules for coefficients 3.It’s better (easier) to use integers (whole numbers). Squaring all the fractions produces This will be a mess when we try to compute the SEs!! So we can use integer coefficients instead.

24. CEP 933: Rules for coefficients 3. It’s better (easier) to use integers (whole numbers). One way to find the integer coefficients is to get the least common denominator (LCD) for the fractions. The LCD is 6 for 1/3 and ½ so we can multiply all the coefficients by 6 and use Also as our book points out, this is the same as using as a coefficient the number of means in the OTHER set.

25. CEP 933: Rules for coefficients There are also some special kinds of coefficients for special contrasts. In particular we will learn about “trend tests” – for those we use weights that represent linear, quadratic, cubic trends, etc. The coefficients are made up especially for testing trends and are on page 742 in our book. For instance for k=5 means (for a 5-level quantitative factor), the linear trend contrast is

26. CEP 933: Data for examples Suppose we want to compare five modes of presentation of course materials, with an outcome representing student learning of the material. We could consider the “lecture” group to be a baseline or control group – all the others seem to use video or technology-rich modes. Group n Id (j) Mean Interactive video 10 1 48.7 CAI 10 2 43.4 Standard video 10 3 47.2 Slide tape 10 4 36.7 Lecture 10 5 40.3

27. CEP 933: Data for examples Source df SS MS F Between 4 969.32 242.33 8.41 Within 45 1296 28.8 Total 49 2265.32 Note that because Fc(4,45) = 3.83, the overall (omnibus) F test is significant, indicating that we can proceed with either planned or post hoc tests. Also E2 = 969.32/2265.32 = .428 or about 43% of score variance is explained by mode of presentation.

28. CEP 933: Planned orthogonal comparisons Planned orthogonal comparisons (POC) are contrasts of a certain type. For any one-way anova with k groups, there are k-1 POCs. POCs are simply contrasts that are “orthogonal” or independent of one another – and we can determine this by looking at each pair of contrasts and seeing if they are independent. Each set of k-1 POCs provides tests of all of the unique information in the k means. Also, there may be more than one set of POCs for any set of k means. To tell whether two contrasts are orthogonal, we multiply together the weights (the cj values) from the contrasts.

29. CEP 933: Planned orthogonal comparisons Consider the following two possible contrasts for the means in the data above. A: Interactive versus standard video [1 vs. 3]: LA = - B: Video (I or S) versus all others [1 and 3 vs. rest]: LB = ( + )/2 - ( + + )/3

30. CEP 933: Planned orthogonal comparisons The following weights are being applied to the 5 group means Group 1 2 3 4 5 Contrast A 1 0 -1 0 0 weights Contrast B .5 -.33 .5 -.33 -.33 weights Note first that the weights within each contrast must sum to zero. Then we check for orthogonality.

31. CEP 933: Planned orthogonal comparisons We compute the products of the pairs of weights for each group: Group 1 2 3 4 5 Products 1* .5 0* -.33 -1*.5 0* -.33 0* -.33 of weights .5 0 -.5 0 0 If we add up the products and they sum to zero, the two contrasts are orthogonal or independent. So here contrasts A and B are orthogonal ( .5 + 0 + -.5 + 0 + 0 = 0). There are always k-1 independent contrasts, and there may be several different sets of independent contrasts for any set of k means.

32. CEP 933: Planned orthogonal comparisons Let’s look at another set of weights. We’ll compare the first four trts to the last (standard lecture) – this will be contrast C – and we will keep contrast B Group 1 2 3 4 5 Contrast C 1 1 1 1 -4 weights Contrast B .5 -.33 .5 -.33 -.33 weights The weights within each contrast do still sum to zero. Then we check for orthogonality.

33. CEP 933: Planned orthogonal comparisons We compute the products of the pairs of weights for each group: Group 1 2 3 4 5 Products 1* .5 1* -.33 1*.5 1* -.33 -4* -.33 of weights .5 -.33 .5 -.33 1.33 We add up the products and here contrasts A and B are NOT orthogonal (.5 + -.33 + .5 + -.33 + 1.33 = 1.67). So B and C are not orthogonal contrasts.

34. CEP 933: Planned orthogonal comparisons How about contrasts A and C? Group 1 2 3 4 5 Contrast A 1 0 -1 0 0 weights Contrast C 1 1 1 1 -4 weights Products 1* 1 0*1 -1*1 0*1 0*-4 of weights 1 0 -1 0 0 Are A and C orthogonal?

35. CEP 933: Planned orthogonal comparisons Finally note that although trend tests are always orthogonal, trend tests don’t make sense for this example. The treatment factor has groups that are qualitatively different – the levels represent different kinds of instruction. So we would not use trend tests on these data even though we could compute them. If the factor were duration of instruction (e.g., 10 minutes, 20 minutes, 30 minutes, etc. through 50 minutes) we could do trend tests.