a priori or planned contrasts n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
MULTIPLE COMPARISON TESTS PowerPoint Presentation
Download Presentation
MULTIPLE COMPARISON TESTS

Loading in 2 Seconds...

play fullscreen
1 / 51

MULTIPLE COMPARISON TESTS - PowerPoint PPT Presentation


  • 111 Views
  • Uploaded on

A PRIORI OR PLANNED CONTRASTS. MULTIPLE COMPARISON TESTS. ANOVA. ANOVA is used to compare means. However, if a difference is detected, and more than two means are being compared, ANOVA cannot tell you where the difference lies.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'MULTIPLE COMPARISON TESTS' - shing


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
anova
ANOVA
  • ANOVA is used to compare means.
  • However, if a difference is detected, and more than two means are being compared, ANOVA cannot tell you where the difference lies.
  • In order to figure out which means differ, you can do a series of tests:
    • Planned or unplanned comparisons of means.
planned or a priori contrasts
PLANNED or A PRIORI CONTRASTS
  • A comparison between means identified as being of utmost interest during the design of a study, prior to data collection.
  • You can only do one or a very small number of planned comparisons, otherwise you risk inflating the Type 1 error rate.
  • You do not need to perform an ANOVA first.
unplanned or a posteriori contrasts
UNPLANNED or A POSTERIORI CONTRASTS
  • A form of “data dredging” or “data snooping”, where you may perform comparisons between all potential pairs of means in order to figure out where the difference(s) lie.
  • No prior justification for comparisons.
  • Increased risk of committing a Type 1 error.
  • The probability of making at least one type 1 error is not greater than α= 0.05.
planned orthogonal and non orthogonal contrasts
PLANNED ORTHOGONAL AND NON-ORTHOGONAL CONTRASTS
  • Planned comparisons may be orthogonal or non-orthogonal.
  • Orthogonal: mutually non-redundant and uncorrelated contrasts (i.e.: independent).
  • Non-Orthogonal: Not independent.
  • For example:
    • 4 means: Y1 ,Y2, Y3, and Y4
    • Orthogonal: Y1- Y2 and Y3- Y4
    • Non-Orthogonal: Y1-Y2 and Y2-Y3
orthogonal contrasts
ORTHOGONAL CONTRASTS
  • Limited number of contrasts can be made, simultaneously.
  • Any set of contrasts may have k-1 number of contrasts.
orthogonal contrasts1
ORTHOGONAL CONTRASTS
  • For Example:
    • k=4 means.
    • Therefore, you can make 3 (i.e.: 4-1) orthogonal contrasts at once.
how do you know if a set of contrasts is orthogonal
HOW DO YOU KNOW IF A SET OF CONTRASTS IS ORTHOGONAL?
  • ∑cijci’j=0
    • where the c’s are the particular coefficients associated with each of the means and the i indicates the particular comparison to which you are referring.
  • Multiply all of the coefficients for each particular mean together across all comparisons.
  • Then add them up!
  • If that sum is equal to zero, then the comparisons that you have in your set may be considered orthogonal.
how do you know if a set of contrasts is orthogonal1
HOW DO YOU KNOW IF A SET OF CONTRASTS IS ORTHOGONAL?
  • For Example:

(After Kirk 1982)

  • (c1)Y1 + (c2)Y2 = Y1 – Y2
  • Therefore c1 = 1 and c2 = -1 because
  • (1)Y1 + (-1)Y2 = Y1-Y2
orthogonal contrasts2
ORTHOGONAL CONTRASTS
  • There are always k-1 non-redundant questions that can be answered.
  • An experimenter may not be interested in asking all of said questions, however.
planned comparisons using a t statistic
PLANNED COMPARISONS USING A t STATISTIC
  • A planned comparison addresses the null hypothesis that all of your comparisons between means will be equal to zero.
    • Ho=Y1-Y2=0
    • Ho= Y3-Y4=0
    • Ho= (Y1+Y2)/2 –(Y3+Y4)/2
  • These types of hypotheses can be tested using a t statistic.
planned comparisons using a t statistic1
PLANNED COMPARISONS USING A t STATISTIC
  • Very similar to a two sample t-test, but the standard error is calculated differently.
  • Specifically, planned comparisons use the pooled sample variance (MSerror)based on all k groups (and the corresponding error degrees of freedom) rather than that based only on the two groups being compared.
  • This step increases precision and power.
planned comparisons using a t statistic2
PLANNED COMPARISONS USING A t STATISTIC
  • Evaluate just like any other t-test.
  • Look up the critical value for t in the same table.
  • If the absolute value of your calculated t statistic exceeds the critical value, the null hypothesis is rejected.
planned comparisons using a t statistic note
PLANNED COMPARISONS USING A t STATISTIC: NOTE
  • All of the t statistic calculations for all of the comparisons in a particular set will use the same MSerror.
  • Thus, the tests themselves are not statistically independent, even though the comparisons that you are making are.
  • However, it has been shown that, if you have a sufficiently large number of degrees of freedom (40+), this shouldn’t matter.

(Norton and Bulgren, as cited by Kirk, 1982)

planned comparisons using an f statistic
PLANNED COMPARISONS USING AN F STATISTIC
  • You can also use an F statistic for these tests, because t2 = F.
  • Different books prefer different methods.
  • The book I liked most used the t statistic, so that’s what I’m going to use throughout.
  • SAS uses F, however.
confidence intervals for orthogonal contrasts
CONFIDENCE INTERVALS FOR ORTHOGONAL CONTRASTS
  • A confidence interval is a way of expressing the precision of an estimate of a parameter.
  • Here, the parameter that we are estimating is the value of the particular contrast that we are making.
  • So, the actual value of the comparison (ψ) should be somewhere between the two extremes of the confidence interval.
confidence intervals for orthogonal contrasts1
CONFIDENCE INTERVALS FOR ORTHOGONAL CONTRASTS
  • The values at the extremes are the 95% confidence limits.
  • With them, you can say that you are 95% confident that the true value of the comparison lies between those two values.
  • If the confidence interval does not include zero, then you can conclude that the null hypothesis can be rejected.
advantages of using confidence intervals
ADVANTAGES OF USING CONFIDENCE INTERVALS
  • When the data are presented this way, it is possible for the experimenter to consider all possible null hypotheses – not just the one that states that the comparison in question will equal 0.
  • If any hypothesized value lies outside ofthe 95% confidence interval, it can be rejected.
choosing a method
CHOOSING A METHOD
  • Orthogonal tests can be done in either way.
  • Both methods make the same assumptions and are equally significant.
assumptions
ASSUMPTIONS
  • Assumptions:
    • The populations are approximately normally distributed.
    • Their variances are homogenous.
  • The t statistic is relatively robust to violations of these assumptions when the number of observations for each sample are equal.
  • However, when the sample sizes are not equal, the t statistic is not robust to the heterogeneity of variances.
how to deal with violations of assumptions
HOW TO DEAL WITH VIOLATIONS OF ASSUMPTIONS
  • When population variances are unequal, you can replace the pooled estimator of variance, MSerror, with individualvariance estimators for the means that you are comparing.
  • There are a number of possible procedures that can be used when the variance between populations is heterogeneous:
    • Cochran and Cox
    • Welch
    • Dixon, Massey, Satterthwaite and Smith
type i errors and orthogonal contrasts
TYPE I ERRORS AND ORTHOGONAL CONTRASTS
  • For C independent contrasts at some level of significance (α), the probability of making one or more Type 1 errors is equal to: 
    • 1-(1-α)C
  • As the number of independent tests increases, so does the probability of committing a Type 1 error.
  • This problem can be reduced (but not eliminated) by restricting the use of multiple t-tests to a priori orthogonal contrasts.
a priori non orthogonal contrasts
A PRIORI NON-ORTHOGONAL CONTRASTS
  • Contrasts of interest that ARE NOT independent.
  • In order to reduce the probability of making a Type 1 error, the significance level (α) is set for the whole family of comparisons that is being made, as opposed to for each individual comparison.
  • For Example:
    • Entire value of α for all comparisons combined is 0.05.
    • The value for each individual comparison would thus be less than that.
when do you do these
WHEN DO YOU DO THESE?
  • When contrasts are planned in advance.
  • They are relatively few in number.
  • BUT the comparisons are non-orthogonal (they are not independent).
  • i.e.: When one wants to contrast a control group mean with experimental group means.
dunn s multiple comparison procedure
DUNN’S MULTIPLE COMPARISON PROCEDURE
  • A.K.A.: Bonferoni t procedure.
  • Involves splitting up the value of α among a set of planned contrasts in an additive way.
  • For example:
    • Total α = 0.05, for all contrasts.
    • One is doing 2 contrasts.
    • α for each contrast could be 0.025, if we wanted to divide up the α equally.
dunn s multiple comparison procedure1
DUNN’S MULTIPLE COMPARISON PROCEDURE
  • If the consequences of making a Type 1 error are not equally serious for all contrasts, then you may choose to divide α unequally across all of the possible comparisons in order to reflect that concern.
dunn s multiple comparison procedure2
DUNN’S MULTIPLE COMPARISON PROCEDURE
  • This procedure also involves the calculation of a t statistic (tD).
  • The calculation involved in finding tD is identical to that for determining t for orthogonal tests:
dunn s multiple comparison procedure3
DUNN’S MULTIPLE COMPARISON PROCEDURE
  • However, you use a different table in order to look up the critical value (tDα;C,v).
    • Your total α value (not the value per comparison).
    • Number of comparisons (C).
    • And v, the number of degrees of freedom.
dunn s multiple comparison procedure one tailed tests
DUNN’S MULTIPLE COMPARISON PROCEDURE: ONE-TAILED TESTS
  • The table also only shows the critical values for two-tailed tests.
  • However, you can determine the approximate value of tDα;C,v for a one-tailed test by using the following equation: 
    • tDα;C,v≈ zα/C + (z3α/C + zα/C)/4(v-2)
      • Where the value of zα/C can be looked up in yet another table (“Areas under the Standard Normal Distribution”).
dunn s multiple comparison procedure4
DUNN’S MULTIPLE COMPARISON PROCEDURE
  • Instead of calculating tD for all contrasts of interest, you can simply calculate the critical difference (ψD) that a particular comparison must exceed in order to be significant:
    • ψD = tDα/2;C,v √(2MSerror/n).
  • Then compare this critical difference value to the absolute values of the differences between the means that you compared.
  • If they exceed ψD, they are significant.
dunn s multiple comparison procedure5
DUNN’S MULTIPLE COMPARISON PROCEDURE
  • For Example:
    • You have 5 means (Y1 through Y5).
    • ΨD = 8.45
    • Differences between means are:
  • Those differences that exceed the calculated value of ΨD (8.45, in this case) are significant.

(After Kirk 1982)

dunn sidak procedure
DUNN-SIDAK PROCEDURE
  • A modification of the Dunn procedure.
    • t statistic (tDS) and critical difference (ψDS).
  • There isn’t much difference between the two procedures at α < 0.01.
  • However, at increased values of α, this procedure is considered to be more powerful and more precise.
  • Calculations are the same for t and ψD.
  • Table is different.
dunn sidak procedure1
DUNN-SIDAK PROCEDURE
  • However, it is not easy to allocate the total value of α unevenly across a particular set of comparisons.
  • This is because the values of α for each individual comparison are related multiplicatively, as opposed to additively.
  • Thus, you can’t simply add the α’s for each comparison together to get the total value of α for all contrasts combined.
dunnett s test
DUNNETT’S TEST
  • For contrasts involving a control mean.
  • Also uses a t statistic (tD’) and critical difference (ψD’).
  • Calculations are the same for t and ψ.
  • Different table.
  • Instead of C, you use k, the number of means (including the control mean).
  • Note: unlike Dunn’s and Dunn-Sidak’s, Dunnet’s procedure is limited to k-1 non-orthogonal comparisons.
choosing a procedure a priori non orthogonal tests
CHOOSING A PROCEDURE : A PRIORI NON-ORTHOGONAL TESTS
  • Often, the use of more than one procedure will appear to be appropriate.
  • In such cases, compute the critical difference (ψ) necessary to reject the null hypothesis for all of the possible procedures.
  • Use the one that gives the smallest critical difference (ψ) value .
a priori orthogonal and non orthogonal contrasts
A PRIORI ORTHOGONAL and NON-ORTHOGONAL CONTRASTS
  • The advantage of being able to make all planned contrasts, not just those that are orthogonal, is gained at the expense of an increase in the probability of making Type 2 errors.
a priori and a posteriori non orthogonal contrasts
A PRIORI and A POSTERIORI NON-ORTHOGONAL CONTRASTS
  • When you have a large number of means, but only comparatively very few contrasts, a priori non-orthogonal contrasts are better suited.
  • However, if you have relatively few means and a larger number of contrasts, you may want to consider doing an a posterioritest instead.
a posteriori contrasts
A POSTERIORI CONTRASTS
  • There are many kinds, all offering different degrees of protection from Type 1 and Type 2 errors:
    • Least Significant Difference (LSD) Test
    • Tukey’s Honestly Significant Difference (HSD) Test
    • Spjtotvoll and Stoline HSD Test
    • Tukey-Kramer HSD Test
    • Scheffé’s S Test
    • Brown-Forsythe BF Procedure
    • Newman-Keuls Test
    • Duncan’s New Multiple Range Test
a posteriori contrasts1
A POSTERIORI CONTRASTS
  • Most are good for doing all possible pair-wise comparisons between means.
  • One (Scheffé’s method) allows you to evaluate all possible contrasts between means, whether they are pair-wise or not.
choosing an appropriate test procedure
CHOOSING AN APPROPRIATE TEST PROCEDURE
  • Trade-off between power and the probability of making Type 1 errors.
  • When a test is conservative, it is less likely that you will make a Type 1 error.
  • But it also would lack power, inflating the Type 2 error rate.
  • You will want to control the Type 1 error rate without loosing too much power.
  • Otherwise, you might reject differences between means that are actually significant.
doing planned contrasts using sas
DOING PLANNED CONTRASTS USING SAS
  • You have a data set with one dependent variable (y) and one independent variable (a).
  • (a) has 4 different treatment levels (1, 2, 3, and 4).
  • You want to do the following comparisons between treatment levels:
    • 1)Y1-Y2
    • 2) Y3-Y4
    • 3) (Y1-Y2)/2 - (Y3-Y4)/2
are they orthogonal
ARE THEY ORTHOGONAL?

YES!

...BUT THEY DON’T HAVE TO BE

doing planned orthogonal contrasts using sas
DOING PLANNED ORTHOGONAL CONTRASTS USING SAS
  • SAS INPUT:

data dataset;

input y a;

cards;

3 1

4 2

7 3

7 4

.............

proc glm;

class a;

model y = a;

contrast 'Compare 1 and 2' a 1 -1 0 0;

contrast 'Compare 3 and 4' a 0 0 1 -1;

contrast 'Compare 1 and 2 with 3 and 4' a 1 1 -1 -1;

run;

Give your dataset an informative name.

Tell SAS what you’ve inputted: column 1 is your y variable (dependent) and column 2 is your a variable (independent).

This is followed by your actual data.

“Model” tells SAS that you want to look at the effects that a has on y.

Use SAS procedure proc glm.

“Class” tells SAS that a is categorical.

Indicate “weights” (kind of like the coefficients).

Then enter your “contrast” statements.

recommended reading
RECOMMENDED READING
  • Kirk RE. 1982. Experimental design: procedures for the behavioural sciences. Second ed. CA: Wadsworth, Inc.
  • Field A, Miles J. 2010. Discovering Statistics Using SAS. London: SAGE Publications Ltd.
  • Institute for Digital Research and Education at UCLA: http://www.ats.ucla.edu/stat/
    • Stata, SAS, SPSS and R.
planned comparisons using a t statistic3
PLANNED COMPARISONS USING A t STATISTIC
  • t = ∑cjYj/ √MSerror∑cj/nj
    • Where c is the coefficient, Y is the corresponding mean, n is the sample size, and MSerror is the Mean Square Error.
planned comparisons using a t statistic4
PLANNED COMPARISONS USING A t STATISTIC
  • For Example, you want to compare 2 means:
    • Y1 = 48.7 and Y2=43.4
    • c1 = 1 and c2 = -1
    • n=10
    • MSerror=28.8
  • t = (1)48.7 + (-1)43.4

[√28.8(12/10) + (-12/10)]

= 2.21

confidence intervals for orthogonal contrasts2
CONFIDENCE INTERVALS FOR ORTHOGONAL CONTRASTS
  • (Y1-Y2) – tα/2,v(SE) ≤ ψ ≤ (Y1-Y2) – tα/2,v(SE)
  • For Example:
    • If, Y1 = 48.7, Y2 = 43.4, and SE(tα(2),df) = 4.8
    • (Y1-Y2) – SE(tα(2),df) = 0.5
    • (Y1-Y2) + SE(tα(2),df) = 10.1
    • Thus, you can be 95% confident that the true value of ψ is between 0.5 and 10.1.
    • Because the confidence interval does not include 0, you can also reject the null hypothesis that Y1-Y2 = 0.

(Example after Kirk 1982)

type i errors and orthogonal contrasts1
TYPE I ERRORS AND ORTHOGONAL CONTRASTS
  • As the number of independent tests increases, so does the probability of committing a Type 1 error.
  • For Example, when α = 0.05:
    • 1-(1-0.05)3=0.14 (C=3)
    • 1-(1-0.05)5=0.23 (C=5)
    • 1-(1-0.05)10=0.40 (C=10)

(Example after Kirk 1982)