1 / 37

Beyond Null Hypothesis Testing

Beyond Null Hypothesis Testing. Supplementary Statistical Techniques. Limitations of NHT. Criticisms of NHT date from the 1930s. Null hypothesis is rarely true. The real question is not about the existence of an effect, but about the nature of the effect:

conan-trevino
Download Presentation

Beyond Null Hypothesis Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Beyond Null Hypothesis Testing Supplementary Statistical Techniques

  2. Limitations of NHT • Criticisms of NHT date from the 1930s. • Null hypothesis is rarely true. • The real question is not about the existence of an effect, but about the nature of the effect: • What is the direction of the effect? • What is the size of the effect? • How important is it? • What are the underlying mechanisms (theory)? PSYC 6130, PROF. J. ELDER

  3. (Canadian Community Health Survey, 2004) Direction of Effect • NHT is reasonably well suited to testing the direction of an effect. • For example, are tall men more or less likely to be wealthy? PSYC 6130, PROF. J. ELDER

  4. Magnitude of Effect • NHT by itself tells us nothing about the magnitude of an effect. • This is really a problem of descriptive statistics. • The simplest descriptor of the magnitude of an effect is a point estimate: PSYC 6130, PROF. J. ELDER

  5. Magnitude of Effect • A problem with a point estimate is that it suggests a certainty we do not really have. • A more complete and useful description of the magnitude of the effect is provided by a confidence interval. PSYC 6130, PROF. J. ELDER

  6. Importance of Effect • However, even a confidence interval does not really tell us whether a treatment or factor is important. • One way to judge whether a difference of means is ‘big’ is to compare the size of the difference of the means to the values of the means themselves, e.g., i.e., wealthy men are roughly 1.2% taller. PSYC 6130, PROF. J. ELDER

  7. Importance of Effect • However it is often more meaningful to compare the treatment effect to the overall variation in the measured variable. • We call this normalized measure of the effect the effect size d. PSYC 6130, PROF. J. ELDER

  8. Importance of Effect • However it is often more meaningful to compare the treatment effect to the overall variation in the measured variable: PSYC 6130, PROF. J. ELDER

  9. Example Effect Sizes Group 1 Group 2 PSYC 6130, PROF. J. ELDER

  10. Importance of Effect • d provides a sense of how much of the variation in the dependent variable is due to the ‘treatment’. PSYC 6130, PROF. J. ELDER

  11. End of Lecture 6 Oct 22, 2008

  12. Theory • Even when augmented with measures of effect size, NHT does not directly tell us about the mechanism by which the treatment impacts the dependent variable. • e.g., Wealthy men are taller because… • Tall men attract wealthy women? • Wealthy men come from wealthy families that provided better care (e.g., nutrition). • To understand these relationships, it is not enough to test the significance of and quantify effects. • Ultimately, we require detailed, mechanistic (causal), testable theories, and experiments that test these theories. • These theories should generate quantitative predictions, that can be compared against experimental outcomes. • The theory that provides the closest quantiative account of the data should be considered our current ‘working hypothesis’ about how the system under study operates. • When comparing theories, we must beware of “Occam’s Razor”. • This process is less dependent on NHT, and more dependent upon model fitting, analysis of variance and cross-validation techniques. PSYC 6130, PROF. J. ELDER

  13. Planning Experiments: Statistical Power

  14. Planning a Study • There are many considerations that go into planning an experiment or study. • Here we focus on the statistical considerations. • Some possible questions: • How many samples (e.g., subjects) will I need for my study? • I already know that I will only have access to n samples (subjects). Will this be enough? • Answering these questions depends on understanding the relationship between sample size, effect size, and statistical power. PSYC 6130, PROF. J. ELDER

  15. Sample Size and Effect Size Codetermine Power + + PSYC 6130, PROF. J. ELDER

  16. Statistical Power • Power is defined as the complement of the Type II error rate. • Thus understanding power means understanding Type II errors. PSYC 6130, PROF. J. ELDER

  17. Type I Errors and the Null Hypothesis Distribution (NHD) • To understand Type I errors, we considered the situation where the null hypothesis is true, and modeled the null hypothesis distribution. PSYC 6130, PROF. J. ELDER

  18. Understanding Type II Errors • To understand the factors that determine Type II errors, we need to model the situation when the null hypothesis is false and the alternative hypothesis is true. • The difficulty is that the alternative hypothesis typically encompasses a range of possible population means, and we do not know which one is the correct mean. • But suppose for the moment we did. This defines the alternative hypothesis distribution (AHD), which follows a non-central t distribution. • We will often approximate this as a normal distribution, in order to compute rough estimates of power. PSYC 6130, PROF. J. ELDER

  19. NHD AHD Sampling Distributions of the Difference of the Means 0.4 0.3 0.2 Probability p 0.1 0 0 PSYC 6130, PROF. J. ELDER

  20. Standardizing the Alternative Hypothesis Distribution • Just as for the NHD, it is useful to standardize the AHD: PSYC 6130, PROF. J. ELDER

  21. NHD AHD Standardized Distributions of the Difference of the Means 0.4 0.3 0.2 Probability p(t) 0.1 0 -4 -2 2 4 6 8 0 t PSYC 6130, PROF. J. ELDER

  22. Planning an Experiment: Approximations • Estimates of effect size are always approximate, and so it is reasonable to make approximations when planning a study. • For example: PSYC 6130, PROF. J. ELDER

  23. Standardized Distributions of the Difference of the Means (Assume homogeneity of variance, equal sample sizes) PSYC 6130, PROF. J. ELDER

  24. NHD AHD Standardized Distributions of the Difference of the Means 0.4 0.3 0.2 Probability p(t) 0.1 0 -4 -2 2 4 6 8 0 t PSYC 6130, PROF. J. ELDER

  25. Pr(t) Pr(t) Expected t value Estimating Power (Non-central t distribution) (Central t distribution) PSYC 6130, PROF. J. ELDER

  26. - + - + + Calculating Power from Sample Size and Effect Size PSYC 6130, PROF. J. ELDER

  27. Planning Experiments • Planning experiments may involve estimating any one of these variables given knowledge or assumptions about the other two: PSYC 6130, PROF. J. ELDER

  28. Example: Height Difference between Men and Women PSYC 6130, PROF. J. ELDER

  29. Example 1. • From this large prior study we know men are on average 5.4” taller than women. • We wish to see if this also applies to University students, i.e., whether male students are taller on average than female students. • What power will we obtain if we have a class of 10 males and 10 females? PSYC 6130, PROF. J. ELDER

  30. PSYC 6130, PROF. J. ELDER

  31. Example 2a. • From this large prior study we know men are on average 5.4” taller than women. • We wish to see if this also applies to University students, i.e., whether male students are taller on average than female students. • What sample size do we need to obtain power of 0.8? PSYC 6130, PROF. J. ELDER

  32. Example 2b. • Suppose we only care about differences greater than 1” • Suppose also that we wish to have power of at least .8 (i.e., 80% chance of rejecting the null hypothesis, given it is false) for a 2-tailed test with a=.05. • What is the maximum sample size worth collecting? PSYC 6130, PROF. J. ELDER

  33. Example 3. • Suppose we are stuck with a sample size of 10 (i.e., 10 men and 10 women). Is it worth doing the study? • Let’s decide that it is not worth doing the study unless we have power of at least .8 (i.e., 80% chance of rejecting the null hypothesis, given it is false) for a 2-tailed test with a=.05. PSYC 6130, PROF. J. ELDER

  34. Manipulating Power • In theory, power can be manipulated by changing • Sample size • Alpha level • Effect size • Increase strength of treatment • Decrease variability • Control of nuisance variables • Matched designs PSYC 6130, PROF. J. ELDER

  35. One-Sample Tests Note the greater power of one-sample tests, relative to two-sample tests! PSYC 6130, PROF. J. ELDER

  36. Unequal Sample Sizes • When samples are of different size, apply same formulas for estimating power, using average sample size. • Most accurate method is to use the harmonic mean: PSYC 6130, PROF. J. ELDER

  37. (e.g., Dunlop et al., 1996) (e.g., Rosenthal, 1991) Effect Size for Paired Sample Designs • Two methods for computing effect size for paired designs: • Either method is fine, as long as you know what it means! PSYC 6130, PROF. J. ELDER

More Related