Understanding Significance Testing: A Comprehensive Guide to Hypothesis Testing

1. STA 291Fall 2009 Lecture 18 Dustin Lueker

2. Significance Test • A way of statistically testing a hypothesis by comparing the data to values predicted by the hypothesis • Data that fall far from the predicted values provide evidence against the hypothesis STA 291 Fall 2009 Lecture 18

3. Logical Procedure • State a hypothesis that you would like to find evidence against • Get data and calculate a statistic • Sample mean • Sample proportion • Hypothesis determines the sampling distribution of our statistic • If the sample value is very unreasonable given our initial hypothesis, then we conclude that the hypothesis is wrong STA 291 Fall 2009 Lecture 18

4. Elements of a Significance Test • Assumptions • Type of data, population distribution, sample size • Hypotheses • Null hypothesis • H0 • Alternative hypothesis • H1 • Test Statistic • Compares point estimate to parameter value under the null hypothesis • P-value • Uses the sampling distribution to quantify evidence against null hypothesis • Small p-value is more contradictory • Conclusion • Report p-value • Make formal rejection decision (optional) • Useful for those that are not familiar with hypothesis testing STA 291 Fall 2009 Lecture 18

5. P-value • How unusual is the observed test statistic when the null hypothesis is assumed true? • The p-value is the probability, assuming that the null hypothesis is true, that the test statistic takes values at least as contradictory to the null hypothesis as the value actually observed • The smaller the p-value, the more strongly the data contradicts the null hypothesis STA 291 Fall 2009 Lecture 18

6. Conclusion • In addition to reporting the p-value, sometimes a formal decision is made about rejecting or not rejecting the null hypothesis • Most studies require small p-values like p<.05 or p<.01 as significant evidence against the null hypothesis • “The results are significant at the 5% level” • α=.05 STA 291 Fall 2009 Lecture 18

7. P-values and Their Significance • p-value<.01 • Highly significant • “Overwhelming evidence” • .01<p-value<.05 • Significant • “Strong evidence” • .05<p-value<.1 • Not Significant • “Weak evidence • p-value>.1 • Not Significant • “No evidence” • Whether or not a p-value is considered significant typically depends on the discipline that is conducting the study STA 291 Fall 2009 Lecture 18

8. Terminology • Significance level • Alpha level • α • Number such that one rejects the null hypothesis if the p-values is less than it • Most common are .05 and .01 • Needs to be chosen before analyzing the data • Why? STA 291 Fall 2009 Lecture 18

9. Type I and Type II Errors STA 291 Fall 2009 Lecture 18

10. Type I and Type II Errors • α=probability of Type I error • β=probability of Type II error • Power=1-β • The smaller the probability of Type I error, the larger the probability of Type II error and the smaller the power • If you ask for very strong evidence to reject the null hypothesis (very small α), it is more likely that you fail to detect a real difference • In reality, α is specified, and the probability of Type II error could be calculated, but the calculations are often difficult STA 291 Fall 2009 Lecture 18

11. Example • In a criminal trial someone is assumed innocent until proven guilty • What type of error (in terms of hypothesis testing) would be made if an innocent person is found guilty? • What type of error would be made if a guilty person is found not guilty? • What does the Power represent (1-β)? • Also, the reason we only do not reject H0 instead of saying that we accept H0 is because of the way our hypothesis tests are set up • Just like in a criminal trial someone is found not guilty instead of innocent STA 291 Fall 2009 Lecture 18

12. How to choose α? • If the consequences of a Type I error are very serious, then α should be small • Criminal trial example • In exploratory research, often a larger probability of Type I error is acceptable • If the sample size increases, both error probabilities decrease STA 291 Fall 2009 Lecture 18

13. How to choose α? • Which area of study would be most likely to use a very small level of significance? • Social Sciences • Medical • Physical Sciences STA 291 Fall 2009 Lecture 18

14. Hypotheses • H0: p=p0 • p0 is the value we are testing against • H1: p≠p0 • Most common alternative hypothesis • This is called a two-sided hypothesis since it includes values falling on two sides of the null hypothesis (above and below) STA 291 Fall 2009 Lecture 18

15. Test Statistic • The z-score has a standard normal distribution • The z-score measures how many estimated standard errors the sample proportion falls from the hypothesized population proportion • The farther the sample proportion falls from p0 the larger the absolute value of the z test statistic, and the stronger the evidence against the null hypothesis • Sample size restrictions STA 291 Fall 2009 Lecture 18

16. Example • Let p denote the proportion of Floridians who think that government environmental regulations are too strict • A telephone poll of 824 people conducted in June 1995 revealed that 26.6% said regulations were too strict • Test H0: p=.5 at α=.05 • Calculate the test statistic • Find the p-value and interpret STA 291 Fall 2009 Lecture 18

17. P-value • Has the advantage that different test results from different tests can be compared • Always a number between 0 and 1, no matter why type of data is being examined • Probability that a standard normal distribution takes values more extreme than the observed z-score • The smaller the p-value, the stronger the evidence against the null hypothesis and in favor of the alternative hypothesis STA 291 Fall 2009 Lecture 18

18. One-Sided Significance Tests • The research hypothesis is usually the alternative hypothesis (H1 or HA) • The alternative is the hypothesis that we want to prove by rejecting the null hypothesis • Assume that we want to prove that μ is larger than a particular number μ0 • We need a one-sided test with hypotheses • Null hypothesis can also be written with an equals sign STA 291 Fall 2009 Lecture 18

19. Example • For a large sample test of the hypothesis the z test statistic equals 1.04 • Find the p-value and interpret • Suppose z=2.5 rather than 1.04, find the p-value • Does this provide stronger or weaker evidence against the null hypothesis? • Now consider the one-sided alternative • Find the p-value and interpret • For one-sided tests, the calculation of the p-value is different • “Everything at least as extreme as the observed value” is everything above the observed value in this case • Notice the alternative hypothesis STA 291 Fall 2009 Lecture 18

20. One-Sided vs. Two-Sided Test • Two sided tests are more common in practice • Look for formulations like • “test whether the mean has changed” • “test whether the mean has increased” • “test whether the mean is thesame” • “test whether the mean has decreased” STA 291 Fall 2009 Lecture 18

21. Example • If someone wanted to test to see if the average miles a social worker drives in a month was at least 2000 miles, what would H1 be? H0? • μ<2000 • μ≤2000 • μ≠2000 • μ≥2000 • μ>2000 • μ=2000 STA 291 Fall 2009 Lecture 18