Chapter 21

1. Chapter 21 More About Tests

2. More About P-values • The p-value is not the probability that the null hypothesis is true. • The p-value is the probability of the observed statistic given that the null hypothesis is true.

3. More About P-values • Interpretation: If the null hypothesis is true, the probability of getting the observed statistic, or one more extreme, is the p-value. • A small p-value means that the observed statistic is unusual if the null hypothesis is true. So, most likely the null hypothesis is incorrect.

4. How Small is Small? • This is determined by an alpha level. • This is our “threshold” or “cut-off” • The most common alpha levels are 0.1, 0.05, and 0.01

5. How Small is Small? • The alpha level used sometimes depends on the situation. • Assessing the reliability of parachutes. We want a very, very small occurrence of failures, so we might choose an alpha level or 0.01, or even smaller. • Pizza: Pepperoni or not? Not a life or death situation so may just be happy with an alpha level of 0.1.

6. Rejecting the Null Hypothesis • When do we reject Ho? • We reject Ho if our p-value is less than the significance level that we choose. • If p-value < a, there is sufficient evidence to reject the null hypothesis • If p-value > a, there is not sufficient evidence to reject the null hypothesis.

7. Different Alpha Levels • Different alpha levels can lead to different conclusions. • Consider a p-value of 0.045. • 0.045 < a = 0.05 and 0.045 < a = 0.1 – for both of these alpha levels we would reject Ho • What if we use a = 0.01? • Not reject the null hypothesis because p-value > a

8. Different Alpha Levels • If a p-value is statistically significant at a certain a level, then it is also significant at all higher a levels. • If a p-value is statistically significant at a certain a level, it is not necessarily significant at lower a levels.

9. Different Alpha Levels - Example • A researcher developing scanners to search for hidden weapons at airports has concluded that a new device is significantly better than the current scanner. He made this decision based on a test using a = 0.05. Would he have made the same decision ata = 0.10? How about a = 0.01?

10. Different Alpha Levels - Example • Ho: no difference between scanners • Ha: new scanner is better • Conclusion: new is better, so Ho was rejected • p-value < a = 0.05, so p-value < a = 0.10 • p-value ? a = 0.01 • The same decision would be made at a = 0.10 but we don’t know what decision would be made with a = 0.01.

11. Errors

12. Errors • Probability of Type I error • Significance level α. • α% of the time our decision to reject Ho will be wrong. • Probability of Type II error • β% of the time our decision not to reject Ho will be wrong. • Error depends on true value of p. • Want this error to be small when po is far away from p.

13. Practical Significance • Often, if n is large, we will reject Ho. • The actual difference between poand the true value of p could be very small. • po and the true value of p could be practically the same. • Statistical significance is not the same as practical significance.

14. Using Tests Wisely • How small should a be? • How plausible is Ho? • If the null hypothesis is strongly held belief, you need a lot of evidence against the null hypothesis to convince people it’s wrong. Would need a small p-value.

15. Using Tests Wisely • What are the consequences of rejecting Ho. • Change in manufacturer procedures, etc. are expensive. We need a lot of evidence against the null hypothesis to convince people to spend a large amount of money.

16. Using Tests Wisely - Example • A company developed a program to reduce the number of elementary school students who read below their grade level. They supplied materials and teacher training for a large-scale test involving nearly 8500 children in different school districts. The percentage of students who did not attain the grade level standard was reduced from 15.9% to 15.1%. A hypothesis test was conducted; Ho was rejected with a p-value of 0.023.