1 / 48

Significance testing and confidence intervals

Significance testing and confidence intervals. Ágnes Hajdu EPIET Introductory course 3.10.2011. The idea of statistical inference. Generalisation to the population. Conclusions based on the sample . Population. Hypotheses. Sample. Inferential statistics.

Faraday
Download Presentation

Significance testing and confidence intervals

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Significance testingand confidence intervals Ágnes Hajdu EPIET Introductorycourse 3.10.2011

  2. The idea of statistical inference Generalisation to the population Conclusions based on the sample Population Hypotheses Sample

  3. Inferential statistics • Usespatterns in the sample data to draw inferences about the population represented, accounting for randomness. • Twobasicapproaches: • Hypothesis testing • Estimation • Commongoal: concludeontheeffect of an independentvariable (exposure) on a dependentvariable (outcome).

  4. The aim of a statistical test To reach a scientific decision (“yes” or “no”) on a difference (or effect), on a probabilistic basis, on observed data.

  5. Why significance testing? Botulism outbreak in Italy: “The risk of illness was higher among diners who ate home preserved green olives (RR=3.6).” Is the association due to chance?

  6. The two hypothesis! When you perform a test of statistical significance you usually reject or do not reject the Null Hypothesis (H0)

  7. Botulism outbreak in Italy • Null hypothesis (H0): “There is no association between consumption of green olives and Botulism.” • Alternative hypothesis(H1): “There is an association between consumption of green olives and Botulism.”

  8. Hypothesis, testing and null hypothesis • Tests of statistical significance • Data not consistent with H0 : • H0 can be rejected in favour of some alternative hypothesis H1 (the objective of our study). • Data are consistent with the H0: • H0 cannot be rejected You cannotsay that the H0 is true. You can only decide to reject it or not reject it.

  9. How to decide when to reject the null hypothesis? H0 rejected using reportedpvalue p-value = probability that our result (e.g. a difference between proportions or a RR) or more extreme values could be observed under the null hypothesis

  10. p values – practicalities Small p values = low degree of compatibility between H0 and the observed data: you reject H0, the test is significant Large p values= high degree of compatibility between H0 and the observed data: you don’t reject H0, the test is not significant We can never reduce to zero the probability that our result was not observed by chance alone

  11. Levels of significance – practicalities We need of a cut-off ! 0.01 0.05 0.10 p value > 0.05 = H0 non rejected (non significant) p value ≤ 0.05 = H0 rejected (significant) BUT: Give always the exact p-value rather than „significant“ vs. „non-significant“.

  12. Examples from the literature • ”The limit for statistical significance was set at p=0.05.” • ”There was a strong relationship (p<0.001).” • ”…, but it did not reach statistical significance (ns).” • „ The relationshipwasstatisticallysignificant (p=0.0361)” p=0.05 Agreed convention Not an absolute truth ”Surely, God loves the 0.06 nearly as much as the 0.05” (Rosnow and Rosenthal, 1991)

  13. p = 0.05 and its errors • Level of significance, usually p = 0.05 • p value used for decision making But still 2 possible errors: • H0should not be rejected, but it was rejected : • Type I or alpha error • H0should be rejected, but it was not rejected : Type II or beta error

  14. Types of errors Truth No diff Diff Decision based on the p value No diff Diff • H0 is “true” but rejected: Type I or  error • H0 is “false” but not rejected: Type II or  error

  15. More on errors • Probability of Type I error: • Value of α is determinedinadvance of the test • The significancelevel is thelevel of αerrorthatwewouldaccept (usually 0.05) • Probability of Type II error: • Value of βdependsonthesize of effect (e.g. RR, OR) and samplesize • 1-β: Statisticalpower of a studytodetect an effecton a specifiedsize (e.g. 0.80) • Fix βinadvance: choose an appropriatesamplesize

  16. Even more onerrors H1 is true H0 is true b a Test statistics T

  17. Principles of significance testing • Formulate the H0 • Test your sample data against H0 • The p value tells you whether your data are • consistent with H0 • i.e, whether your sample data are consistent with a chance finding (large p value), or whether there is reason to believe that there is a true difference (association)between the groups you tested • You can only reject H0, or fail to reject it!

  18. Quantifying the association • Test of association of exposure and outcome • E.g. Chi2 test or Fisher’s exact test • Comparison of proportions • Chi2-value quantifies the association • The larger the Chi2-value, the smaller the p value • the more the observed data deviate from the assumption of independence (no effect).

  19. Chi-square value = sum of all cells: for each cell, subtract the expected number from the observed number, square the difference, and divide by the expected number

  20. Botulism outbreak in Italy2x2 table Expected number of ill and not ill for each cell : Ill Non ill x10% ill Olives 52 x 90% non-ill 47 5 No olives x10% ill 83 x 90% non-ill 8 75 13 122 135 Expected proportion of ill and not ill : 10 % 90 %

  21. Chi-square value = 5.73 Botulism outbreak in Italy p = 0.016 Ill Non ill Olives No olives

  22. Botulism outbreak in Italy “The relative risk (RR) of illness among diners who ate home preserved green olives was 3.6 (p=0.016).” The p-value is smaller than the chosen significance level of a = 5%. →Null hypothesis can be rejected. There is a 0.016 probability (16/1000) thattheobservedassociationcouldhaveoccuredbychance, iftherewere no trueassociationbetween eatingolives and illness.

  23. Epidemiology and statistics

  24. Criticism on significance testing “Epidemiological application need more than a decision as to whether chance alone could have produced association.” (Rothman et al. 2008) →Estimation of an effect measure(e.g. RR, OR) rather than significance testing.

  25. Why estimation? Botulism outbreak in Italy: “The risk of illness was higher among diners who ate home preserved green olives (RR=3.6).” How confident can we be in the result? What is the precision of our point estimate?

  26. The epidemiologist needs measurements rather than probabilities 2 is a test of association OR, RR are measures of association on a continuous scale infinite number of possible values The best estimate = point estimate Range of values allowing for random variability: Confidence interval precision of the point estimate

  27. Confidence interval (CI) Range of values, on the basis of the sample data, in which the population value (or true value) may lie. • Frequently used formulation: „If the data collection and analysis could be replicated many times, the CI should include the true value of the measure95% of the time.”

  28. Confidenceinterval (CI) e.g. CI for means 95% CI = x – 1.96 SEuptox + 1.96 SE a = 5% 1 - α α/2 α/2 s Lower limit upper limit of 95% CI of 95% CI Indicatestheamount of random errorintheestimate Can becalculatedforany „teststatistic“, e.g.: means, proportions,ORs, RRs

  29. CI terminology Point estimate Confidence interval RR = 1.45 (0.99 – 2.1) Lower confidence limit Upper confidence limit

  30. Width of confidence interval depends on … • The amount of variability in the data • The size of the sample • The arbitrary level of confidence you desire for your study (usually 90%, 95%, 99%) A common way to use CI regarding OR/RR is : If 1.0 is included in CI non significant If 1.0 is not included in CI  significant

  31. A B Large RR RR = 1 Looking the CI Study A, large sample, precise results, narrow CI – SIGNIFICANT Study B, small size, large CI - NON SIGNIFICANT Study A, effect close to NO EFFECT Study B, no information about absence of large effect

  32. 1 RR  20 studies with different results... More studies are better or worse? • Decision making based on results from a collection of studies is not facilitated when each study is classified as a YES or NO decision. • Need to look at the point estimation and its CI • But also consider its clinical or biological significance

  33. Botulism outbreak in Italy • How confident can we be in the result? • Relative risk = 3.6 (point estimate) • 95% CI for the relative risk: (1.17 ; 11.07) The probability that the CI from 1.17to 11.07 includes the true relative risk is 95%.

  34. Botulism outbreak in Italy “The risk of illness was higher among diners who ate home preserved green olives (RR=3.6, 95% CI 1.17 to 11.07).”

  35. The p-value (or CI) function • A graph showing the pvalue for all possible values of the estimate (e.g. OR or RR). • Quantitative overview of the statistical relation between exposure and disease for the set of data. • All confidence intervals can be read from the curve. • The function can be constructed from the confidence limits in Episheet.

  36. Example: Chlordiazopoxide use and congenital heart disease OR = (4 x 1250) / (4 x 386) = 3.2 p=0.08 ; 95% CI=0.81–13 From Rothman K

  37. 3.2 p=0.08 Odds ratio 0.81 - 13

  38. Example: Chlordiazopoxide use and congenital heart disease – large study OR = (1090 x 15000) / (1000 x 14910) = 1.1 p=0.04 ; 95% CI=1.05-1.2 From Rothman K

  39. Precision and strength of association Strength Precision

  40. Confidence interval provides more information than p value • Magnitude of the effect (strength of association) • Direction of the effect (RR > or < 1) • Precision of the point estimate of the effect (variability) p value can not provide them !

  41. What we have to evaluate the study • 2A test of association. It depends on sample size. • p valueProbability that equal (or more extreme) results can be observed by chance alone • OR, RRDirection & strength of association if > 1 risk factor if < 1 protective factor (independently from sample size) • CI Magnitude and precision of effect

  42. Comments on p-values and CIs • Presence of significance does not prove clinical or biological relevance of an effect. • A lack of significance is not necessarily a lack of an effect: “Absence of evidence is not evidence of absence”.

  43. Comments on pvalues and CIs • A huge effect in a small sample or a small effect in a large sample can result in identical pvalues. • A statistical test will always give a significant result if the sample is big enough. • p values and CIs do not provide any information on the possibility that the observed association is due to bias or confounding.

  44. 2 and Relative Risk Cases Non cases Total 2 = 1.3 E 9 51 60 p = 0.13 NE 5 55 60 RR = 1.8 Total 14 106 120 95% CI [ 0.6 - 4.9 ] Cases Non cases Total 2 = 12 E 90 510 600 p = 0.0002 NE 50 550 600 RR = 1.8 Total 140 1060 1200 95% CI [ 1.3-2.5 ] « Too large a difference and you are doomed to statistical significance »

  45. Common source outbreak suspected Exposure cases non cases AR% Yes 15 20 42.8% No 50 200 20.0% Total 65 220 2 = 9.1 p = 0.002 RR = 2.1 95%CI= 1.4-3.4 23% REMEMBER: These values do not provide any information on the possibility that the observed association is due to a bias or confounding.

  46. Recommendations • Always look at the raw data (2x2-table). How many cases can be explained by the exposure? • Interpret with caution associations that achieve statistical significance. • Double caution if this statistical significance is not expected. • Use confidence intervals to describe your results. • Report pvalues precisely.

  47. Suggested reading • KJ Rothman, S Greenland, TL Lash, Modern Epidemiology, Lippincott Williams & Wilkins, Philadelphia, PA, 2008 • SN Goodman, R Royall, Evidence and Scientific Research, AJPH 78, 1568, 1988 • SN Goodman, Toward Evidence-Based Medical Statistics. 1: The P Value Fallacy, Ann Intern Med. 130, 995, 1999 • C Poole, Low P-Values or Narrow Confidence Intervals: Which are more Durable? Epidemiology 12, 291, 2001

  48. Previous lecturers • Alain Moren • Paolo D’Ancona • Lisa King • Preben Aavitsland • DorisRadun • Manuel Dehnert

More Related