Making all research results publically available: the cry of systematic reviewers

1. Making all research results publically available: the cry of systematic reviewers

2. Outline of talk • Definitions • History • Who contributes to publication bias • Impact of publication bias • Funnel plots and interpreting them • Outcome reporting bias • Critical appraisal of systematic review

3. Memory jog • Grey literature • “That which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers” • Publication bias • “Investigators, reviewers, and editors submit or accept manuscripts for publication based on the direction or strength of the study findings”

4. Publication bias • 1959 • 294 reports from 4 leading psychology journals • 97.3% reported statistically positive results • 1986-1987 • 456 reports from three leading from psychology journals and 3 healthcare journals (NEJM, Am J Epi, Am J Pub Health) • 97% of psychology journals reported positive results • 85% of Medical journals reported positive results

5. Publication bias: contributors (1) • Researchers • Are likely the major source • Peer reviewers • Experimental evidence has shown that reviewers are highly influenced by the direction and strength of results in a submitted manuscript1 • Editors

6. Publication bias: contributors (2) • Editors! Letter from the editor for a major environmental/toxicological journal to the author of a submitted manuscript

7. Consequences of non-publication bias(1) • Management (survival) of ovarian cancer • results of 13 published trials • results of 16 trials (including 3 registered only) • Pooling the results of published trials only • statistically favouring combination chemotherapy compared to alkylating agent (16% advantage) • Pooling all 16 trials • non-significant advantage of 5% • Provides clinicians and patients alike with differing estimates as to the purported effectiveness of a cancer intervention.

8. Consequences of non-publication bias(2) • Methods • compared the results of 365 published trials with 102 ‘grey’ trials, included in 33 systematic reviews • Excluding the results of grey literature exaggerated the treatment effectiveness by 15%, on average • Grey literature accounts for approximately 25% of studies included in systematic reviews • The 102 grey literature randomized trials included more than 23,000 participants

9. Publication Bias: impact • Consequences for Systematic Reviews and Meta-Analyses • Biased summary estimates that are falsely positive, precise and accurate • Consequences of guideline and health policy development • Practice may be influenced (and even mandated) by false conclusions

10. Publication Bias: Detection • The Funnel Plot • A measure of the treatment effect size plotted against a measure of the study’s sample size or precision • The precision of the estimation of the true effect increases with larger sample sizes • Funnel plots investigate whether studies with little precision (small studies) give different results from studies with greater precision (larger studies)

11. Funnel Plots

12. Interpreting Funnel Plots • The ability of researches to identify bias using funnel plots was shown in one study to be 53%

13. Funnel plot asymmetry Causes: • Selection bias • Publication bias – (One of many reasons!) • Language bias • Citation bias • Multiple publication bias • True Heterogeneity • Intensity of intervention • Characteristics of the patient population • Methodological Quality • Outcome measure and analysis • Chance

14. Publication Bias: time lag bias • Statistically significant positive studies published before null studies • Systematic reviews –a cross section cut in time • Thus, trials with positive results could dominate the literature and could introduce bias for several years

15. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised trials • As a rule of thumb, tests for funnel plot asymmetry should not be used when there are fewer than 10 studies. • test power is usually too low to distinguish chance from real asymmetry • When there is evidence of funnel plot asymmetry, publication bias is only one possible explanation • As far as possible, a testing strategy should be specified in advance. Applying and reporting many tests is discouraged: if more than one test is used, all test results should be reported.

16. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised trials • Test results should be interpreted in the context of visual inspection of funnel plots. • for example, are there studies with markedly different intervention effect estimates or studies that are highly influential in the asymmetry test? Even if an asymmetry test is statistically significant, publication bias can probably be excluded if small studies tend to lead to lower estimates of benefit than larger studies or if there are no studies with significant results

17. (intra-study) Publication bias • Selective reporting bias • Outcome reporting bias • Typically statistically positive • Selected by investigators (post hoc) • Data analyses reporting bias

18. Some salient results • Nearly two-thirds had a change in at least one primary outcome between the protocol and publication • Statistically significant outcomes had a higher likelihood of being reported compared to non-significant ones

19. Do researchers have a social obligation to study participants? • Other realms of life • airline industry • hotels

20. The SSRI story • Selective serotonin reuptake inhibitors [SSRI] commonly used class of antidepressants to treat major depression in children • Concerns that use of these drugs increase risk of suicide • Results of systematic review • published data support use of paxil • addition of unpublished data tip the harm/benefit balance and do not support use of drug

21. PRISMA statement • Guideline for reporting systematic reviews and meta-analyses • Item 15 • “Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies)..”

22. Rationale for reporting assessment of bias across studies • Reviewers should explore the possibility that the available data are biased. • They may examine results from the available studies for clues that suggest there may be: • missing studies (publication bias) • missing data from the included studies (selective reporting bias)