Can unpublished data be of value in systematic reviews of adverse effects?
1 / 26

Su Golder, MRC Fellow in Health Services Research - PowerPoint PPT Presentation

  • Uploaded on

Can unpublished data be of value in systematic reviews of adverse effects?. Su Golder, MRC Fellow in Health Services Research This research is being undertaken as part of an MRC fellowship

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Su Golder, MRC Fellow in Health Services Research' - hollis

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Su golder mrc fellow in health services research spg3 york ac uk

Can unpublished data be of value in systematic reviews of adverse effects?

Su Golder, MRC Fellow in Health Services Research

Su golder mrc fellow in health services research spg3 york ac uk

  • This research is being undertaken as part of an MRC fellowship

  • The views expressed in this presentation are those of the author and not necessarily those of the MRC

  • CRD is part of the National Institute for Health Research (NIHR) and is a department of the University of York

Introduction fellowship

  • Why adverse effects are important?

  • Why include adverse effects in systematic reviews?

  • Does unpublished adverse effects data differ from published adverse effects data?

  • Should we include unpublished data in systematic reviews of adverse effects?

Why adverse effects are important
Why adverse effects are important fellowship

  • Definition

    • ‘A harmful or undesirable outcome that occurs during or after the use of a drug or intervention for which there is at least reasonable possibility of a causal relation’

      Chou et al 2010

  • Why adverse effects are important

    • Can be serious – hospitalisation, long-term disability, death (USA: 4th to 6th leading cause of death)

    • Quality of life, Compliance

    • Cost (estimates of cost to NHS of £2 billion per year)

Why include harmful effects in systematic reviews
Why include harmful effects in systematic reviews fellowship

  • Considering only benefits leads to bias

    • Need to assess benefit/ harm balance

  • Detailed evaluation of safety needed when:

    • Narrow margin between benefit and harm (aspirin/ CVD)

    • A number of equally effective treatments with different safety profiles

    • When adverse effects cause withdrawal from treatment

Methodological overview of studies on adverse effects
Methodological Overview of Studies on Adverse Effects fellowship

  • Comparisons of

    • Industry funded data versus non-industry funded data

    • Different information sources (such as MEDLINE versus EMBASE)

    • Different search strategies (such as adverse effect related subheadings, textwords and indexing terms)

    • Author affiliation (such as clinician or researcher)

    • Journal impact factor (high versus low)

    • Country and date of study

    • Different study designs (such as RCTs versus observational studies)

    • Unpublished versus published data

Introduction published and unpublished data
Introduction: published and unpublished data fellowship

  • If unpublished data differs systematically from published data, systematic review limited to published data may be biased

  • Significant positive outcomes more likely to be published than nonsignificant outcomes. Hence, systematic reviews based on published literature may overestimate efficacy of an intervention

  • Impact of including only published adverse effects not been clarified

Methods fellowship

  • Search strategy: 10 databases, bibliographies, handsearching key journals, conference proceedings, websources, contacted experts, citation searching

  • Inclusion criteria: compared adverse effects of healthcare intervention according to publication status

  • Analysis:

    • Descriptive comparison for rates and number of cases

    • Ratio of odds ratio where odds ratio or relative risk presented

      “pooled risk ratio for the adverse outcome from unpublished data,’’ divided by ‘‘pooled risk ratio for the adverse outcome from published studies.’’

Results fellowship

Availability of unpublished data
Availability of unpublished data fellowship

  • Case Reports

    • Two out of three studies found more unpublished than published case reports of adverse effects (Bennett et al 2005, Tramer et al 1997, Cosmi et al 2000)

  • Trials

    • A higher percentage of unpublished trials report adverse effects than published trials (Hemminki 1980)

Differences in published versus unpublished data
Differences in published versus unpublished data fellowship

  • Case reports

    • One study of the rank order of adverse effects from published and unpublished case reports found the most frequent adverse effects in published cases to be respiratory and nervous system adverse reactions, and in unpublished cases to be thyroid and skin disorders (Loke et al 2004)

  • Trials

    • Five studies reported the risk ratio (RR) from published and unpublished trials (Hemminki 2000, MacLean et al 2003, Ross et al 1997, Wallace et al 2006, Whittington et al 2004)

Differences in published versus unpublished data1
Differences in published versus unpublished data fellowship

  • Case reports

    • Unpublished case reports will generate a different picture of the relative frequencies of specific adverse effects

  • Trials

    • No clear evidence that data on adverse effects from published and unpublished trials differed

    • Inclusion of unpublished data could provide more precise estimates of adverse effects

Future fellowship

  • More Reviews

    • Increasing number of reviews including adverse effects either as secondary outcome (in addition to effectiveness) or as primary outcome

  • Better Reporting


  • Help and support

    • Cochrane Adverse Effects Methods Group

    • Discussion List

Guidance fellowship

  • CRD’s Guidance

    • Systematic Reviews: CRD’s guidance for undertaking reviews in health care. 2009

  • Cochrane Handbook

    • Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.2 [updated September 2009]. The Cochrane Collaboration, 2009. Available from

Other guidance
Other Guidance fellowship

  • Loke YK, Price D, Herxheimer A. Systematic reviews of adverse effects: framework for a structured approach. BMC Med Res Methodol 2007;7:32

  • Chou R, Aronson N, Atkins D, Ismaila AS, Santaguida P, Smith DH, Whitlock E, Wilt TJ, Moher D. AHRQ series paper 4: assessing harms when comparing medical interventions: AHRQ and the effective health-care program J Clin Epidemiol 2010 May;63(5):502-12

Included papers
Included Papers fellowship

  • Bennett CL, Nebeker JR, Lyons EA, Samore MH, Feldman MD, McKoy JM, et al. The Research on Adverse Drug Events and Reports (RADAR) project. JAMA. 2005:293:2131-40.

  • Cosmi B, Castelvetri C, Milandri M, Rubboli A, Conforti A. The evaluation of rare adverse drug events in Cochrane reviews: the incidence of thrombotic thrombocytopenic purpura after ticlopidine plus aspirin for coronary stenting. 8th Annual Cochrane Colloquium Abstracts, October 2000 Cape Town, Africa.

  • Hemminki E, McPherson K. Value of drug-licensing documents in studying the effect of postmenopausal hormone therapy on cardiovascular disease. Lancet 2000:355:566-9.

  • Loke YK, Derry S, Aronson JK. A comparison of three different sources of data in assessing the frequencies of adverse reactions to amiodarone. Br J Clin Pharm 2004:57:616-21.

  • MacLean CH, Morton SC, Ofman JJ, Roth EA, Shekelle PG, Southern CE-BPC. How useful are unpublished data from the Food and Drug Administration in meta-analysis? J Clin Epidemiol 2003:56:44-51.

  • Ross SD, Kupelnick B, Kumashiro M, Arellano FM, Mohanty N, Allen IE. Risk of serious adverse events in hypertensive patients receiving isradipine: a meta-analysis. J Hum Hypertens 1997:11:743-51.

  • Tramer MR, Moore RA, McQuay HJ. Propofol and bradycardia: causation, frequency and severity. Br J Anaesth 1997:78:642-51.

  • Wallace AE, Neily J, Weeks WB, Friedman MJ. A cumulative meta-analysis of selective serotonin reuptake inhibitors in pediatric depression: did unpublished studies influence the efficacy/safety debate? J Child Adolesc Psychopharmacol 2006:16:37-58.

  • Whittington CJ, Kendall T, Fonagy P, Cottrell D, Cotgrove A, Boddington E. Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data. Lancet 2004:363:1341.

Publications fellowship

  • Golder S, Loke YK, McIntosh HM. Room for improvement? A survey of the methods used in systematic reviews of adverse effects. BMC Medical Research Methodology 2006;27;6:3.

  • Golder S, Loke YK. Is there evidence for biased reporting of published adverse effects data in pharmaceutical industry-funded studies? British Journal of Clinical Pharmacology 2008:66(6)767-773.

  • Golder S, Loke YK. Search strategies to identify information on adverse effects: a systematic review. Journal of the Medical Library Association 2009:97(2):84–92.

  • Golder S, Loke YK, Bland M. Unpublished data can be of value in systematic reviews of adverse effects: methodological overview. Journal of Clinical Epidemiology 2010;63(10):1071-1081.

  • Golder S, Loke YK. Sources of information on adverse effects: a systematic review. Health Information and Libraries Journal 2010;27(3):176-90.

  • Loke YK, Golder SP, Vandenbrouche JP. Comprehensive evaluations of the adverse effects of drugs: importance of appropriate study selection and data sources. Therapeutic Advances in Drug Safety. 2011 [in press]

Industry data
Industry Data fellowship

  • Raw data from industry

    • No evidence that the numbers of adverse effects from industry and non-industry funded studies differed.

  • Interpretation and conclusion of industry studies

    • Bias may be introduced in the interpretation and conclusions of industry funded studies

Sources fellowship


    • MEDLINE retrieved the highest number of relevant references in only four out of thirteen cases:

      • One compared MEDLINE with IPA

      • One compared MEDLINE with reference books

      • Two searched for non-drug interventions


    • In eight out of ten cases searching EMBASE retrieved more relevant references than MEDLINE.

Sources fellowship

  • Derwent Drug File vs EMBASE vs MEDLINE

    • In four out of five cases searching Derwent Drug File retrieved more relevant references than EMBASE or MEDLINE

  • Industry Submissions

    • In two out of four cases data retrieved from industry submissions retrieved the highest number of relevant records and in each case many records were unique

  • International Pharmaceutical Abstracts (IPA)

    • Searching IPA retrieved either the lowest or joint lowest number of relevant records in five out of seven cases

Search strategies
Search strategies fellowship

  • Evaluated generic adverse effects search strategies

    • Badgett et al 1999

      • (ae or co or po or de).fs or CASE REPORT/ and HUMAN/

    • Golder et al 2006

      • (ae or co or de).fs or (safe or safety or side effect* or undesirable effect* or treatment emergent or tolerability or toxicity or adrs or (adverse adj2 (effect or effects or reaction or reactions or event or events or outcome or outcomes)).ti,ab

Search strategies1
Search strategies fellowship

  • Evaluated searches for named adverse effects

    • Wieland et al 2005

      • “human” [MESH] AND journal article [pt] AND breast neoplasms [majr:noexp] AND (risk [mh:noexp] OR risk factors [mh:noexp] OR follow-up studies [mh:noexp] OR odds ratio [mh:noexp])

Search strategies2
Search strategies fellowship

  • Be careful of large numbers of results

    • High sensitivity can be achieved but with low precision

    • Search strategies which have been evaluated, achieved sensitivity 97% to 100%, with precision 0.9% to 2.8% (NNR: 111 to 36)

  • Don’t rely on electronic searches alone

    • Even using generic adverse effects free text and indexing terms and synonyms for named adverse effect likely to miss relevant articles

Other factors
Other factors fellowship

  • Author affiliation

    • Surgeons and those working in screening reported less complications

  • Impact factor

    • High impact factor journals reported higher rates of adverse effects

  • Year of publication

    • No significant difference

  • Country Setting

    • No significant difference

Study designs
Study designs fellowship

  • RCTs less heterogeneous but smaller number of participants

  • Most confidence intervals of the incidence or risk ratio of adverse effects overlap between study designs

  • Most study designs agree, in terms of finding an increase, decrease or no difference, in adverse effects

  • No consistent pattern of reporting between study designs

    • Observational studies may not pose a major threat of bias in the reporting of adverse effects data but caution is still needed