Low priority questions addressed Important outcomes not assessed

1. Reducing waste in deciding what research to doIain ChalmersCoordinator, James Lind InitiativeNIHR Trainees MeetingLeeds, 26 November2013

2. Questions relevant to clinicians & patients? Appropriate design and methods? Accessible full publication? Unbiased and usable report? Low priority questions addressed Important outcomes not assessed Clinicians and patients not involved in setting research agendas Over 50% of studies never published in full Biased under-reporting of studies with disappointing results Over 30% of trial interventions not sufficiently described Over 50% of planned study outcomes not reported Most new research not interpreted in the context of systematic assessment of other relevant evidence Over 50% studies designed without reference to systematic reviews of existing evidence Over 50% of studies fail to take adequate steps to reduce biases, e.g. unconcealed treatment allocation 50% 50% 50% 85% Research waste = over $85 Billion / year

3. Avoidable waste in producing and reporting research evidence. Lancet series, 8 Jan 2014 Questions relevant to users of research? Accessible, full research reports? Unbiased and usable reports? Efficient research regulation and delivery? Appropriate research design, conduct and analysis? Low priority questions addressed Important outcomes not assessed Over 50% studies designed without reference to systematic reviews of existing evidence Over 50% of studies never published in full Biased under-reporting of studies with disappointing results Biased reporting of data within studies Over 30% of trial interventions not sufficiently described Over 50% of planned study outcomes not reported Most new research not interpreted in the context of systematic assessment of other relevant evidence Hyper-regulation of research Inefficient delivery of research Poor re-use of data Failure to promote evaluative research as an integral element of good clinical practice Over 50% of studies fail to take adequate steps to reduce biases Studies with inadequate statistical power Inadequate replication of initial observations Research waste

4. Avoidable waste in deciding what research to do. Paper 1,Lancet series, 2014 Questions relevant to users of research? Accessible, full research reports? Unbiased and usable reports? Efficient research regulation and delivery? Appropriate research design, conduct and analysis? Low priority questions addressed Important outcomes not assessed Over 50% studies designed without reference to systematic reviews of existing evidence Over 50% of studies never published in full Biased under-reporting of studies with disappointing results Biased reporting of data within studies Over 30% of trial interventions not sufficiently described Over 50% of planned study outcomes not reported Most new research not interpreted in the context of systematic assessment of other relevant evidence Hyper-regulation of research Inefficient delivery of research Poor re-use of data Failure to promote evaluative research as an integral element of good clinical practice Over 50% of studies fail to take adequate steps to reduce biases Studies with inadequate statistical power Inadequate replication of initial observations Research waste

5. Iain Chalmers,James Lind Initiative, Oxford, UK Michael B Bracken,Schools of Public Health and Medicine, Yale University, New Haven, USABen Djulbegovic, Moffitt Cancer Centre & Research Institute, University of South Florida, USA Silvio Garattini,Mario Negri Pharmacological Research Institute, Milan, ItalyJonathan Grant,RAND Europe, Cambridge, UKMetin Gulmezoglu,WHO, Geneva, SwitzerlandDavid Howells,National Stroke Research Institute, Melbourne, AustraliaJohn PA Ioannidis,School of Medicine and Department of Statistics, Stanford University, USASandy Oliver, Institute of Education, London, UK

6. 1 Waste resulting from ignoring the needs of potential users of research 2 Waste resulting from ignoring what is already known or already being researched

7. 1 Waste resulting from ignoring the needs of potential users of research

8. Avoidable waste in deciding what research to do. Paper 1,Lancet series, 2014 Questions relevant to users of research? Accessible, full research reports? Unbiased and usable reports? Efficient research regulation and delivery? Appropriate research design, conduct and analysis? Low priority questions addressed Important outcomes not assessed Over 50% studies designed without reference to systematic reviews of existing evidence Over 50% of studies never published in full Biased under-reporting of studies with disappointing results Biased reporting of data within studies Over 30% of trial interventions not sufficiently described Over 50% of planned study outcomes not reported Most new research not interpreted in the context of systematic assessment of other relevant evidence Hyper-regulation of research Inefficient delivery of research Poor re-use of data Failure to promote evaluative research as an integral element of good clinical practice Over 50% of studies fail to take adequate steps to reduce biases Studies with inadequate statistical power Inadequate replication of initial observations Research waste

9. Low priority questions addressed in research on treatments for osteoarthritis of the kneeTallon, Chard and Dieppe. Lancet, 2000.

10. The UK Database of Uncertainties about the Effects of Treatments To publish uncertainties about the effects of treatments which cannot currently be answered by referring to relevant and reliable, up-to-date systematic reviews of existing research evidence To promote Priority Setting Partnerships involving patients and clinicians to identify and promote their shared priorities for therapeutic research

11. Interventions mentioned in research priorities identified by James Lind Alliance patient-clinician Priority Setting Partnerships, and among registered trials, 2003-2012. (Chalmers et al. 2013)

12. Important outcomes not assessed

13. Priority treatment outcome from a survey of patients with rheumatoid arthritis was not pain It was fatigue

15. Alessandro Liberati

16. "Studies that involved patients to a greater extent were more likely to have achieved recruitment targets (χ2 = 4.58, P<0.05), defined as reaching at least 90% of the target.“Ennis L, Wyke T. Impact of patient involvement in mental health research: longitudinal study. Br J Psychiatr Published online ahead of print 12 Sept 2013. doi: 10.1192/bjp.bp.112.119818.

17. 2 Waste resulting from ignoring what is already known or already being researched

18. Avoidable waste in deciding what research to do,Lancet series, 2013 Questions relevant to users of research? Accessible, full research reports? Unbiased and usable reports? Efficient research regulation and delivery? Appropriate research design, conduct and analysis? Low priority questions addressed Important outcomes not assessed Over 50% studies designed without reference to systematic reviews of existing evidence Over 50% of studies never published in full Biased under-reporting of studies with disappointing results Biased reporting of data within studies Over 30% of trial interventions not sufficiently described Over 50% of planned study outcomes not reported Most new research not interpreted in the context of systematic assessment of other relevant evidence Hyper-regulation of research Inefficient delivery of research Poor re-use of data Failure to promote evaluative research as an integral element of good clinical practice Over 50% of studies fail to take adequate steps to reduce biases Studies with inadequate statistical power Inadequate replication of initial observations Research waste

19. "It is essential that existing sources of evidence, especially systematic reviews, are considered carefully prior to undertaking research… Research which duplicates other work unnecessarily, or which is not of sufficient quality to contribute something useful to existing knowledge, is in itself unethical."Department of Health 2001

20. Waste resulting from studies designed without reference to systematic reviews of existing evidence, published and unpublished,and ongoing studies

22. STUDIES IN ANIMALS 20 animal studies: “The results of this review did not show convincing evidence to substantiate the decision to perform trials with nimodipine in large numbers of patients. Stroke 2001;32:2433-8. STUDIES IN HUMANS Horn J, Limburg M. Calcium antagonists for acute ischemic stroke. The Cochrane Database of Systematic Reviews, 2000 “46 trials were identified of which 28 were included (7521 patients). No effect of calcium antagonists on poor outcome at the end of follow-up (OR 1.07, 95% CI 0.97/1.18), or on death at end of follow-up (OR 1.10, 95% CI 0.98/1.24) was found.”

23. Proportion (%) of clinical trials registered by 1999 andpublished by 2007(from Ross et al. PLoS Med 2009;6(9): e1000144). Country Size Phase Funder UK HTA program UK HTA program

25. Sense about Science’s campaign: All trials registered; all trials published If you haven’t already done so, add your support at www.alltrials.net

26. Retrospective cumulative meta-analyses demonstrate waste

27. In use-inspired basic research Sena et al. 2010

28. In pure applied research Cumulative meta-analysis showing effect of antenatal corticosteroids on risk of neonatal death (Sinclair 1995)

29. The Cochrane Collaboration

31. Avoidable waste in deciding what research to do,Lancet series, 2013 Questions relevant to users of research? Accessible, full research reports? Unbiased and usable reports? Efficient research regulation and delivery? Appropriate research design, conduct and analysis? Low priority questions addressed Important outcomes not assessed Over 50% studies designed without reference to systematic reviews of existing evidence Over 50% of studies never published in full Biased under-reporting of studies with disappointing results Biased reporting of data within studies Over 30% of trial interventions not sufficiently described Over 50% of planned study outcomes not reported Most new research not interpreted in the context of systematic assessment of other relevant evidence Hyper-regulation of research Inefficient delivery of research Poor re-use of data Failure to promote evaluative research as an integral element of good clinical practice Over 50% of studies fail to take adequate steps to reduce biases Studies with inadequate statistical power Inadequate replication of initial observations Research waste

32. An example of dividends from well designed pure applied research Ian Roberts and his colleagues did the CRASH trial to address uncertainty about the effects of giving systemic steroids for people with acute traumatic brain injury, a treatment that had been in use for over three decades.

33. A systematic review of existing knowledge Alderson P, Roberts I (1997). BMJ 314:1855-9; and Cochrane Database of Systematic Reviews. The review revealed important uncertainty about whether systemic steroids did more good than harm.

35. Addressing an important uncertaintyBecause the systematic review and a survey of clinical practice had revealed important uncertainty, a large, publicly-funded, multicentre randomized trialwas organised; the trial was registered prospectively; the protocol for the trial was published

36. Lancet 2004;364:1321-28

38. The report of the CRASH trial is exemplary because: it refers to current uncertainty about the effects of a treatment, shown in a systematic review of all the existing evidence, and in variations in clinical practice it notes that the trial was registered and the protocol published prospectively it sets the new results in the context ofan updated systematic review of all of the existing evidence it provides readers with all the evidence neededfor action to prevent thousands of iatrogenic deaths

39. Patients have suffered and died unnecessarily because of:● failure to address important uncertainties about the effects of treatments ● biased under-reporting of research ● failure to review existing evidence systematically when planning new research

40. Alessandro Liberati