Keeping up with the evidence:

1. Keeping up with the evidence: Systematic searching vs literature surveillance (or Why Librarians Do it Better…) Sarah Greenley Information Specialist BMJ Knowledge, London, UK (With thanks to Mick Arber and Alex McNeil)

2. Overview of presentation • Background to project • Methods of evidence-based resources • Literature review of existing research • Clinical Evidence (CE) search and appraisal • BMJ Updates • Phase I: 10 CE topics • Phase II: 217 topics • Implications of missing studies

3. Pre-appraised resources • Traditional “full blown” EBM impractical for many health professionals • Move towards “pre-appraised” evidence • Searching, critical appraisal, summary of evidence performed by others • Examination of methods necessary

4. Methods of Evidence-based resources Not all “evidence-based” resources are the same • Systematic searches and explicit selection criteria for initial search and updates e.g. Clinical Evidence, Cochrane systematic reviews • Systematic literature surveillance for updating– defined set of journals/resources e.g. UptoDate, Dynamed, Infopoems, EB journals • Collection of other resources e.g. TRIP, Sumsearch • Citing references

5. Literature review • Examination of methods of EB resources • Search on Medline, Embase, The Cochrane Library, additional web-based resources • “Secondary journals/literature” “literature surveillance/scanning” • Studies comparing yield of search, time to appraise, speed of incorporation of evidence, missing data

6. Challenge of updating: systematic searching • Databases index thousands of journals, Medline c4500, Embase c6500 • Includes foreign language material • Studies unlikely to be missed • Time consuming to incorporate new evidence, costly • Indexing delays? • Authoring, editing, peer review

7. Challenge of updating: surveillance • Scanning defined list of resources 85-500 • Newly published material incorporated quickly • “Added value” of commentaries, ratings • Possibility of bias: reporting positive trials • Language restrictions, missing information • Missed studies?

8. CE information specialists Scoping searches Topic planning Foundation literature searches Systematic annual update searches Critical appraisal of all search results Checking of author inclusion/exclusion forms

9. Time • Ear wax: 18 update refs • Secondary prevention of MI: 5285 • Additional searches: adverse events, observational studies, new comparisons, new options • Ordering papers, editing, production, peer review • Subsequently published studies – is there a better way to identify them?

10. BMJ Updates • McMaster University • Alerting services • Over 110 premier clinical journals • Articles rated for quality and relevance • Searchable database from 2002 • Also basis for EB Journals

11. Phase 1 10 CE topics Compare time to search Compare time to appraise Compare yield of results Phase 2 All CE topics Highlight missing studies Examine implications Our study: using BMJupdates vs searching

12. Sickle cell Hernia Chickenpox Obesity Prostatis Glaucoma Tuberculosis Gout Acne Halitosis Volume of last CE annual update (retrospective checking of search assessment form) Volume of BMJUpdates published in same period (Search on BMJupdates database limited to therapy/prevention) Phase 1

14. Key issues • Surveillance ongoing not retrospective • BMJUpdates found 1 extra study in chickenpox – delay on database searching • CE update search: 237selected results compared with 18 for BMJUpdates • Some topics yielded no BMJUpdates results • Only around 10% of CE results selected for authors • CE search and appraisal estimate for TB=18 hours • BMJ Updates search and appraisal estimate for TB=3 hours

15. Phase II objectives Check every reference cited in CE (excluding background references) against list of 110 BMJUpdates journals How many studies would have been missed in total? How many studies would have been missed for updates? Did missing studies change CE categorisation?

16. Phase II methods • Print out each CE chapter • Exclude background references • Check cited reference against BMJUpdates list of journals • Identify new references (substantive changes) • Identify changes in categorisation

17. Total number of topics checked:217 Total number of references checked:6794 123 topics with new evidence Results tabulated on Excel spreadsheet Phase II

18. Phase II results • 60% of all references were missing • 54% of “new evidence” references missing • 27 topics where all “new evidence” references were on BMJUpdates journal list • 32 topics where 100% of “new evidence” references were not on BMJUpdates journal list

19. Best BMJUpdates coverage Section %of refs missing Diabetes 28% CVD 31% Sleep 33% Pregnancy 40% Musculoskeletal 47% Respiratory 47%

20. Worst BMJUpdates coverage Section % refs missing ENT 71 Poisoning 71 Cancer 72 Skin 78 Kidney 81 Men’s health 83 Oral 84 Eye 86

21. Changes in categorisation

22. Across 16 updated topics 30 instances where “missing” references led to change in categorisation “Taxane based combination chemotherapy One systematic review and three RCTs added; categorisation changed from Likely to be beneficial to Trade off between benefits and harms.” Categorisation changes

23. English language bias • Exclusion of foreign language material can be important. • Translated Turkish trial changed categorisation in Chronic Suppurative Otitis Media • German language ENT topics

24. Conclusions • Literature surveillance is a useful method to stay aware of new evidence published in major journals in a digest format • Limited scope of their surveillance means not suitable for full updating of evidence-based resources but could be useful “extra” • Evidence-based resources should use explicit and reproducible methods

25. Contact information Sarah Greenley sgreenley@bmjgroup.com