Clinical Epidemiology Boot Camp: Systematic Reviews - PowerPoint PPT Presentation

belden
clinical epidemiology boot camp systematic reviews n.
Skip this Video
Loading SlideShow in 5 Seconds..
Clinical Epidemiology Boot Camp: Systematic Reviews PowerPoint Presentation
Download Presentation
Clinical Epidemiology Boot Camp: Systematic Reviews

play fullscreen
1 / 47
Download Presentation
Clinical Epidemiology Boot Camp: Systematic Reviews
115 Views
Download Presentation

Clinical Epidemiology Boot Camp: Systematic Reviews

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Clinical Epidemiology Boot Camp:Systematic Reviews Selina Liu MD MSc FRCPC Research Fellow Resident Research Career Development Program September 19, 2012

  2. Outline • Introduction – Evidence-Based Medicine (EBM) • Levels of evidence • To discuss the definition of a systematic review • vs. traditional/narrative reviews • The process of conducting a systematic review • Strengths & limitations of systematic reviews • To describe how to critically appraise a systematic review • Example of a systematic review

  3. Evidence-Based Medicine • What is Evidence-Based Medicine? • “…the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients” • “ It’s about integrating individual clinical expertise and the best external evidence” • philosophical origins – date back to mid-19th century Paris (or possibly earlier) Sackett DL et al. BMJ. 1996;312(7023):71-2

  4. Evidence-Based Medicine • Five Steps of Evidence-Based Medicine • 1. Asking Focused Questions • Translation of uncertainty to an answerable question • 2. Finding the Evidence • Systematic retrieval of the best evidence available • 3. Critical Appraisal • Testing evidence for validity, clinical relevance, and applicability • 4. Making a Decision • Application of results in practice • 5. Evaluating Performance • Auditing evidence-based decisions Oxford Centre for Evidence-Based Medicine (CEBM) www.cebm.net

  5. Evidence-Based Medicine • Why Evidence-Based Medicine? • clinical decision making is complex! Mulrow CD, Cook DJ, Davidoff F. Ann Intern Med. 1997;126(5):389-91

  6. Evidence-Based Medicine • How do we practice Evidence-Based Medicine? Can be difficult: • “information overload”  difficult for clinicians to “keep up” with all of the latest evidence • often there are multiple studies examining the same or similar questions • may be of variable quality, generalizability • estimated time required for reading (general medicine): • 19 articles per day, 365 days per year Davidoff F et al. BMJ. 1995;310(6987):1085-6

  7. Evidence-Based Medicine • Weighing the evidence - “Levels of Evidence” OCEBM Levels of Evidence Working Group. “The Oxford 2011 Levels of Evidence”, Oxford Centre for Evidence-Based Medicine. www.cebm.net/index.aspx?o=5653

  8. Evidence-Based Medicine of RCTs Systematic reviews of cohort studies

  9. Systematic Reviews • What is a Systematic Review? • the application of strategies that limit bias in the assembly, critical appraisal, and synthesis of all relevant studies on a specific topic • use rigorous, standardized methods for selecting & assessing articles Oxford Centre for Evidence-Based Medicine www.cebm.net/?o=1116 OR • a report that summarizes all evidence that can be drawn from research (or other sources), that is relevant to a specific clinical question

  10. Systematic Reviews • Systematic Reviews vs. Traditional Review Articles • traditional review articles • written by senior expert in the field, summarizes evidence and recommendations • usually address broad areas/questions (i.e. “management of T2DM”) • often lack structure • may include personal experience/anecdotal evidence Fletcher RH & Fletcher SW 2005. Clinical Epidemiology: The Essentials

  11. Systematic Reviews • Systematic Review vs. Traditional/Narrative Review Cook DJ, Mulrow CD, Haynes RB. Ann Intern Med. 1997;126(5):376-380

  12. Systematic Reviews • Guyatt G et al. 2008. Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition)

  13. Systematic Reviews • Process of Conducting a Systematic Review 1. Define the question 2. Conduct literature search 3. Apply inclusion and exclusion criteria 4. Create data abstraction 5. Conduct analysis • Guyatt G et al. 2008. Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition)

  14. Systematic Reviews - Process • 1. Define the Question • single, focused question • i.e. What is the effect of cinnamon on glycemic control in diabetes? • specify inclusion and exclusion criteria • Population, Intervention or Exposure, Outcome, Methodology • For the systematic review to be useful: • strong studies of the question should be available, but their results should not be so much in agreement that the question is already answered! • there should not be so few studies of the question that each individual study could be fully critiqued directly

  15. Systematic Reviews - Process • 2. Conduct literature search • need to ensure that all of the appropriate studies are included • NOT just a biased sample of studies • decide on information sources • i.e.MEDLINE, recent reviews, textbooks, experts in the field, articles cited by references already found by other approaches, databases of articles, clinical trial registries etc. • identify titles and abstracts

  16. Systematic Reviews - Process • 3. Apply inclusion and exclusion criteria • Apply inclusion and exclusion criteria to titles and abstracts • obtain full articles for eligible titles and abstracts • Apply inclusion and exclusion criteria to full articles • Select final eligible articles • Assess agreement on study selection • of the initial titles and abstracts retrieved, usually only a small proportion of articles are selected

  17. Systematic Reviews - Process • 4. Create data abstraction • Assess methodologic quality of each article • Assess agreement on validity assessment • Data abstraction • Participants • Interventions and Comparison Interventions • Study Design • Results

  18. Systematic Reviews - Process • 5. Conduct analysis • Summarize data • If appropriate: meta-analysis – statistical technique to combine quantitative data • usually combine studies vs. combine patients • Describe results – often graphically • Forest Plot – shows point estimate and confidence interval (for RCTs, observational studies) • Summary Receiver-Operator Curves (for studies of diagnostic tests) • Explore heterogeneity, conduct subgroup analysis (if appropriate) • Explore possibility of publication bias (and other biases)

  19. Systematic Reviews - Process • How to decide if appropriate to perform a meta-analysis? • Two general approaches: • 1. statistical test for homogeneity • BUT – even if fail to reject H0 (i.e. no evidence of a statistically significant difference between studies), usually have high risk of false-negative (saying studies are homogeneous when they really are not) • Limited power - meta-analyses are usually of few number of studies, - affected also by number of patients/study, distribution of patients among studies • 2. informed judgement

  20. Systematic Reviews - Process • Meta-analysis – mathematical models: • Fixed-Effect Model • Assumes that studies are of exactly the same question, so results differ only by chance • Confidence intervals calculated may imply more precision (i.e. are narrower) than in reality (since studies usually differ somewhat) • Random-Effects Model • Assumes that the studies address somewhat different questions, but that they form a closely related family of studies of a similar question • Studies taken to be a random sample of all studies bearing on the question • Produces WIDER confidence intervals (more “realistic”) Fletcher RH & Fletcher SW. 2005. Clinical Epidemiology: the Essentials (4th Edition)

  21. Systematic Reviews – Forest Plot

  22. Systematic Reviews - Bias • Several types of bias: • publication bias • published studies may be systematically different than unpublished studies (“positive” studies vs. “negative” studies?) • language bias • i.e. if only English-language articles are selected • size bias • large studies that result in several publications may be more readily noticed than smaller studies • bias related to funding? • industry-sponsored studies

  23. How to detect publication bias? • Funnel plots – plot effect vs. study size/precision symmetrical, peaked distribution (inverted funnel) • Guyatt G et al. 2008. Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition)

  24. How to detect publication bias? • Funnel plots asymmetrical distribution • Guyatt G et al. 2008. Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition)

  25. Systematic Reviews - Strengths • provide an efficient way to become familiar with the best available research evidence for a focused clinical question • can establish whether results are consistent, generalizable across populations/settings, treatment variations, and whether findings vary by certain subgroups • can extend the available literature (if the review team has obtained unpublished information from the primary authors) • meta-analyses – may provide a more precise estimate of the underlying “true effect” than any individual study Garg AX, Hackam D, Tonelli M. 2008. Clin J Am SocNephrol. 3(1):253-60.

  26. Systematic Reviews - Limitations • summarized results are limited by the quality of the primary studies • “garbage in garbage out” • results dependent on selection of included articles • quality threshold, publication bias, language bias etc. • meta-analyses - may be inappropriate to mathematically combine primary study results if the primary studies differ in design, quality, population, intervention etc. • subjectivity involved in deciding whether to pool or not • subjectivity in interpretation of summarized results (“over-interpretation) Garg AX, Hackam D, Tonelli M. 2008. Clin J Am SocNephrol. 3(1):253-60.

  27. Systematic Reviews – Critical Appraisal Oxman AD, Cook DJ, Guyatt GH, Evidence-Based Medicine Working Group. JAMA. 1994;272(17):1367-71 • Tonelli M, Hackam D, Garg AX. Methods Mol Biol. 2009;473:217-33

  28. Critical Appraisal – Tools Oxford Centre for Evidence-Based Medicine http://www.cebm.net/index.aspx?o=1157

  29. Critical Appraisal – Tools Oxford Centre for Evidence-Based Medicine http://www.cebm.net/index.aspx?o=1157

  30. Critical Appraisal – Tools • AMSTAR – 2007 • Assessment of Multiple SysTemAticReviews • Shea BJ, Grimshaw JM, Wells GA et al. • 11 item tool • developed via exploratory factor analysis of a 37-item assessment tool • Shea BJ, Grimshaw JM, Wells GA et al. BMC Med Res Methodol. 2007;7:10

  31. Critical Appraisal - Tools • Shea BJ, Grimshaw JM, Wells GA et al. BMC Med Res Methodol. 2007;7:10

  32. Reporting Systematic Reviews - Tools • The PRISMA Statement – 2009 • Preferred Reporting Items for Systematic reviews and Meta-Analyses • Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group • PLoS Medicine • Annals of Internal Medicine • BMJ • Journal of Clinical Epidemiology • Open Medicine • International Journal of Surgery

  33. The PRISMA Statement - 2009 • Aim – to help authors improve the reporting of systematic reviews and meta-analyses • **NOT intended to be a quality assessment tool • 27 item checklist • four-phase flow diagram • update and expansion of prior QUOROM statement • QUality Of Reporting Of Meta-analyses - 1996 • focused on reporting of meta-analyses of randomized controlled trials

  34. Table 1. Checklist of items to include when reporting a systematic review or meta-analysis. Moher D, Liberati A, Tetzlaff J, Altman DG, et al. (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(7): e1000097. doi:10.1371/journal.pmed.1000097

  35. Figure 1. Flow of information through the different phases of a systematic review. Moher D, Liberati A, Tetzlaff J, Altman DG, et al. (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(7): e1000097. doi:10.1371/journal.pmed.1000097

  36. Example of a Systematic Review Cochrane Database of Systematic Reviews2012, Issue 9:CD007170

  37. Cinnamon for Diabetes Mellitus • Objective – to evaluate the effects of cinnamon in patients with diabetes mellitus • Selection criteria – all RCTs comparing the effects of orally administered monopreparations of cinnamon (Cinnamomum spp.) to placebo, active medication or no treatment in persons with either type 1 or type 2 diabetes mellitus • Primary outcomes – fasting blood glucose, post-prandial glucose, adverse events • Secondary outcomes – HbA1c, serum insulin, HOMA-IR, HRQoL, morbidity, costs Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

  38. Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

  39. Risk of Bias Summary • Review authors’ judgements about each “risk of bias” item for each included study Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

  40. Risk of Bias Graph • Review authors’ judgements about each “risk of bias” item presented as percentages across all included studies Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

  41. Results – Cinnamon vs. Placebo • Primary outcome – fasting blood glucose Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

  42. Results – Cinnamon vs. Placebo • Primary outcome – adverse events Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

  43. Results – Cinnamon vs. Placebo • Secondary outcome – HbA1c Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

  44. Results • Other primary outcomes: • Post-prandial glucose – 1trial • Other secondary outcomes: • Serum insulin – 2 trials • HOMA-IR – 2 trials • HRQoL, morbidity, costs – no trials Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

  45. Conclusions • Insufficient evidence to support the use of cinnamon for type 1 or type 2 diabetes • Further trials required: • To address methodologic issues in current studies • i.e. allocation concealment, blinding • To include other important endpoints • HRQOL, diabetes complications, cost Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

  46. Useful Resources • Guyatt G, Rennie D, Meade MO, Cook DJ. 2008. Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition). New York NY, McGraw-Hill • available online via Western Libraries • Oxford Centre for Evidence-Based Medicine (CEBM) • www.cebm.net • Cochrane Database of Systematic Reviews • www.thecochranelibrary.com

  47. References • Sackett DL, Rosenberg WMC, Muir Gray JA, Haynes RB, Richardson WS. BMJ. 1996;312(7023):71-2 • Mulrow CD, Cook DJ, Davidoff F. Ann Intern Med. 1997;126(5):389-91 • Davidoff F, Haynes B, Sackett D, Smith R. BMJ. 1995;310(6987):1085-6 • Oxford Centre for Evidence-Based Medicine (CEBM) www.cebm.net • OCEBM Levels of Evidence Working Group. “The Oxford 2011 Levels of Evidence”, Oxford Centre for Evidence-Based Medicine. www.cebm.net/index.aspx?o=5653 • Fletcher RH & Fletcher SW. 2005. Clinical Epidemiology: the Essentials (4thEdition). Baltimore MD, Lippincott Williams & Wilkins • Guyatt G, Rennie D, Meade MO, Cook DJ. 2008. Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition). New York NY, McGraw-Hill • Cook DJ, Mulrow CD, Haynes RB. Ann Intern Med. 1997;126(5):376-380 • Garg AX, Hackam D, Tonelli M. 2008. Clin J Am SocNephrol. 3(1):253-60. • Oxman AD, Cook DJ, Guyatt GH, Evidence-Based Medicine Working Group. JAMA. 1994;272(17):1367-71 • Tonelli M, Hackam D, Garg AX. Methods Mol Biol.2009;473:217-33 • Cochrane Database of Systematic Reviews www.thecochranelibrary.com