systematic reviews and meta analyses l.
Skip this Video
Loading SlideShow in 5 Seconds..
Systematic Reviews and Meta-Analyses PowerPoint Presentation
Download Presentation
Systematic Reviews and Meta-Analyses

Loading in 2 Seconds...

play fullscreen
1 / 71

Systematic Reviews and Meta-Analyses - PowerPoint PPT Presentation

  • Uploaded on

Systematic Reviews and Meta-Analyses. And the Cochrane Library. Ritz Kakuma, MSc (PhD Candidate) Department of Epidemiology & Biostatistics McGill University. Outline. Background on SR/MA Cochrane Collaboration Cochrane Library Example on Mammography Screening for Breast Cancer.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Systematic Reviews and Meta-Analyses' - LionelDale

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
systematic reviews and meta analyses
Systematic Reviews and Meta-Analyses

And the

Cochrane Library

Ritz Kakuma, MSc (PhD Candidate)Department of Epidemiology & Biostatistics McGill University

  • Background on SR/MA
  • Cochrane Collaboration
  • Cochrane Library
  • Example on Mammography Screening for Breast Cancer
problems with today s medical literature
Problems with Today's Medical Literature
  • Most studies are too small
    • Inconclusive, often conflicting results
  • Traditional reviews are unstructured & subjective
    • Biased conclusions
who can keep up
  • Over 425,000 trials published to date
  • Over 20,000 new trials published annually


for general physicians to keep current
For General Physiciansto keep current:

Read 19 new articles per day which appear in medical journals

19 x 2 hrs (Critical Appraisal) = 38 hrs per day

Davidoff F et al. (1995)

EBM; A new journal to help doctors identify

the information they need. BMJ 310:1085-86.

2002 g welch s gabbe statistics usage in the amer j obstet gynecol has anything changed 180 584 6
2002 G. WELCH, S. GABBEStatistics Usage in the Amer J Obstet Gynecol: has anything changed? 180;584-6

Statistical Quality of Medical Research getting better…

  • All clinical papers: Jan - June 1994 (Vol. 170, No. 1 to 6) vs. Jan – June 1999 (Vol. 180, No. 1 to 6)
  • Inappropriate statistics used in 31.7% (46/145) in 1994 and 9.8% (19/195) in 1999.


problems with standard reviews
Problems with Standard Reviews
  • Lack of scientific purpose (question)
  • Undocumented methods of literature search
  • Unstated criteria for selecting studies
  • No methodological assessment of selected studies
  • Inadequate assessment of inter-study differences in results
  • No attempt at quantitative synthesis (pooling) to take advantage of increased power
why systematic reviews
Why Systematic Reviews?
  • Help to deal with the volume of literature
  • Help resolve conflicting results
  • Scientific rather than subjective summarization of literature
  • Can reveal new evidence
  • Identify knowledge gaps
  • More reliable evidence with which to aid decision making
  • Guide clinical research by providing new hypotheses
the cochrane collaboration origins
The Cochrane Collaboration - origins
  • Archie Cochrane
    • “It is surely a great criticism of our profession that we have not organized a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials.”
  • Pregnancy and childbirth work - 1980s
  • Founded 1993
aims and objectives of the cc

Canadian Cochrane Network & Centre

Aims and objectives of the CC

“The Cochrane Collaboration is an international organization that aims to help people make well­informed decisions about healthcare by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of healthcare interventions”

cc built on 10 principles

Canadian Cochrane Network & Centre

CC built on 10 Principles
  • collaboration
  • building on the enthusiasm of individuals
  • avoiding duplication
  • minimizing bias
  • keeping up to date
  • striving for relevance
  • promoting access
  • ensuring quality
  • continuity
  • enabling wide participation
organization of the cc

Canadian Cochrane Network & Centre

Organization of the CC
  • Cochrane Centres (n=12)
  • Collaborative Review groups (n=51)
  • Fields (n=11)
  • Networks (n=1)
  • Methods working groups (n=10)
cochrane centres

Canadian Cochrane Network & Centre

Cochrane Centres
  • Australasian CC
  • Brazilian CC
  • Canadian CC
  • Chinese CC
  • Dutch CC
  • German CC
  • Iberoamerican CC
  • Italian CC
  • Nordic CC
  • South African CC
  • UK CC
  • US CC (Rhode Island, Boston, San Francisco Branches)
cochrane review groups

Canadian Cochrane Network & Centre

Cochrane Review Groups

Infectious Diseases Group

Inflammatory Bowel Disease Group

Injuries Group

Lung Cancer Group

Menstrual Disorders and Subfertility Group

Metabolic and Endocrine Disorders Group

Methodology Review Group

Movement Disorders Group

Multiple Sclerosis Group

Musculoskeletal Group

Musculoskeletal Injuries Group

Neonatal Group

Neuromuscular Disease Group

Oral Health Group

Pain, Palliative and Supportive Care

Peripheral Vascular Diseases Group

Pregnancy and Childbirth Group

Prostatic Diseases and Urologic Cancers Group

Renal Group

Schizophrenia Group

Sexually Transmitted Diseases Group

Skin Group

Stroke Group

Subfertility Group (see Menstrual Disorders)

Tobacco Addiction Group

Upper Gastrointestinal & Pancreatic Diseases Grp

Wounds Group

Acute Respiratory Infections Group

Airways Group

Anaesthesia Group

Back Group

Breast Cancer Group

Colorectal Cancer Group

Consumers and Communication Group

Cystic Fibrosis and Genetic Disorders Group

Dementia and Cognitive Improvement Group

Depression, Anxiety and Neurosis Group

Developmental, Psychosocial & Learning Problems Group

Drugs and Alcohol Group

Ear, Nose and Throat Disorders Group

Effective Practice and Organisation of Care Group

Epilepsy Group

Eyes and Vision Group

Fertility Regulation Group

Gynaecological Cancer Group

Haematological Malignancies Group

Heart Group

Hepato-Biliary Group


Hypertension Group

Incontinence Group

cochrane fields network

Canadian Cochrane Network & Centre

Cochrane Fields & Network
  • Cancer Network Field
  • Child Health Field
  • Complementary Medicine Field
  • Health Care of Older People Field
  • Health Promotion and Public Health Field
  • Neurological Network Field
  • Occupational Health Field
  • Prehospital & Emergency Health Field
  • Primary Health Care Field
  • Rehabilitation and Related Therapies Field
  • Vaccines Field
  • Consumer Network
cochrane methods groups

Canadian Cochrane Network & Centre

Cochrane Methods Groups
  • Applicability and Recommendations
  • Health Economics
  • Health-Related Quality of Life
  • Individual Patient Data Meta-Analyses
  • Non-randomised Studies
  • Prospective Meta-Analysis
  • Qualitative Methods
  • Reporting Bias Methods
  • Screening and Diagnostic Tests
  • Statistical Methods
cochrane activities

Canadian Cochrane Network & Centre

Cochrane Activities
  • Produce and update systematic reviews
  • Hand search for RCTs
  • Medline (and others) enhancement
  • Review methodology
the cochrane library

Canadian Cochrane Network & Centre

The Cochrane Library
  • unique source of reliable and up-to-date information on the effects of interventions in health care
  • To provide information and evidence to support decisions taken in health care and to inform those receiving care
  • Published on a quarterly basis
when should you use the clib

Canadian Cochrane Network & Centre

When should you use the CLIB?

For questions on effectiveness

  • What is the effectiveness of treatment x
  • What is an effective treatment for y
  • Is z effective in treating y
  • Is z better than x at treating y
when not to use the cochrane library

Canadian Cochrane Network & Centre

When not to use the Cochrane Library

General healthcare questions

  • causal, prognosis, epidemiology, etc.
  • Statistics (prevalence and incidence)
  • Primary research other than RCTs
  • Guidelines
  • Current research
the cochrane database of systematic reviews cdsr

Canadian Cochrane Network & Centre

The Cochrane Database of Systematic Reviews (CDSR)

Complete Reviews

  • full text, regularly updated systematic reviews of the effects of health care (2,170 reviews)
  • prepared and maintained by the Collaboration Review Groups


  • protocols of reviews currently being prepared, incl. expected date of completion (1,500 protocols)
  • includes background, objectives and methods sections
the database of abstracts of reviews of effectiveness dare

Canadian Cochrane Network & Centre

The Database of Abstracts of Reviews of Effectiveness (DARE)

prepared by the National Health Services Centre for Reviews and Dissemination at the University of York, UK.

Abstracts of quality assessed systematic reviews

  • structured abstracts assessing the quality of previously published SRs & summarizing findings (4,118 reviews)

Other reviews: bibliographic details only

  • references to published SRs NOT assessed for quality (800 reviews)
the cochrane central register of controlled trials central

Canadian Cochrane Network & Centre

The Cochrane Central Register of Controlled Trials (CENTRAL)


  • Reference list of ALL identified published randomized trials
  • Latest issue 427,807 RCTs

(Medline: 97,827 articles identified as RCT publication type)

the cochrane database of methodology reviews cdmr

Canadian Cochrane Network & Centre

The Cochrane Database of Methodology Reviews (CDMR)

Complete Reviews

  • Full-text SRs of methodological studies (10 reviews)
  • Highly structured and systematic, covering a specific and well-defined area of methodology
  • prepared and maintained by the Cochrane Methodology Review Groups


  • protocols of reviews currently being prepared, incl. expected date of completion (8 protocols)
  • includes background, objectives and methods sections
cochrane methodology register cmr

Canadian Cochrane Network & Centre

Cochrane Methodology Register (CMR)


  • published reports of empirical studies of methods used in reviews (5,968 reports)
  • methodological studies directly relevant to conducting a review
health technology assessment database hta

Canadian Cochrane Network & Centre

Health Technology Assessment Database (HTA)


  • HTA covers prevention and rehabilitation, vaccines, pharmaceuticals and devices, medical and surgical procedures and the systems within which health is protected and maintained
  • Ongoing projects and completed publications from HTA organizations (4,395 citations)
nhs economic evaluation database nhs eed

Canadian Cochrane Network & Centre

NHS Economic Evaluation Database (NHS EED)


  • Structured abstracts of articles reporting economic evaluations of health care interventions
  • Bibliographic details of articles on relevant topics (i.e., burden of illness, economic methodology, reviews of economic evaluations)
  • N=15,041 citations
searching the clib

Canadian Cochrane Network & Centre

Searching the CLIB

Some basics…

ALL contents of ALL records in ALL databases are searched

Ignores: punctuation & numbers

Boolean terms: AND, OR, NEXT, NEAR – within 6 words both ways, NOT

Restricting searches

  • ‘Options’ page and choose desired restrictions
  • At the end of the term, add:

:AU – Author :TI – Title :ME - MeSH terms

:AB – Abstract :KY - Keywords

searching the clib29

Canadian Cochrane Network & Centre

Searching the CLIB


  • Keywords drawn from MeSH thesaurus of U.S. NLM
  • Accompanies some but not all records
  • Organized hierarchically in ‘trees’
  • Permuted Index – an index of all words that appear in the MeSH thesaurus  used to located specific MeSH terms
at mcgill university and affiliated sites

Canadian Cochrane Network & Centre

At McGill University and affiliated sites:

The pathway to the CLIB is:

McGill HSL library homepage > Databases > > Cochrane Library > web

(no login or password nec.)

useful materials

Canadian Cochrane Network & Centre

Useful Materials
  • The Cochrane Library: Self Training Guide
  • Interpretation of Odd-ratio diagrams
  • WWW links, HTA database
  • Cochrane Handbook
  • User manual
  • Glossary, etc...


quality of cochrane reviews
Quality of Cochrane Reviews
  • Comparison of 36 Cochrane reviews with 39 paper-based journals  Cochrane reviews less prone to bias (Jadad et al. 1998)
    • Explicit reporting of eligibility criteria (35/36 vs 18/39)
    • Assessed trial quality (36/36 vs 7/39)
    • No language restriction (36/36 vs 32/39)
quality of cochrane reviews43
Quality of Cochrane Reviews
  • Also not perfect (Olsen et al. 2001)
  • 52 Cochrane reviews from 1998
  • 18% - conclusions not backed up by evidence
    • All overestimated effect of intervention
discordant reviews
Discordant Reviews
  • Direction of effect
  • Significance
  • Magnitude of effect
  • Interpretation of results

 Numerous possible reasons

sources of discordance
Sources of discordance
  • Different research question
    • Target population
    • Intervention being studied
    • Outcome measures
    • Setting
  • Study selection
    • Search strategy
    • Eligibility criteria
sources of discordance46
Sources of discordance
  • Data extraction
    • Methods of measuring outcomes and endpoints
    • Extent of human error
  • Quality assessment method
    • 25 scales and 9 checklists avail for assessing RCT quality (Moher et al. 1996)
      • Inconsistent quality depending on instrument used
    • Additional 8 instruments (Juni et al. 1999)
    • QUOROM (Quality of Reporting of Meta-Analyses) = most comprehensive(Moher et al. 1999, Shea et al. 2001)
sources of discordance47
Sources of discordance
  • Analysis
    • Appropriateness of combining results
    • Method of synthesis: Descriptive, meta-analysis
    • Statistical methods
      • Bayesian
      • Meta-regression
      • Frequentist
  • Interpretation of evidence
decision algorithm for interpreting discordant reviews
Decision Algorithm for interpreting discordant reviews

Jadad, Cook & Browman. A guide to interpreting discordant systematic reviews. CMAJ. 1997 May 15;156(10):1411-6.


Screening for breast cancer with mammography among women aged 50-69 yearsReport prepared for the Breast Cancer Screening Unit,Cancer Branch, Health CanadaRitsuko KakumaMarch 2002

srs on mammography screening for breast cancer
SRs on mammography screening for breast cancer
  • Fletcher et al. Report of the International Workshop on Screening for Breast Cancer. J Natl Cancer Inst 1993; 85:1644-1656.
  • Kerlikowske et al. Efficacy of screening mammography. A meta-analysis. JAMA 1995; 273:149-154.
  • Olsen O & Gøtzsche PC. Is screening for breast cancer with mammography justifiable? Lancet 2000.
fletcher et al review
Fletcher et al. review

Research Question:

  • To assess current state of knowledge about BRCA screening, identify knowledge gaps

Study Sample:

  • Women aged 40-79 years stratified into 3 groups: 40-49, 50-69, 70+
fletcher et al review52
Fletcher et al. review

Methodological issues:

  • Int’l Workshop on screening for BRCA in Feb 1993 (by NCI) – characteristics of participants?
  • Search strategy not reported
  • Study selection method not described
  • Data source: published and unpublished data provided by the w/s participants
  • Validity of trials assessed:
    • Randomization, Compliance, Contamination
    • Quality and frequency of screening test
    • Adequacy of FU
    • Validity of outcome assessment
    • Generalizability of results
fletcher et al review53
Fletcher et al. review


  • 7 trials identified
  • For age 50-69 age group, only 5 trials provided data
    • New York, Two-County, Malmo, Edinburgh and Canada
  • No statistical pooling of data
  • Together, Fletcher et al. concluded that there was substantial benefit of screening on BRCA mortality  reduction of 30% after 10-12 years of FU
kerlikowske et al review
Kerlikowske et al. review

Research Question:

  • To determine efficacy of screening in reducing BRCA mortality by age, # MM views per screen, screening interval and duration of FU


  • Women aged 40-74 years stratified into 3 groups: 40-49, 40-74, 50-74
kerlikowske et al review55
Kerlikowske et al. review


  • Search: MEDLINE, manual (reference lists) and consultations with colleagues and experts
    • Search terms, dates covered
  • Study Selection
    • English language restriction, explicit inclusion criteria
    • Not restricted to RCTs
  • Data extraction
    • 2 indep extractors & 3rd for resolving discrepancies
    • Published and unpublished data used
  • Quality assessment not reported..not done?
  • Analysis
    • Pooled data of all included trials using fixed effects
kerlikowske et al review56
Kerlikowske et al. review

Results / Conclusions:

  • 9 studies (7 RCTs & 2 CCS) have data on 50-74 year age group
kerlikowske et al review57
Kerlikowske et al. review

Results / Conclusions:

  • All studies – beneficial effect of screening (4 statistically significant)
  • All trials included in the analysis (not CCSs)
  • MM screening reduced BRCA mortality by 27% in this age group after 7- 9 years and 24% after 10-12 years regardless of # MM views, screening interval, duration of screening or addition of CBE
olsen g tzsche review
Olsen & Gøtzsche review

Research Question:

  • To assess effect of MM screening for BRCA on mortality and morbidity
  • Review methodological quality of trials – focus on 3 most important sources of bias
    • Randomization
    • blind outcome assessment
    • post-Rx exclusions


  • Women without previously diagnosed BRCA on women aged <50 years and 50+ years
olsen g tzsche review59
Olsen & Gøtzsche review


  • Search: MEDLINE, common author names search, reference lists, specialized trials registers, letters, abstracts, grey literature
    • Search terms, dates covered
  • Study Selection:
    • Explicit inclusion criteria (RCT / quasi-RCT on MM screening vs. no MM screening)
  • Two indep reviewers for study selection, quality assessment & data extraction
    • Authors of studies contacted for missing information
  • Quality assessment: High, Medium, Poor or Flawed
    • Rx process, baseline comparability, post-Rx exclusions, Consistency of number of women randomized
  • Analysis
    • Intent-to-Treat analysis, with fixed effect unless heterogeneous
    • Stratified by quality (excluded flawed)
olsen g tzsche review60
Olsen & Gøtzsche review

Results / Conclusions:

  • 7 trials identified, none with high quality (2 medium, 3 poor, 2 flawed)
  • Total mortality
    • no impact of screening after 7- and 13 yrs FU for both medium and poor quality trials
  • BRCA mortality
    • no beneficial effect of screening in medium quality trials
    • Beneficial effect among POOR quality trials pooled RR=0.69, CI 0.55-0.80 at 7 yearspooled RR=0.64, CI 0.54-0.76 at 13 years
  • Interventions for screened group higher
    • Over-diagnosis, increased use of more aggressive treatment such as mastectomies and tumorectomies (20-30%)
identified trials
Identified Trials
  • Sampling Randomization: Canadian study recruited volunteers, other used population-based lists
  • Assessment of pre-existing BRCA: pre- vs. post-Rx
impact on effect measure
Impact on effect measure
  • Contamination & Compliance : underestimate
    • Introduction of screening in controls
    • Screened vs. invited to be screened
  • Post-Rx exclusions: overestimate
    • Identification of BRCA at baseline more complete for screened group..first round of screening used to exclude
    • Dropouts and losses: equally distributed?
  • Adequacy of Randomization: overestimate
    • Typically overestimate treatment effect by 30-40%(Schulz et al. 1995; Moher et al. 1998; Kjaergard et al. 2001)
  • Outcome Assessment: overestimate
    • Misclassification
    • Determination of cause of death may be difficult in light of multiple illnesses
comparability of trials
Comparability of Trials
  • Same research question?
    • Effectiveness vs. efficacy
  • Same Target population?
  • Appropriate study sample?
  • Intervention?
    • Screening modality, frequency of screening
  • Outcome assessment
    • Cause of death identified the same way?
  • Analytical methods
fletcher et al
Fletcher et al.
  • No precise research question
  • Inadequate description of methods used
    • Search strategy, trial selection, data extraction
  • Acknowledgement of difference in quality of trials but conclude based on ALL evidence
  • Source of some data unclear
  • Quality assessment carried out
  • NOT synthesizing data = appropriate
  • But evidence does not support their conclusion that BRCA mortality is reduced by 30% as a result of screening
kerlikowske et al
Kerlikowske et al.
  • Clear research question
  • Explicit reporting of methods used
  • No indication that quality assessment was done
  • Some numbers extracted don’t match reference cited
  • Missing key articles for trials (I.e., original report)
  • Despite good reporting of methods, caution needed in interpreting results because of the questionable nature of data used
olsen and gotzsche
Olsen and Gotzsche
  • Clear research question
  • Explicit reporting of methods used
  • Strong methodology
  • Addressed important question:Reduced BRCA mortality not necessarily equal to improved survival
  • Did not assess baseline comparability stratified by age groups, which is how the data was analyzed
  • Most rigorous and properly done review of the 3
  • Brought potential adverse effects of screening to forefront
conclusions of assessment
Conclusions of assessment:
  • All existing evidence has methodological issues
  • No conclusive evidence exists to support MM
  • Other issues:
    • Accuracy of MM (false positives?)
    • Detection of slow-growing tumors that may never develop into invasive cancers
    • Psychological impact of positive results
  • US and Canada continue to recommend MM screening for women in this age group
overall conclusions
Overall conclusions
  • MA not appropriate in many cases
  • Must be able to identify the potential sources of bias in both the systematic review as well as the studies included in the review
  • Cochrane reviews are not always perfect but generally better than non-Cochrane reviews