Penn center for evidence based practice
This presentation is the property of its rightful owner.
Sponsored Links
1 / 73

PENN CENTER FOR EVIDENCE-BASED PRACTICE PowerPoint PPT Presentation


  • 94 Views
  • Uploaded on
  • Presentation posted in: General

PENN CENTER FOR EVIDENCE-BASED PRACTICE. Introduction to Systematic Reviews and Meta-Analyses Craig A Umscheid , MD, MSCE, FACP Assistant Professor of Medicine and Epidemiology Director, Penn Center for Evidence-based Practice Senior Associate Director, ECRI-Penn AHRQ EPC

Download Presentation

PENN CENTER FOR EVIDENCE-BASED PRACTICE

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Penn center for evidence based practice

PENN CENTER FOR EVIDENCE-BASED PRACTICE

Introduction to Systematic Reviews and Meta-Analyses

Craig A Umscheid, MD, MSCE, FACP

Assistant Professor of Medicine and Epidemiology

Director, Penn Center for Evidence-based Practice

Senior Associate Director, ECRI-Penn AHRQ EPC

TEACH Level II Workshop 4

NYAM

August 8th, 2013


Outline

Outline

  • The Case

  • Definitions

  • Steps to Consider when Performing or Appraising a SR/MA

  • Reporting Tool

  • Conclusions


The case

The Case


Setting the stage images of the h1n1 influenza epidemic

Setting the Stage: Images of the H1N1 Influenza Epidemic


2009 h1n1 cases seen in one mexican hospital

2009 H1N1 Cases Seen in One Mexican Hospital

JAMA. 2009;302(17):1880-1887


The case1

The Case

You’re the CMO of a moderate sized hospital in Upstate NY, 100s of miles from a quarternary care hospital, and you hear about the below patient from the Director of your MICU:

  • 38 yo physician admitted to your hospital with fever, cough, SOB, and muscle aches worsening over three days

  • PMH of high cholesterol and seasonal allergies, on a statin for cholesterol

  • SH of 1 drink per day, no tobacco or illicits, 2 kids (3y and 1y), wife, and 2 cats

  • VS: BP 110/70 HR 115 RR 28 O2 sat 90%

  • Multilobar pneumonia on CXR, lab tests diagnostic for H1N1

  • Admitted to the ICU for hypoxemia

  • Despite intubation and the use of salvage therapies, hypoxemia persists and patient deteriorates

  • Within first week of his ICU stay, patient codes and despite heroic efforts by his team he is pronounced dead


Demographics of h1n1 patients in critical care

Demographics of H1N1 Patients in Critical Care

JAMA. 2009;302(17):1880-1887


Demographics of hospitalized pts with h1n1

Demographics of Hospitalized Pts with H1N1

JAMA. 2009;302(17):1880-1887

JAMA. 2009;302(17):1872-1879


Extra corporeal membrane oxygenation ecmo

Extra Corporeal Membrane Oxygenation (ECMO)


The case continued

The Case (continued)

  • Should our moderate-sized hospital invest in an ECMO program?


Should our moderate sized hospital invest in an ecmo program

Should our moderate-sized hospital invest in an ECMO program?

  • Yes

  • No

  • Don’t know, need more information


Definitions

Definitions


Narrative review

“Narrative review”

  • Traditional review by content expert

  • Most common method of summarizing a field

  • Limitations:

    • Lack systematic methods to identify, appraise, and synthesize information

    • Potential for selective inclusion and exclusion of studies to support a position


Original research study

“Original research study”

  • Well designed, well conducted studies can provide unbiased information

  • Limitations:

    • False negative results

    • False positive results

    • Weak external validity


Systematic reviews vs meta analyses

“Systematic reviews” vs. “Meta-analyses”

Meta-analysis

Statistical pooling of the results of individual studies

Aims to produce a single estimate of treatment effect

  • Systematic review

    • Literature review prepared using a systematic approach to minimize bias and random error in identification of information

    • May or may not include a meta-analysis


Systematic review meta analysis

Systematic Review/Meta-analysis

  • Address limitations of narrative reviews:

    • Uses prespecified, explicit methods to:

      • reduce bias

      • allow for replication

      • reveal areas of uncertainty or in need of further research

  • Address limitations of original research studies:

    • Pools data across studies to:

      • give more precise estimates of effect

      • reduce false positive or false negative results

      • improve generalizability

      • explore between-study differences in results

BMJ. Volume 315. November 22, 1997.


Meta analysis medline publications

Meta-analysis Medline publications


Steps to consider when performing or appraising a sr ma

Steps to Consider when Performing or Appraising a SR/MA


Seven key steps for sr ma

Seven Key Steps for SR/MA

  • Define the question

  • Establish eligibility criteria

  • Search literature

  • Extract data

  • Evaluate individual study quality

  • Synthesize data qualitatively and/or quantitatively (MA)

  • Explore heterogeneity


1 define the question

1. Define the question

  • Questions most commonly address therapeutic interventions, harms, and diagnostic tests

  • Type of question can point to type of studies to seek


1 define the question cont

1. Define the question (cont)

Population

  • age range, gender, race, diagnosis, severity class

    Intervention or exposure

  • specific drug or device, doses, frequencies, IV, PO, inpatient, outpatient, management strategies

    Comparison group

  • placebo, usual care, active comparator

    Outcomes

  • process measures, clinical outcomes, cost


Pico for ecmo sr

PICO for ECMO SR

  • Patients

    • Adults

    • Acute respiratory failure from H1N1 pneumonia

    • Any critical care unit in the world

  • Intervention

    • ECMO

  • Comparator

    • Standard Care

  • Outcomes

    • Mortality


Seven key steps for sr ma1

Seven Key Steps for SR/MA

  • Define the question

  • Establish eligibility criteria

  • Search literature

  • Extract data

  • Evaluate individual study quality

  • Synthesize data qualitatively and/or quantitatively (MA)

  • Explore heterogeneity


2 establish eligibility criteria

2. Establish eligibility criteria

  • Study designs

    • RCT, cohort, case-control, cross-sectional

  • Publication status

    • Peer-reviewed publications only

    • Also include abstracts?

  • Language

    • English, other languages?

  • Other

    • Publication year

    • Study size

    • Minimum follow-up


A note on publication bias

A Note on Publication Bias

  • Bias can be introduced in a SR by not including relevant studies:

    • Unpublished (Gray Literature)

    • Published in non-English language

    • Published only in meeting proceedings or as abstracts

    • Published but not identified


Cumulative publication rate for rcts

Cumulative Publication Rate for RCTs

Scherer. CDSR. 2007.


Submission and publication time

Submission and Publication Time

Ioannidis. JAMA. 1998.


Funnel plots assessing publication bias

Funnel Plots: Assessing Publication Bias

No Publication Bias Publication Bias


Funnel plot of effects of homeopathy

Funnel Plot of Effects of Homeopathy

Linde, Lancet, 1997


Funnel plot following trim and fill

Funnel Plot following Trim and Fill


Eligibility criteria for ecmo sr

Eligibility criteria for ECMO SR

  • Only included published studies

  • Studies were retrieved if:

    • reported on the use of ECMO in patients with influenza OR

    • any controlled trial reporting comparisons between those managed with and without ECMO

  • Studies had to report mortality rates

  • Minimum of ten patients in each group

  • No language restrictions


Seven key steps for sr ma2

Seven Key Steps for SR/MA

  • Define the question

  • Establish eligibility criteria

  • Search literature

  • Extract data

  • Evaluate individual study quality

  • Synthesize data qualitatively and/or quantitatively (MA)

  • Explore heterogeneity


3 search literature

3. Search literature

  • Use librarians to help devise search strategy, and filters to limit your search

  • Identify information resources to search

    • Electronic literature databases

    • Reference lists of key papers and review articles

    • Hand searches of high value journals

    • Discussion with experts in the field

    • Contact pharmaceutical or device manufacturers

    • Trial registries

  • Use sampling of resources to increase yield

  • Screen titles, abstracts and full text

    • Single versus two independent reviewers?

    • Automatation?

      • Abstrckr. Available at: http://tuftscaes.org/citation_screening/


Search filters

Search Filters

Umscheid. CID. 2013.


Electronic literature databases

Electronic Literature Databases

  • Medline (Medical Literature Analysis and Retrieval System Online)

    • Premier database of medical research

    • PubMed interface is free

    • Ovid interface supported thru institutional accounts, and allows for more complex searching

  • Embase (ExcerptaMedica Database)

    • Large European database similar in scope and content to Medline

    • Includes many conference proceedings

    • Up to 70% citations in Embase not in Medline

  • Cochrane Controlled Trials Registry

    • Fastest, most reliable method for determining if a controlled trial has been published on any topic


Search strategy for ecmo sr

Search Strategy for ECMO SR


Seven key steps for sr ma3

Seven Key Steps for SR/MA

  • Define the question

  • Establish eligibility criteria

  • Search literature

  • Extract data

  • Evaluate individual study quality

  • Synthesize data qualitatively and/or quantitatively (MA)

  • Explore heterogeneity


4 extract data

4. Extract Data

  • Set priorities for data to extract

    • Anticipate structure and content of final evidence tables

    • Resist temptation to extract everything

  • Establish quality control

    • Dual extraction vs. solo extraction with random checks

  • Consider free electronic resources to help

    • Systematic Review Data Repository. Agency for Healthcare Research and Quality. 2013. Available at: http://srdr.ahrq.gov/.


Data extraction for ecmo sr

Data extraction for ECMO SR

Pulmonary


Seven key steps for sr ma4

Seven Key Steps for SR/MA

  • Define the question

  • Establish eligibility criteria

  • Search literature

  • Extract data

  • Evaluate individual study quality

  • Synthesize data qualitatively and/or quantitatively (MA)

  • Explore heterogeneity


5 evaluate individual study quality

5. Evaluate individual study quality

  • A study’s quality is a measure of its internal validity

  • Biased studies lead to erroneous findings in systematic reviews

  • Decide on how to evaluate quality

    • Use established scales (e.g. Jadad scale for RCTs)

    • Select components most likely to distinguish studies with higher risks of bias from those with lower risks

  • Components versus scales

    • Scales have advantage of providing single summary measure of quality

    • Multiple scales exist, few are validated, and there are differences in items included and weighting of items across scales

    • Component analysis has advantage of assessing specific aspects of quality


Quality scales

Quality scales

Moher et al. Control ClinTrials. 1995.


Evaluating quality of ecmo rcts

Evaluating quality of ECMO RCTs


Study quality continued

Study quality (continued)

  • Decide how quality evaluation will be used

    • To determine eligibility of study for SR

    • To weight studies

    • For subgroup analyses

  • Subgroup analyses based on quality scales or criteria can help determine effect of bias in studies


Examining effect of lwmh on risk of dvt stratified by study quality

Examining Effect of LWMH on Risk of DVT Stratified by Study Quality

Nurmohamed. Lancet. 1992.


Quality of rcts in ecmo sr

Quality of RCTs in ECMO SR


Seven key steps for sr ma5

Seven Key Steps for SR/MA

  • Define the question

  • Establish eligibility criteria

  • Search literature

  • Extract data

  • Evaluate individual study quality

  • Synthesize data qualitatively and/or quantitatively (MA)

  • Explore heterogeneity


6 synthesize data

6. Synthesize data

  • Qualitative synthesis using evidence tables and written evidence summaries

  • Quantitative synthesis using meta-analysis

    • RevMan. Cochrane. 2012. Available at: http://ims.cochrane.org/revman.

    • OpenMeta[Analyst]. 2012. Available at: http://www.cebm.brown.edu/open_meta.


Evidence table for ecmo sr

Evidence table for ECMO SR


Meta analysis

Meta-analysis

  • Estimate a summary measure and its variance for each study

  • Weight each study according to its sample size

    -Studies with more “events” get greater weight

  • Statistically combine or pool results from each study to obtain a weighted average

    -Not just a simple average

  • Assess for statistical heterogeneity


Forest plot of effect of ecmo on mortality in arf

Forest plot of effect of ECMO on mortality in ARF

2009


Assessing heterogeneity

Assessing heterogeneity

  • Heterogeneity

    • Are differences in study-specific estimates statistically significant?

  • Assessment

    • Visual examination of Forest Plot

    • Statistical tests


Efficacy of bcg vaccination

Efficacy of BCG Vaccination

Colditz, JAMA, 1994


Statistical tests of heterogeneity

Statistical tests of heterogeneity

  • Assess whether study-specific estimates are statistically different from one another

  • Q test (i.e. chi squared test)

    • Most common test used

    • Power is generally low with small number of studies

    • Excessive power with large number of studies

  • I squared

    • Percentage of variation across studies that is due to heterogeneity between studies and not chance

    • 0 to 100%

      • >50% is considered moderate to high

  • Importantly, heterogeneity can exist even in absence of statistical significance

BMJ VOLUME 327 6 SEPTEMBER 2003 p557-60


Forest plot of effect of ecmo on mortality in arf1

Forest plot of effect of ECMO on mortality in ARF

2009


Fixed effects vs random effects analyses

Fixed effects vs. Random effects Analyses

  • Fixed Effects

    • Assumes there is one underlying effect of intervention which all individual studies are measuring

    • Assumes any differences between individual study results is a consequence of sampling variation (i.e. chance)

  • Random Effects

    • Assumes any differences between individual study results is a consequence of chance as well as real differences in the effects of the intervention across different studies

    • Offers a more conservative (less precise) pooled estimate

    • If there is significant heterogeneity, should analyze using random effects


Forest plot of effect of ecmo on mortality in arf2

Forest plot of effect of ECMO on mortality in ARF

2009


Seven key steps for sr ma6

Seven Key Steps for SR/MA

  • Define the question

  • Establish eligibility criteria

  • Search literature

  • Extract data

  • Evaluate individual study quality

  • Synthesize data qualitatively and/or quantitatively (MA)

  • Explore heterogeneity


7 explore heterogeneity between studies

7. Explore heterogeneity between studies

  • Differences in study design, populations, intervention, comparator, outcome definitions and conduct of trials can lead to differing results between studies

  • Examination of differences can help one understand the conditions most likely to yield positive or negative effects from an intervention


  • Meta analysis of azt on mortality

    Meta-analysis of AZT on Mortality

    Search identified 8 studies

    7 studies: no effect

    1 study: large effect

    Why do you think 1 study was not pooled with the others?


    Forest plot of effect of ecmo on mortality in arf3

    Forest plot of effect of ECMO on mortality in ARF

    2009


    Study heterogeneity

    Study heterogeneity

    • If studies are heterogenous:

      • “Don’t ignore, instead explore!”

    • Exploration of heterogeneity is a critical objective of meta-analysis


    Seven key steps for sr ma7

    Seven Key Steps for SR/MA

    • Define the question

    • Establish eligibility criteria

    • Search literature

    • Extract data

    • Evaluate individual study quality

    • Synthesize data qualitatively and/or quantitatively (MA)

    • Explore heterogeneity


    Reporting tool for a sr and ma

    Reporting Tool for a SR and MA


    Back to our case

    Back to Our Case

    • Should our moderate-sized hospital invest in an ECMO program?


    Should our moderate sized hospital invest in an ecmo program1

    Should our moderate-sized hospital invest in an ECMO program?

    • Yes

    • No

    • Don’t know, need more information


    Conclusions

    Conclusions

    • SR/MA is scientifically rigorous process of identifying the available evidence on a particular topic

    • SR are both qualitative and quantitative (MA)

    • MA involves statistical pooling of available data that results in weighted average

    • Exploration of heterogeneity in MA is critical

    • Potential limitations of SR/MA include poor quality of identified studies, publication bias, and heterogeneity


    References

    References

    General Systematic Review Resources

    • Umscheid CA. A primer on performing systematic reviews and meta-analyses. Clinical Infectious Diseases. 2013.

    • Egger M, Davey Smith G, Altman D (eds). Systematic Reviews in Health Care. Meta-analysis in Context. London: BMJ Books, 2001.

      Resources for Systematic Reviews Examining Harm

    • Chou R, Helfand M. Challenges in systematic reviews that assess treatment harms. Annals of Internal Medicine. 2005: 142(12): 1090-1099.

    • Loke YK, Price D, Herxheimer A. Systematic reviews of adverse effects: framework for a structured approach. BMC Medical Research Methodology. 2007. 7:32.

    • Hammad TA, Pinheiro SP, Neyarapally GA. Secondary use of RCTs to evaluate drug safety: a review of methodologic considerations. Clinical Trials. 2011.

      Resource for Systematic Reviews Examining Diagnostic Tests

    • Smetana GW, Umscheid CA, Chang S, Matchar DB. Methods guide for authors of systematic reviews of medical tests: a collaboration between the Agency for Healthcare Research and Quality and the Journal of General Internal Medicine. J Gen Intern Med. 2012;27: Suppl 1:S1-3.


    Penn center for evidence based practice

    http://www.uphs.upenn.edu/cep/

    My email: [email protected]


  • Login