How to read a systematic review the fast tool
This presentation is the property of its rightful owner.
Sponsored Links
1 / 49

How to read a Systematic Review: The FAST tool PowerPoint PPT Presentation


  • 139 Views
  • Uploaded on
  • Presentation posted in: General

How to read a Systematic Review: The FAST tool. F ind A ppraise S ynthesise T ransferability. Paul Glasziou Centre for Evidence Based Medicine University of Oxford www.cebm.net. Are RCTs always needed for treatment questions?. Some immediate & dramatic effects don’t need RCTs*

Download Presentation

How to read a Systematic Review: The FAST tool

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


How to read a systematic review the fast tool

How to read a Systematic Review: The FAST tool

Find

Appraise

Synthesise

Transferability

Paul Glasziou

Centre for Evidence Based Medicine

University of Oxford

www.cebm.net


Are rcts always needed for treatment questions

Are RCTs always needed for treatment questions?

  • Some immediate & dramatic effects don’t need RCTs*

  • Example:

  • Child with nasal foreign body

    • Dislodged with Parent Kiss method

    • Case series of success 15/19

      • Botma J Laryngol Otol 2000

* Glasziou, Chalmers, Rawlins, McCulloch BMJ 2007


What do you do

What do you do?

  • For an acutely ill patient, you do a search

  • You find several studies: some show significant results but many others don’t


How to read a systematic review the fast tool

Forest Plot/Blobbogram: of these 17 studies

  • Which is the smallest study?

  • Which is the largest study?

  • How many are statistically significant?

  • Which studies are “large enough”?


How to read a systematic review the fast tool

Of these 17 studies: of streptokinase for MI


How large should the study be

How large should the study be?


What sample size is needed

What sample size is needed?

For disease X the usual mortality rate is 0%

What sample size is needed to detect a reduction in mortality?

  • 100

  • 1,000

  • 100,000

  • 1,000,000


Sample size caf rule 1 the 50 50 rule proportions

Sample Size: Café Rule 1The 50:50 Rule (proportions)

50 events are needed in the control group:

(For an 80% chance of finding a 50% reduction)

Glasziou P, Doll H. Was the study big enough? Two cafe rules. Evid Based Med. 2006;11(3):69-70.


What sample size is needed1

What sample size is needed?

  • There is usually a 12% mortality rate

    • You think your treatment will lower mortality by 50%

  • What sample size is needed?


What sample size is needed2

What sample size is needed?

  • There is usually a 12% mortality rate

    • You think your treatment will lower mortality by 50%

  • What sample size is needed?

  • 12% means

    • 12/100 or 24/200 or 48/400

    • and 50 per 417

  • Control + Treatment Groups = 834 in total


Systematic review or meta analysis

Systematic Review or meta-analysis?

  • A Systematic Review is a review of a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review.

  • Statistical methods (meta-analysis) may or may not be used to analyze and summarize the results of the included studies.


Is the review any good fast appraisal

Is the review any good?FAST appraisal

  • Question – What is the PICO?

  • Finding

    • Did they find most studies?

  • Appraisal

    • Did they select good ones?

  • Synthesis

    • What to they all mean?

  • Transferability of results


Why do i need to check the review

FIND

APPRAISE

SYNTHESISE

TRANSFERABLE

Why do I need to check the review?

Most reviews do not pass minimum criteria

A study of 158 reviews*

  • Only 2 met all 10 criteria

  • Median was only 1 of 10 criteria met

FAST tool = 4 criteria

* McAlister Annals of Intern Med 1999


What it the review question pico

FIND

APPRAISE

SYNTHESISE

TRANSFERABLE

What it the review question (PICO)?

  • Population

  • Intervention

  • Comparison

  • Outcome(s)


Do pedometers increase activity and improve health

Do pedometers increase activity and improve health?

  • Find: what is your search strategy?

    • Databases?

    • Terms?

    • Other methods?

Do yourself then

Get neighbour’s help


F ind did they find all studies

FIND

APPRAISE

SYNTHESISE

TRANSFERABLE

FIND: Did they find all Studies?

  • Check for existing systematic review?

  • Good initial search

    • Terms (text and MeSH)

    • At least 2 Databases: MEDLINE, EMBASE, CINAHL, CCTR, ...

  • Plus a Secondary search

    • Check references of relevant papers & reviews and

    • Find terms (words or MeSH terms) you didn’t use

    • Search again! (snowballing)


Is finding all published studies enough

FIND

APPRAISE

SYNTHESISE

TRANSFERABLE

Is finding all published studies enough?

  • Negative studies less likely to be published than ‘Positive’

  • How does this happen?

  • Follow-up of 737 studies at Johns Hopkins (Dickersin, JAMA, 1992)

    • Positive SUBMITTED more than negative (2.5 times)


Registered vs published studies ovarian cancer chemotherapy single v combined

FIND

APPRAISE

SYNTHESISE

TRANSFERABLE

Registered vs Published StudiesOvarian Cancer chemotherapy: single v combined

Simes, J. Clin Oncol, 86, p1529


Registered vs published studies ovarian cancer chemotherapy single v combined1

FIND

APPRAISE

SYNTHESISE

TRANSFERABLE

Registered vs Published StudiesOvarian Cancer chemotherapy: single v combined

Simes, J. Clin Oncol, 86, p1529


Which are biased which ok

FIND

APPRAISE

SYNTHESISE

TRANSFERABLE

Which are biased? Which OK?

  • All positive studies

  • All studies with more than 100 patients

  • All studies published in BMJ, Lancet, JAMA or NEJM

  • All studies registered studies


Publication bias solution

FIND

APPRAISE

SYNTHESISE

TRANSFERABLE

Publication Bias: Solution

  • All trials registered at inception,

    • The National Clinical Trials Registry: Cancer Trials

    • National Institutes of Health Inventory of Clinical Trials and Studies

    • International Registry of Perinatal Trials

  • Meta-Registry of trial Registries

    • www.controlled-trials.com


  • Appraise select studies

    FIND

    APPRAISE

    SYNTHESISE

    TRANSFERABLE

    APPRAISE & select studies

    Did they select only the good quality studies?


    Miscalculating nnt

    FIND

    APPRAISE

    SYNTHESISE

    TRANSFERABLE

    Miscalculating NNT

    1. Does the death penalty for miscalculation of an NNT discourage future miscalculation?

    2. Should we have the death penalty for miscalculation of NNT?


    Selective criticism of evidence biased appraisal increases polarization

    FIND

    APPRAISE

    SYNTHESISE

    TRANSFERABLE

    Selective Criticism of EvidenceBiased appraisal increases polarization

    Lord et al, J Pers Soc Psy, 1979, p2098


    Selective criticism of evidence

    FIND

    APPRAISE

    SYNTHESISE

    TRANSFERABLE

    Selective Criticism of Evidence

    28 reviewers assessed one “study” results randomly positive or negative

    (Cog Ther Res, 1977, p161-75)


    A ssessment how can you avoid biased selection of studies

    FIND

    APPRAISE

    SYNTHESISE

    TRANSFERABLE

    Assessment: How can you avoid biased selection of studies?

    • Assessment and selection should be:

      Standardized “Objective” OR

      Blinded to Results

    * assessment of quality blind to study outcome


    S ynthesis pooling the results

    FIND

    APPRAISE

    SYNTHESISE

    TRANSFERABLE

    Synthesis: pooling the results


    Meta analysis forest plot

    FIND

    APPRAISE

    SYNTHESISE

    TRANSFERABLE

    Meta-analysis (Forest) plot


    T ransferable use in my patients

    FIND

    APPRAISE

    SYNTHESISE

    TRANSFERABLE

    Transferable? Use in my patients

    Is the AVERAGE effect similar across studies?

    • If NO, then WHY?

      • Study methods (RAMbo - biases)

      • PICO (Patients, Intervention, …)

    • If YES, then 2 questions

      • Effect in different individuals?

      • Which version of treatment?


    Meta analysis forest plot1

    FIND

    APPRAISE

    SYNTHESISE

    TRANSFERABLE

    Meta-analysis (Forest) plot

    Are the results similar across studies? 3 tests

    • Eyeball” test – do they look they same?

    • Test of “Null hypothesis” of no variation (p-value)

    • Proportion of variation not due to chance (I2)


    Are these trials different

    FIND

    APPRAISE

    SYNTHESISE

    TRANSFERABLE

    Are these trials different?


    Conclusion ebm and systematic review

    EBM (quick & dirty)

    Steps

    Ask Question

    Search

    Appraise

    Apply

    Time: 90 seconds

    < 20 articles

    This patient survives!

    Systematic Review

    Steps

    Ask Question

    Search ++++ x 2

    Appraise x 2

    Synthesize

    Apply

    Time: 6 months, team

    < 2,000 articles

    This patient is dead

    ConclusionEBM and Systematic Review

    Find a systematic review!! (and appraise it FAST)


    Pros and cons of systematic reviews

    Pros and cons of systematic reviews

    • Advantages

      • Larger numbers & power

      • Robustness across PICOs

    • Disadvantages

      • May conclude small biases are real effects


    Is the review any good fast appraisal1

    Is the review any good?FAST appraisal

    • Question – What is the PICO?

    • Finding

      • Did they find most studies?

    • Appraisal

      • Did they select good ones?

    • Synthesis

      • What to they all mean?

    • Transferability of results


    How to read a systematic review the fast tool

    Combined results


    Using review results what do i do with my patient

    FIND

    APPRAISE

    SYNTHESISE

    TRANSFERABLE

    Using review results: what do I do with my patient?

    • STUDY: meta-analysis of behavioural interventions for insomnia adults

      • “.. confirms the efficacy of behavioral interventions for person with chronic insomnia.“

    • PROBLEM: No regimens for ‘behavioural intervention’ described

      • Author asked: “what specific treatment regime (or regimes) would you recommend based on your review?”

      • Author response: “It was found that cognitive, behavioral and relaxation therapies all in general lead to similar improvements in sleep outcomes---although cognitive approaches might have been a bit better. The references for these studies are found in the article. “

    Rx

    “Behavioural

    Intervention”


    Summary systematic reviews

    Summary: systematic reviews

    • Advantages

      • Larger numbers & power

      • Robustness across PICOs

    • Disadvantages

      • May conclude small biases are real effects


    The results are studies similar

    The results: Are studies similar?

    • What are the overall results?

    • Similarity of results

      • Heterogeneity statistic

    • Similarity of question (PICO)

      • Your judgement!


    Traditional cf systematic reviews

    Traditional

    Many questions

    No search methods

    No inclusion criteria

    No combining studies

    Systematic

    One question

    Explicit search

    Explicit inclusion criteria

    Combine study results(meta-analysis)

    Traditional cf systematic reviews


    How to read a systematic review the fast tool

    Which are

    (i) statistically significant * and (ii) Clinically significant +?

    (a*+)

    (b+)

    (c*)

    (d)

    Minimum clinical

    Important difference

    No difference


    How to read a systematic review the fast tool

    • Which studies (presented as Odds Ratio):

      • Are not statistically significant?

      • Have < 50 patients in the control arm?


    Sample size cafe rule 2 continuous the 17 sd squared rule

    Sample Size: Cafe Rule 2 - continuousthe 17/(SD squared) Rule

    For continuous outcomes, number per arm is

    17 / (“worthwhile difference” measured in SDs)


    How to read a systematic review the fast tool

    In a small randomized double-blind trial of a new treatment for acute myocardial infarction, the mortality in the treated group was half that in the control group, but the difference was not significant. We can conclude that:

    • The treatment is useless

    • there is no point in continuing to develop the treatment

    • the reduction in mortality is so great that we should introduce the treatment immediately

    • we should keep adding cases to the trial until the Normal test for comparison of two proportions is significant

    • we should carry out a new trial of much greater size


    The best evidence for different types of question

    The best evidence for different types of question


    Has the systematic reviewer done a good job

    Has the systematic reviewer done a good job?


    Are the studies equivalent

    FIND

    APPRAISE

    SYNTHESISE

    TRANSFERABLE

    Are the studies equivalent?

    • Are variations in results between studies consistent with chance?

      (Test of homogeneity: has low power)

    • If NO, then WHY?

      • Variation in study methods (RAMbo - biases)

      • Variation in PICO (Patients, Intervention, …)


    Is bed rest ever helpful a systematic review of trials

    Is bed rest ever helpful?A systematic review of trials*

    • 10 trials of bed rest after spinal puncture

      • no change in headache with bed rest

      • Increase in back pain

    • Protocols in UK neurology units - 80% still recommend bed rest after LPSerpell M, BMJ 1998;316:1709–10

    • …evidence of harm available for 17 years preceding...

    *Allen, Glasziou, Del Mar. Lancet, 1999


  • Login