evidence based medicine and critical appraisal n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Evidence-Based Medicine and Critical Appraisal PowerPoint Presentation
Download Presentation
Evidence-Based Medicine and Critical Appraisal

Loading in 2 Seconds...

play fullscreen
1 / 55

Evidence-Based Medicine and Critical Appraisal - PowerPoint PPT Presentation


  • 300 Views
  • Uploaded on

Evidence-Based Medicine and Critical Appraisal. Ben Rehman Director London Medicines Information Service Northwick Park Hospital. Introduction. Evidence-Based Medicine What is it? Why do we need it? How do we do it? Critical appraisal—RCTs

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Evidence-Based Medicine and Critical Appraisal' - Samuel


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
evidence based medicine and critical appraisal

Evidence-Based Medicine and Critical Appraisal

Ben Rehman

Director

London Medicines Information Service

Northwick Park Hospital

introduction
Introduction
  • Evidence-Based Medicine
    • What is it?
    • Why do we need it?
    • How do we do it?
      • Critical appraisal—RCTs
      • Brief introduction to other study designs including types of pharmacoeconomic studies
    • How should we use it?
      • At patient level?
      • As part of broader decision making and resource optimisation?
  • Example critical appraisal workshop
evidence based medicine what is it
Evidence-Based Medicine—What is it?
  • the conscientious, systematic, explicit, and judicious use of current best evidence in patient care
  • best research evidence replaces or supports current practice
  • integration of evidence and patient or population values
  • optimising health gain
    • clinical evidence
    • pharmacoeconomics
evidence based medicine why do we need it
Evidence-Based Medicine—Why do we need it?
  • Health professionals face problems ensuring best possible patient care:
    • information overload
    • inadequate traditional sources
    • opinion based approach flawed
    • disparity between experience and up-to-date knowledge
  • Healthcare systems have finite resources
  • But… it’s not a complete panacea!
evidence based medicine how do we do it
Evidence-Based Medicine—How do we do it?
  • Form an answerable question
  • Obtain current best evidence
  • Critically appraise the evidence
  • Integrate appraised evidence with clinical expertise and patient values
  • Evaluate our successes and failures
how do we do ebm 1 form an answerable question
How do we do EBM?1. Form an answerable question
  • Should contain:
    • Intervention you are interested in
    • Patient or population you are interested in (clearly for drug interventions you require the indication here too)
    • Outcomes you are interested in
  • So for the example paper, our original question might have been:
    • Is adalimumab effective in patients with Crohn’s disease previously intolerant to or refractory to infliximab?
how do we do ebm 2 obtain current best evidence
How do we do EBM?2. Obtain current best evidence
  • Group discussion
    • What are good sources of evidence?
      • What have you used in the past and found useful?
    • What makes them good?
      • What criteria do you apply?
    • How can we find good evidence?
      • What portals and sources do we have available?
how do we do ebm 2 obtain current best evidence1
How do we do EBM?2. Obtain current best evidence
  • Discussion point: good sources of evidence for EBM
how do we do ebm 2 obtain current best evidence2
How do we do EBM?2. Obtain current best evidence
  • An appropriate search strategy is important
  • We should be aware of the hierarchy of evidence:
    • Systematic reviews
    • Meta-analyses
    • Randomised-controlled trials
    • Cohort studies
how do we do ebm 2 obtain current best evidence3
How do we do EBM?2. Obtain current best evidence

Its a guide—it’s not necessarily this simple! Consider:

  • A large RCT vs. a meta-analysis of small RCTs?
  • A meta-analysis of observational studies?

Hierarchy becomes complicated

The hierarchy of evidence:

how do we do ebm 3 critical appraisal
How do we do EBM?3. Critical appraisal
  • Consider the following concepts
    • Internal validity
      • Does it prove what it set out to prove?
    • External validity
      • Are the results true in the wider world?
    • Bias
      • Is there systematic error?
how do we do ebm 3 critical appraisal placebo controlled rcts1
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • Are this study’s results valid?
    • Did the trial address a clearly focused issue?
      • Consider the intervention and the outcomes considered
    • Is an RCT an appropriate method to answer this question?
    • Was the assignment of patients to treatment randomised?
      • Why is it important?
      • Acceptable vs dubious methods?
      • Is the process well described?
      • Was the process concealed?
how do we do ebm 3 critical appraisal placebo controlled rcts2
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • Are this study’s results valid?
    • Were the groups similar at the start of the trial?
      • Could any differences have affected outcome?
      • Was any method used to balance randomisation (stratification)?
    • Was follow-up sufficient?
      • Length
      • Completeness
    • Was there sufficient power?
      • Were there enough participants to minimise the play of chance?
    • Was an intention-to-treat analysis performed?
how do we do ebm 3 critical appraisal placebo controlled rcts3
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • Are the results of this individual study important for us?
    • Define the population studied
      • Inclusion and exclusion criteria
    • Endpoints
      • Measurement of outcome
        • Clinical relevance
        • Surrogate and composite markers
        • Validity
      • Primary vs. secondary
    • Balance of beneficial outcomes against side-effects
how do we do ebm 3 critical appraisal placebo controlled rcts4
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • Are the results of this individual study important for us?
    • What is the magnitude of the treatment effect?
      • How is it described?
        • Proportions of people, a measurement (mean or median differences), a survival curve?
      • How is it expressed?
        • With proportions see terms like relative risk reduction, absolute risk reduction, number needed to treat
    • Is what is being expressed clear?
    • Is it clinically significant?
how do we do ebm 3 critical appraisal placebo controlled rcts5
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • Definitions for measures of association and effectiveness
    • Control event rate
      • the rate at which an outcome occurs in the control population
    • Experimental event rate
      • the rate at which an outcome occurs in the experimental population
how do we do ebm 3 critical appraisal placebo controlled rcts6
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • Definitions for measures of association and effectiveness
    • Absolute risk (AR)
      • Probability an individual will experience a specified outcome during a specified period
      • Range of 0 to 1, or expressed as a percentage
    • Relative risk (RR)
      • How many times more likely (RR > 1) or less likely (RR < 1) an event is to happen in one group compared with another
    • Absolute risk reduction (ARR)
      • Absolute difference in risk between experimental and control groups
    • Relative risk reduction (RRR)
      • Proportional reduction in risk between experimental and control participants in a trial
how do we do ebm 3 critical appraisal placebo controlled rcts7
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • Definitions for measures of association and effectiveness
    • Number needed to treat (NNT)
      • A measure of treatment effectiveness
      • Measures the people who need to be treated with an intervention over a period of time to prevent an additional adverse outcome or achieve an additional beneficial outcome
      • Reciprocal of ARR
how do we do ebm 3 critical appraisal placebo controlled rcts8
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • Definitions for measures of association and effectiveness
    • Odds ratio (OR)
      • Measure of treatment effectiveness—likelihood of event happening in experimental group vs. control group
      • Effects being measured may be adverse (e.g. death or disability) or desirable (e.g. survival)
      • If events rare OR analogous to relative risk (RR); as event rates increase the OR and RR diverge
    • When interpreting remember:
      • Closer OR is to 1  smaller the difference in effect between experimental intervention and control intervention
      • OR > 1 ≡ effects of treatment more than control
      • OR < 1 ≡ effects of treatment less than control
      • OR 95% CI includes 1 ≡ effect cannot be deemed statistically significant
how do we do ebm 3 critical appraisal placebo controlled rcts9
Absolute vs. relative risk reduction as measures—an example

Pros and cons of each measure

Clinical significance?

Proportional difference?

Ease of interpretation?

How do we do EBM?3. Critical appraisal—placebo controlled RCTs
how do we do ebm 3 critical appraisal placebo controlled rcts10
An example—4S study (Scandinavian Simvastatin Survival Study)1

Purpose

Evaluate effect of cholesterol lowering with simvastatin on mortality and morbidity in patients with coronary heart disease (CHD)

Participants

Patients with stable angina or history of myocardial infarction more than 6 months previously

Serum cholesterol > 6.2mmol/l

Excluded patients with arrhythmias and heart failure

Run in of 8 weeks of dietary therapy

If cholesterol still raised (5.5-8.0 mmol/L) randomised to receive simvastatin (20mg esculating to 40mg) or placebo

Lancet. 1994 Nov 19;344(8934):1383-9

How do we do EBM?3. Critical appraisal—placebo controlled RCTs
how do we do ebm 3 critical appraisal placebo controlled rcts11
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • 4S study
    • Randomisation and allocation

4444 patients randomised to double-blind treatment with simvastatin or placebo

2223 patients placebo

2221 were simvastatin

Mean follow-up was 5.4 years

how do we do ebm 3 critical appraisal placebo controlled rcts12
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • 4S study
    • Results
    • All cause mortality:

256 patients (12%) in placebo group dead vs 182 (8%) in simvastatin group

    • Coronary deaths:

189 coronary deaths in the placebo group vs. 111 in the simvastatin group

    • Morbidity (coronary event):

622 patients (28%) in placebo group vs. 431 (19%) in the simvastatin group

    • So, on the face of it, the results look extremely promising. How do the measures of association we discussed earlier play out?
how do we do ebm 3 critical appraisal placebo controlled rcts13
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • 4S study results—all cause mortality

OUTCOME

Dead Alive

Control group (placebo) n = 2223 256 1967

Experimental group (simvastatin) n = 2221182 2039

Control event rate = 256 = 0.115 (11.5%) 2223

Control odds of event = 256 = 0.130 1967

Experimental event rate = 182 = 0.082 (8.2%) 2221

Experimental odds of event = 182 = 0.089 2039

how do we do ebm 3 critical appraisal placebo controlled rcts14
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • 4S study results—all cause mortality

Relative risk reduction CER – EER = 0.115 – 0.082 = 0.29 (29%) RRR CER 0.115

Absolute risk reduction CER – EER= 0.115 – 0.082 = 0.033 (3.3%)ARR

Number needed to treat _1_ = _1_ = 30.1NNT ARR 0.033

Relative risk EER = 0.082 = 0.71 RR CER 0.115

Odds ratio = odds of event in experimental group = 0.089 = 0.69OR odds of event in control group 0.130

how do we do ebm 3 critical appraisal placebo controlled rcts15
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • 4S study results—all cause mortality
how do we do ebm 3 critical appraisal placebo controlled rcts16
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • NNT examples—when should we adopt therapy?

Intervention

Outcome

NNT

how do we do ebm 3 critical appraisal placebo controlled rcts17
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • Discussion point: numbers needed to treat
    • When do you think therapy should be adopted?
    • Clearly there is no one answer to this but what might be the considerations?
how do we do ebm 3 critical appraisal placebo controlled rcts18
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • Statistical approaches to uncertainty
    • Why do they exist?
      • Cannot include all individuals in a population in trial
      • Need to quantify uncertainty
    • p values and confidence intervals are the measures used, but what are they and what are the differences?
how do we do ebm 3 critical appraisal placebo controlled rcts19
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • Statistical approaches to uncertainty
    • p values
      • assess the significance of the difference between a sample estimate and a hypothesised value
      • tell us the probability that an observed effect occurred by chance if in truth there is no effect
      • But… doesn’t quantify the size of an effect, nor the direction
      • p<0.05 → commonly reject null hypothesis
      • ideally p<<<0.05 and trial reports the actual value
how do we do ebm 3 critical appraisal placebo controlled rcts20
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • Statistical approaches to uncertainty
    • Confidence intervals
      • Range of values around the sample estimate (i.e. the value found in the study)
      • The range has specified probability (usually 95%) so CI acts as hypothesis test for the range
      • Should be seen with most measures of association
how do we do ebm 3 critical appraisal placebo controlled rcts21
How do we do EBM?3. Critical appraisal—placebo controlled RCTs
  • Statistical approaches to uncertainty—please remember!
    • statistical significance ≠ clinical significance… what is actually being measured?
    • sample size is still important
      • larger sample = less uncertainty = narrower CI and smaller observed effect considered significant
    • there can still be error
      • rejecting null hypothesis when it’s true (type 1)
      • Not rejecting the null hypothesis when it’s false (type 2)
how do we do ebm 3 critical appraisal rcts of therapeutic equivalence
How do we do EBM?3. Critical appraisal—RCTs of therapeutic equivalence
  • Increasingly used by pharma.
  • Designed to show a new treatment is not inferior to standard treatment by a predefined clinically acceptable endpoint
  • Rely on assumptions that can be hard to validate
how do we do ebm 3 critical appraisal rcts of therapeutic equivalence1
How do we do EBM?3. Critical appraisal—RCTs of therapeutic equivalence
  • Trials based on the concept that:
    • new treatment is non-inferior
    • but would exhibit therapeutic efficacy if a placebo controlled RCT could/was performed
    • new treatment offers ancillary benefits
  • Note: non-inferior so efficacy determined by effect against placebo not comparator
how do we do ebm 3 critical appraisal rcts of therapeutic equivalence2
How do we do EBM?3. Critical appraisal—RCTs of therapeutic equivalence
  • Estimation of non-inferiority margin
    • Use statistical and clinical reasoning to determine what is non-inferior
    • If there are a variety of placebo controlled RCTs this will be similar to a meta-analysis
  • Non-inferiority margin determined as a fraction (f) and specifies an acceptable magnitude for treatment effect that must be preserved
slide38

How do we do EBM?3. Critical appraisal—RCTs of therapeutic equivalence

  • What to look for
    • These trials should be designed in a manner consistent with the historical placebo-controlled trials
    • The active comparator should be well established with predictable quantifiable, and consistent effects
    • What constitutes non-inferiority should be determined prior to initiation of trial
    • Protocol deviations and poor adherence may have a larger impact on quality than in conventional trials
    • Analysis should be by intention-to-treat AND on-treatment, and should also report absolute AND relative effects
how do we do ebm 3 critical appraisal rcts of therapeutic equivalence3
How do we do EBM?3. Critical appraisal—RCTs of therapeutic equivalence
  • So when appraising these trials ask yourself:
    • Was the active control previously shown to be effective?
    • Were study patients and outcome variables similar to those in the original trials that established the efficacy of the active control?
    • Were both regimens applied in optimal fashion?
    • Was the appropriate null hypothesis tested?
    • Was the equivalence margin specified before the study?
    • Was the trial large enough?
    • Was the analysis intention-to-treat AND on-treatment, and should also report absolute AND relative effects?
    • PLUS usual assessment of size and precision of treatment effect!
how do we do ebm 3 critical appraisal other study designs
How do we do EBM?3. Critical appraisal—other study designs
  • Many other types of study in the hierarchy e.g. systematic reviews, case-control studies, cohort studies
  • Details of appraisal not included today but worth thinking about most in terms of:
    • Internal validity
    • Bias
    • External validity
how do we do ebm 3 critical appraisal other study designs1

Exposed

Population

Outcome

Sample

Time

Outcome

Not exposed

How do we do EBM?3. Critical appraisal—other study designs
  • Cohort study
how do we do ebm 3 critical appraisal other study designs2

Study

Population

Exposed

Cases

Not exposed

Controls

Exposed

Not exposed

Time

How do we do EBM?3. Critical appraisal—other study designs
  • Case control study
how do we do ebm 3 economic evidence
How do we do EBM?3. Economic evidence
  • Important in decision making
  • Considers costs and consequences of alternate courses of action
  • Scope depends on level of decision making they inform
  • Costs always measured the same way (although scope may vary), measurement of consequence varies according to study type
how do we do ebm 3 critical appraisal what is economic evidence
How do we do EBM?3. Critical appraisal—what is economic evidence?
  • Cost minimisation analyses
    • analyse difference in costs where there is no difference in outcome
    • narrow focus
  • Cost-effectiveness analyses
    • outcome expressed in natural units (e.g. validated single marker)
    • again fairly narrow scope—generally 2 interventions for single indication
how do we do ebm 3 critical appraisal what is economic evidence1
How do we do EBM?3. Critical appraisal—what is economic evidence?
  • Cost-utility analyses
    • Costs and consequences but consequence measured as utility
    • Utility = a value of health state rather than a natural marker (usually QALY)
    • QALY is a measure of a person’s length of life weighted by a valuation of their health-related quality-of-life (HRQL) over that period
    • Can compare incongruent interventions
how do we do ebm 3 critical appraisal what is economic evidence2
How do we do EBM?3. Critical appraisal—what is economic evidence?
  • Cost utility analysis
    • Used by NICE
      • Consider outcome in terms valuable to patients i.e. life expectancy and quality
      • Uses a standard measure (utility)
      • Comprehensive consideration of costs
      • NICE state that:

Technologies can be considered to be cost effective if their health benefits are greater than the opportunity costs measured in terms of the health benefits associated with programmes that may be displaced to fund the new technology. In other words, the general consequences for the wider group of patients in the NHS are considered alongside the effects for those patients who may directly benefit from the technology of interest.

how do we do ebm 3 critical appraisal what is economic evidence3
How do we do EBM?3. Critical appraisal—what is economic evidence?
  • Cost-benefit analyses
    • Broadest possible scope!
    • Allocation of resources between difference sectors of economy
    • Maximising social welfare
    • Rely on assigning cost to healthcare intervention—incredible complicated in practice, particular in NHS where individuals don’t pay at point of delivery
how do we do ebm 3 critical appraisal further reading
How do we do EBM?3. Critical appraisal—further reading
  • A lot is published! But texts I’ve found useful include:
    • Straus SE, Richardson WS, Paul Glasziou, Haynes RB. Evidence-based Medicine: How to Practice and Teach EBM, Third Edition. Churchill Livingstone: Edinburgh, 2005
    • Guyatt GH, et al. Users’ Guides to the Medical Literature: II. How to use an article about therapy or prevention. A. Are the results of the study valid? JAMA 1993; 270: 2598-2601
    • Guyatt GH, et al. Users’ Guides to the Medical Literature: II. How to use an article about therapy or prevention. B. What are the results and will they help me in caring for my patients? JAMA 1994; 271: 59-63
    • Evidence Based Medicine (EBM) journal notebook series—available online
    • Clinical evidence online. Excellent terminology glossary and explanations
how do we use ebm 4 applying results of research to our individual patient or population
How do we use EBM?4. Applying results of research to our individual patient or population
  • Is our patient/population so different from those in the study that its results cannot apply?
  • Is the treatment feasible for us?
  • What are the potential benefits and harms from therapy?
  • What are our patients’ values and preferences for both the outcome and side-effects?
  • Is there any economic analysis supporting therapy?
example critical appraisal workshop
Example critical appraisal workshop
  • The scenario
    • You are asked to review the evidence and to contribute towards an individual funding request for submission to a PCT.
    • The patient has refractory Crohn’s disease specified as being previously intolerant to azathioprine, methotrexate, salazopyrine, and infliximab
example critical appraisal workshop1
Example critical appraisal workshop
  • The background
    • Adlimumab is licensed for severe active Crohn’s unresponsive to corticosteroids and conventional immunosupression
    • Use in relation to infliximab is not well defined. The SMC reviewed adalimumab for Crohn’s in 2007 but not consider place in therapy wrt infliximab
    • A NICE TAG is pending
example critical appraisal workshop2
Example critical appraisal workshop
  • Your role
    • You have identified the GAIN study
    • You now need to appraise it and to reach a conclusion as its validity with reference to the circumstance described
    • You can use the CASP tool to help
example critical appraisal of rct workshop
Example critical appraisal of RCT workshop
  • Working through each point from the CASP tool:
  • Did the study ask a clearly-focused question?
  • Was this a randomised controlled trial (RCT) and was it appropriately so?
  • Were participants appropriately allocated to intervention and control groups?
  • Were participants, staff and study personnel ‘blind’ to participants’ study group?
  • Were all of the participants who entered the trial accounted for at its conclusion?
example critical appraisal of rct workshop1
Example critical appraisal of RCT workshop
  • Were the participants in all groups followed up and data collected in the same way?
  • Did the study have enough participants to minimise the play of chance?
  • How are the results presented and what is the main result?
  • How precise are these results?
  • Were all important outcomes considered so the results can be applied?
example critical appraisal of rct workshop2
Good points

Potential drawbacks

Example critical appraisal of RCT workshop

Overall, what are the main messages with reference to this patient?