evidence based caveat emptor ii n.
Skip this Video
Loading SlideShow in 5 Seconds..
Download Presentation

Loading in 2 Seconds...

play fullscreen
1 / 34


  • Uploaded on

EVIDENCE-BASED? CAVEAT EMPTOR II!. Presented to the NAMI Symposium: “What Do We Mean By Evidence-based Practices?” by Bryan R. Luce, Ph.D., MBA The MEDTAP Institute at UBC Monday, June 20 2005 Austin, Texas. Today, being evidence-based is “de rigueur”.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'EVIDENCE-BASED? CAVEAT EMPTOR II!' - villette

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
evidence based caveat emptor ii


Presented to the NAMI Symposium: “What Do We Mean By Evidence-based Practices?”


Bryan R. Luce, Ph.D., MBA

The MEDTAP Institute at UBC

Monday, June 20 2005

Austin, Texas

selected organizations using ebm
Selected Organizations Using “EBM”
  • BCBS’s Technology Evaluation Center
  • US Preventive Services Task Force
  • Clinical Practice Guidelines
  • CMS Medicare Coverage Advisory Committee (MCAC)
  • AHRQ’s Evidence-Based Practice Centers
organizations using ebm cont
Organizations Using EBM (Cont.)
  • UK’s National Institute for Clinical Excellence (NICE)
  • AMCP’s Format for Formulary Submission
    • Multiple MCOs
  • OHSU Drug Effectiveness Review Project
    • 13 Medicaid Agencies
    • Consumers’ Union BestBuyDrugs website
    • AARP’s ResearchRx website
organizations using ebm cont1
Organizations Using EBM (Cont.)
  • CMS’s MMA: Comparative Effectiveness
  • CMS Interim Coverage: PCT/Registries
  • Institute of Medicine: EBM/Comparative Effectiveness Private-Public Initiative
evidence based caveat emptor

Evidence-Based?Caveat Emptor!

Earl P. Steinberg, MD

Bryan R. Luce, Ph.D.

Health Affairs

January, February 2005

caveat emptor i conclusions
Caveat Emptor I: Conclusions
  • Rigorous methods for evaluating clinical studies now exist, but
    • Methods not applied consistently
    • Results not always interpreted properly


    • Much variation in validity of “EB” decisions


    • Evidence may be available for some but not all issues


caveat emptor i conclusions cont
Caveat Emptor I: Conclusions (cont.)

Available evidence may not be applicable to decision at hand


  • Decision-makers should not blindly assume the evidence-based label truly applies

And, finally…

  • EBM analysts should guide decision-makers as to applicability to populations of interest and conditions to which evidence applies.
now caveat emptor ii
Now…Caveat Emptor “II”
  • Poor agreement…
    • Definition of “EBM”
    • Definition of “systematic review”
    • The relevance and scope of evidence
  • Other issues to be resolved
  • Recommendation for evidence-based evaluations for policy decisions
definitional clarity of ebm
Definitional Clarity of EBM
  • EBM is not a unitary concept*
    • Evidence based individual physician-patient decisions
    • Evidence-based policy/group-focused decisions
  • Each application has/should have own unique definition and designation

(Eddy, 2005)

ebm s early concept
EBM’s Early Concept
  • Based on systematic review (meta-analysis) of high quality RCT evidence as proposed by Archie Cochrane and others.
  • Early definition: “The conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.”
  • Debate in UK literature: “Leads to cookbook medicine”?
sackett s evolving definition
Sackett’s Evolving Definition*

Countering cookbook charge, Sackett et al added: “The practice of EBM means integrating individual clinical expertise with the best available external clinical evidence from systematic research [and] individual patients’ predicaments, rights and preferences in making clinical decisions about their care”.

*Sackett et al., EBM, What it is and what it isn’t. 1996, BMJ


Sackett et al sought to expand the concept of EBM to include clinical judgment and patient values. This could (and probably should) include real world clinical practice issues such as convenience, adherence factors, even economic effects as they may pertain to patient adherence.

Result: EBM for individual decision-making no longer prisoner of RCT; also, is a much less precise process & concept.

what about policy group ebm
What About Policy/Group EBM?
  • Terminology: Since policy/group EB decision-making is different concept, we need different term:
    • How about “EBG”? (Eddy, 2005)
  • Second, definition?
    • We could find none in literature
    • So let’s look at practice
ebg in practice three groups
EBG in Practice: Three Groups
  • The “efficacy” group
  • The “effectiveness” group
  • The “cost-effectiveness” group
more definitions
More Definitions
  • Efficacy: A healthcare intervention is considered “efficacious” when there is evidence that the intervention is beneficial when administered by experts in a research setting.
  • Effectiveness: An intervention is considered “effective” when there is evidence of benefit when administered by community-based physicians to real-world patients in real-world clinical settings.

Steinberg, Luce, 2005

the ebg efficacy group
The EBG “Efficacy” Group
  • Mantra is “minimize bias”
    • Opt for maximizing internal validity of studies at expense of generalizability
  • Examples of organizations:
    • Cochrane Collaboration
    • OHSU’s Drug Effectiveness Review Project
      • Medicaid agencies (?), Consumers Union, AARP
the ebg effectiveness group
The EBG “Effectiveness” Group
  • Mantra: “Is it effective in the real world?”
    • Opt for generalizability to populations and clinical settings of interest (incl non-experimental evidence)
  • Examples of organizations:
    • CMS (MCAC, MMA, PCT/Registry Interim Coverage Policy)
    • AHRQ’s EPCs
    • BCBS TEC
    • Clinical Practice Guidelines
    • IOM’s Effectiveness Initiative
the ebg cost effectiveness group
The EBG Cost-Effectiveness Group
  • Mantra is “real world effectiveness and real world value for money”
    • Opt for generalizability often over long haul and accuracy over precision
  • Examples of organizations:
    • Academy of Managed Care Pharmacy
    • U.K.’s National Institute for Clinical Excellence (NICE)
    • AHRQ’s EPCs
possible ebg definition
Possible EBG Definition

“The conscientious, explicit and judicious use of current best evidence in making health policy decisions affecting groups of patients.”

But this begs the question, do we care about efficacy, effectiveness or cost-effectiveness?

My answer: Not Efficacy! Thus….

suggested ebg definition
Suggested EBG Definition

“The conscientious, explicit and judicious use of current best evidence in making health policy decisions affecting groups of patients. The practice of ‘EBG’ means integrating evidence from real world practice with the best available external (RCT) clinical evidence from systematic research. EBG may or may not include economic evaluation.”

next systematic review
Next… Systematic Review
  • How defined?
  • How performed?
sackett s definition
Sackett’s Definition

“A summary of the medical literature that uses explicit methods to perform a thorough literature search and critical appraisal of individual studies and that uses appropriate statistical techniques to combine these valid studies.”

Sackett, et al. 2001

does practice measure up to sackett s definition
Does Practice Measure up to Sackett’s Definition?
  • Mainly pertains to meta-analysis of similar RCTs, thus
    • Pertains to efficacy group
    • Meta-analysis portion of effectiveness and CE groups
  • But, non-efficacy groups also speak of using systematic review process
    • Using term descriptively
    • Do not imply formal statistical meta-analysis
however as aside
However (as aside)…
  • If formal statistical approach is desired, it is possible to employ a formal Bayesian statistical approach to different types of evidence (e.g. RCT & observational data). (Or, as AMCP: decision modeling)
  • NICE accepts such studies.
  • But, main point is….
when systematic review is claimed
When “Systematic Review” is claimed…
  • You do not know what is meant.

So….we need

    • New terminology
    • With definitions
suggested terminology definitions
Suggested Terminology & Definitions
  • “Systematic review of trials”, or “SRT” would refer to the structured meta-analysis process and generally result in best estimate of efficacy;
  • “Systematic review of evidence” or “SRE” would refer to the broader, less structure review of evidence and generally result in best estimate of effectiveness. Best practice would include SRT.
relevance scope of evidence for ebg process
Relevance & Scope of Evidence for EBG Process

“Best” evidence should include or consider:

  • All relevant RCTs
  • All relevant evidence from observational studies (eg claims data bases)
  • Estimates of final health outcomes using surrogate markers when needed
  • Economic outcomes of affected parties
  • Patient-reported outcomes including preferences
  • Possibly case studies, expert opinion, testimonials
some ebg related issues
Some EBG-Related Issues
  • If effectiveness is what is wanted, to what extent and under what circumstances does “efficacy” suffice for “effectiveness”?
  • How does one translate efficacy to effectiveness?
other ebg related issues cont
Other EBG-Related Issues (cont.)
  • Under what circumstances is evidence considered to be inadequate?
  • To what extent should inadequate or lack of evidence mean “not effective”?
wrap up
Wrap Up
  • Caveat Emptor I: “Decision-makers should not blindly assume the evidence-based label truly applies”
  • Caveat Emptor II: “We need to be explicit with respect to terminology & definitions and we need to agree on what is the appropriate evidence base for E-B policy/group decision-making.”
  • Just as Sackett et all responded to “cookbook” criticisms by acknowledging need for clinical judgment and patient preferences/values for EBM…
  • EBG requires multiple types of evidence to evaluate real world effectiveness
    • Modeling will commonly be needed
    • Answers will not necessarily be unambiguous, but…
with respect to the primacy of effectiveness over efficacy
With respect to the primacy of effectiveness over efficacy…

“…an imprecise estimate of the right concept is superior to a precise estimate of a wrong concept” (Mishan, 1972)

To put it another way…