slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Outline PowerPoint Presentation
Download Presentation
Outline

Loading in 2 Seconds...

play fullscreen
1 / 26

Outline - PowerPoint PPT Presentation


  • 134 Views
  • Uploaded on

Updating Surveillance System: Assessing the Need for Updating Comparative Effectiveness Reviews The University of Ottawa Evidence–based Practice Center The RAND Southern California Evidence-based Practice Center Alexander Tsertsvadze MD, MSc Date: September 11, 2012. Outline. Background

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Outline' - gamma


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Updating Surveillance System:Assessing the Need for Updating Comparative Effectiveness Reviews The University of Ottawa Evidence–based Practice Center The RAND Southern California Evidence-based Practice Center Alexander Tsertsvadze MD, MScDate: September 11, 2012

outline
Outline

Background

Why update systematic reviews (SRs)? What is an update of a SR?

When/how to update a SR?

AHRQ updating surveillance project

Methods

Overview of the process

The Ottawa/RAND EPC methodology

Findings and implications

Brief overview

background
Background

Why update SRs?

SRs play a central role in comparative effectiveness research

With the emergence of new evidence over time, conclusion(s)

of any given SR may become out of date and sometimes

misleading

Changes in the evidence can have significant implications for

clinical practice guidelines and forclinical and consumer

decision-making

background1
Background

What is an update of SR?

“A discrete event with the aim to search for and

identify new evidence to incorporate into a

previously completed systematic review”

An update must involve a search for new studies

Moher and Tsertsvadze. Lancet 2006; 367:881-3

Cochrane Hand Book section 3.2.2

background2
Background

Shojania 2007

Sets of quantitative/qualitative signals indicating need for updating tested on

100 SRs; within 2 yrs, 23% of SRs had signals indicating the need for updating;

the median time to the emergence of signal indicating the need for updating

was 5.5 years

French 2005

Of updated SRs (n=254), 9% (n=23) had changes in conclusions within 4 yrs

Garritty 2010

53/103 of respondents estimated >50% of their respective SRs were likely

out-of-date

background3
Background

One SR found very few methods and strategies used for updating SRs

The authors identified the need for standardized reliable/efficient

method(s) for updating

(Moher and Tsertsvadze 2007; Tsertsvadze 2011)

background4
Background

In 2007-09, Ottawa and RAND Evidence-based Practice Centers (EPCs) within the

Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program

developed a methodology for determining the need of updating Comparative

Effectiveness Reviews (CERs)

  • Abbreviated searches for new evidence
  • Identification of signals indicating the need of updating by applying a priori setcriteria to the new evidence
  • Expert opinion on validity of CER conclusions
  • Topic-specific safety alerts from FDA and Health Canada
  • Rules/guidance for determining the updating status of conclusion(s) for each key question (KQ) within CER and updating priority for each CER (low, medium, high)

(Shojania 2007; Shekelle 2009)

background5
Background

Objectives

  • Recognizing the importance of keeping CERs up-to-date, the AHRQ commissioned the RAND and Ottawa EPCs in collaboration with ECRI Institute to implement an updating surveillance system project
  • This collaborative project would use the previously developed methodology to assess already published CERs (n=42) for the need of updating

___________________________________________________

- CERs unlike SRs contain several key questions/conclusions

- Updating vs. assessing the need for updating

slide9

Original CER

0

ECRI surveillance

Contacting experts

Start of review

Abbreviated searches for new evidence using the same strategy as in the CER

Updating surveillance process

Screening and extracting data from relevant studies

Identifying qualitative/quantitative signals

FDA/Health Canada

alerts

Signal detection completed

Expert opinion

Deciding on updating status of conclusion(s) for each KQ within CER and determining updating priority for each CER

Low priority

Medium priority

High priority

3rd month

Goes to 2nd cycle of assessment 6 months later

Refereed for updating

End of review

methods abbreviated search for new evidence
MethodsAbbreviated search for new evidence

Search strategies – the same as used in the original CER but limited to

5 general medicine journals

(Ann Intern Med, BMJ , JAMA, Lancet and NEJM)

5specialty journals (specific to a topic of CER; most frequently cited journals in CER)

slide11

Original CER

ECRI surveillance

Contacting experts

Abbreviated searches for new evidence using the same strategy as in the CER

Screening and extracting data from relevant studies

Identifying qualitative/quantitative signals

FDA/Health Canada

alerts

Signal detection completed

Expert opinion

Deciding on updating status of conclusion(s) for each KQ within CER and determining updating priority for each CER

Low priority

Medium priority

High priority

Goes to 2nd cycle of assessment 6 months later

Refereed for updating

methods contacting content experts
MethodsContacting content experts

Expert Opinion

CER-specific clinical experts

(content experts, technical expert panel members, expert peer reviewer)

Other clinical experts

(external, local)

Feeds into the decision on updating status of KQ specific conclusion(s) in each CER

slide13

Original CER

ECRI surveillance

Contacting experts

Abbreviated searches for new evidence using the same strategy as in the CER

Screening and extracting data from relevant studies

Identifying qualitative/quantitative signals

FDA/Health Canada

alerts

Signal detection completed

Expert opinion

Deciding on updating status of conclusion(s) for each KQ within CER and determining updating priority for each CER

Low priority

Medium priority

High priority

Goes to 2nd cycle of assessment 6 months later

Refereed for updating

methods safety surveillance alerts
MethodsSafety surveillance alerts

FDA/Health Canada alerts on CER topics (monthly)

On CERs assigned to RAND EPC

On CERs assigned to Ottawa EPC

Feeds into the decision on updating status of KQ specific conclusion(s) in each CER

slide15

Original CER

ECRI surveillance

Contacting experts

Abbreviated searches for new evidence using the same strategy as in the CER

Screening and extracting data from relevant studies

Identifying qualitative/quantitative signals

FDA/Health Canada

alerts

Signal detection completed

Expert opinion

Deciding on updating status of conclusion(s) for each KQ within CER and determining updating priority for each CER

Low priority

Medium priority

High priority

Goes to 2nd cycle of assessment 6 months later

Refereed for updating

methods
Methods

Identifying signals for updating

Qualitative signals

  • Potentially invalidating change in evidence
  • [pivotal trial/MA]
  • Opposing findings in effectiveness
  • Substantial harm
  • A superior new treatment
  • Major change in evidence
  • [pivotal trial/MA]
  • Important changes in effectiveness short of opposing findings
  • Clinically important expansion of treatment
  • Clinically important caveat
  • [non-pivotal trial/MA]
  • Opposing findings in effectiveness/harms
methods1
Methods

Identifying signals for updating

Quantitative

Change in statistical significance

significant vs. non-significant

non-significant vs. significant

At least 50% change in effect size

slide18

Original CER

ECRI surveillance

Contacting experts

Abbreviated searches for new evidence using the same strategy as in the CER

Screening and extracting data from relevant studies

Identifying qualitative/quantitative signals

FDA/Health Canada

alerts

Signal detection completed

Expert opinion

Deciding on updating status of conclusion(s) for each KQ within CER and determining updating priority for each CER

Low priority

Medium priority

High priority

Goes to 2nd cycle of assessment 6 months later

Refereed for updating

methods rules for assessing updating status of conclusion s for each kq
MethodsRules for assessing updating status of conclusion(s) for each KQ
  • No new evidence
  • Only confirmatory evidence
  • All responding experts: conclusions as still valid

Up to date

  • Some new evidence
  • A minority of responding experts: having new evidence that might change the conclusion

Possibly out of date

  • Substantial new evidence
  • Majority of responding experts: having new evidence that might change the conclusion

Probably out of date

  • New evidence that rendered the CER conclusion out of date or no longer applicable
    • E.g. withdrawal of a drug or surgical device from the market, a black box warning from FDA, etc

Out of date

methods2
Methods

Rules for assigning the updating priority to a CER

high, medium, or low

Out of date

How many conclusions?

Possibly out of date

Probably out of date

Magnitude/

direction of changes in estimates

Potential changes in practice or therapy

How out of date are conclusions?

Drugs withdrawn from the market /black box warning

Availability of a new treatment

methods3
Methods
  • A brief report summarizing the findings from the surveillance process
    • Title page lists the final priority for updating (“low,” “medium,” or “high”)
    • Description of methods and results
    • Summary table by conclusion:
      • findings of the literature search
      • findings from FDA and Health Canada
      • expert opinion
      • assessment of the degree to which the conclusion maybe out-of-date
    • Evidence table
findings
Findings

1st year of surveillance

14 CERs were assessed for updating status and were

assigned updating priorities as follows:

2 (14.2%) = ‘high priority’

3 (21.4%) = ‘medium priority’

9 (64.4%) = ‘low priority’

implications
Implications
  • We have established a Surveillance Program that has evaluated 14 CERs over the course of 12 months
  • We have shown that a program for regular and active surveillance of CERs is feasible
implications1
Implications
  • To our knowledge, the surveillance assessment project is the first large scale effort that has applied a set of methods to assess the currency of evidence-based reports (or systematic reviews) in a structured and standardized manner
  • The application of this methodology proves to be a relatively feasible, efficient, and at the same time, comprehensive approach for assessing the need for updating individual CERs across a wide range of health interventions
references
References
  • Moher D, Tsertsvadze A. Systematic Reviews: When is an Update an Update? Lancet 2006; 367: 881-883
  • Garritty C, Tsertsvadze A, Tricco AC, Sampson M, Moher D. Updating systematic reviews: an international survey. PloS one 2010; 5(4):e9914.
  • French SD, McDonald S, McKenzie JE, Green S. Investing in updating: how do conclusions change when Cochrane systematic reviews are updated? BMC Med Res Methodol 2005;5:33.
  • Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, Sampson M et al. A systematic review identified few methods and strategies describing when and how to update systematic reviews. J ClinEpidemiol 2007; 60(11):1095-10e1.
  • Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med 2007;147:224e33.
  • Shekelle P, Newberry S, Maglione M et al. Assessment of the need to update comparative effectiveness reviews: Report of an initial rapid program assessment (2005-2009) [Internet]. 2009 Sep 10.
  • Shekelle PG, Newberry SJ, Wu H, Suttorp M, Motala A, Lim Y-W, Balk EM, Chung M, Yu WW, Lee J, Gaylor JM, Moher D, Ansari MT, Skidmore R, Garritty C. Identifying Signals for Updating Systematic Reviews: A Comparison of Two Methods. Rockville (MD): Agency for Healthcare Research and Quality. June 2011. Methods Research Report. AHRQ Publication No. 11-EHC042-EF.
  • Tsertsvadze A, Maglione M, Chou R, Garritty C, Coleman C, Lux L et al. Updating comparative effectiveness reviews: current efforts in AHRQ's Effective Health Care Program. J ClinEpidemiol 2011; 64(11):1208-1215.