Meeting the challenge of maintaining cochrane reviews
This presentation is the property of its rightful owner.
Sponsored Links
1 / 44

“Meeting the challenge of maintaining Cochrane Reviews” PowerPoint PPT Presentation


  • 98 Views
  • Uploaded on
  • Presentation posted in: General

“Meeting the challenge of maintaining Cochrane Reviews”. Author: Dr David Tovey FRCGP Editor in Chief, The Cochrane Library. Plan. Introduction Previous Cochrane project “Fit for purpose” project Future challenges and opportunities for co-operation Questions. Acknowledgements.

Download Presentation

“Meeting the challenge of maintaining Cochrane Reviews”

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Meeting the challenge of maintaining cochrane reviews

“Meeting the challenge of maintaining Cochrane Reviews”

Author: Dr David Tovey FRCGPEditor in Chief, The Cochrane Library


Meeting the challenge of maintaining cochrane reviews

Plan

  • Introduction

  • Previous Cochrane project

  • “Fit for purpose” project

  • Future challenges and opportunities for co-operation

  • Questions

AHRQ 2012 Annual Conference


Acknowledgements

Acknowledgements

  • Sally Hopewell, Yemisi Takwoingi, Alex Sutton, Rachel Marshall and Bazian Ltd.

  • This project was funded by the NIHR Cochrane – NHS Engagement Award Scheme (project number 10/4000/01). The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the Department of Health.


Acknowledgements1

Acknowledgements

  • Cochrane Musculoskeletal Group

  • Cochrane Pain, Palliative and Supportive Care Group

  • Cochrane Infectious Disease Group

  • Cochrane Wounds Group

  • Cochrane Neonatal Group

  • Cochrane Airways Group

  • Karla Soares-Weiser

AHRQ 2012 Annual Conference


Introduction protecting archie cochrane s vision

Introduction: Protecting Archie Cochrane’s vision

  • “It is surely a great criticism of our profession that we have not organised a critical summary by speciality and subspecialty adapted periodically of all relevant randomised controlled trials”

AHRQ 2012 Annual Conference


Introduction protecting archie cochrane s vision1

Introduction: Protecting Archie Cochrane’s vision

  • “It is surely a great criticism of our profession that we have not organised a critical summary by speciality and subspecialty adapted periodically of all relevant randomised controlled trials”

Credible& Safe

AHRQ 2012 Annual Conference


Introduction the size of the challenge

Introduction: the size of the challenge

AHRQ 2012 Annual Conference


Introduction the size of the challenge1

Introduction: the size of the challenge

AHRQ 2012 Annual Conference


Introduction the size of the challenge2

Introduction: the size of the challenge

AHRQ 2012 Annual Conference


Introduction the size of the challenge3

Introduction: the size of the challenge

“The boulder in my rucksack”

AHRQ 2012 Annual Conference


Previous cochrane projects the updating officer project

Previous Cochrane projects: the updating officer project

  • 8 reviews selected

  • On average these reviews each took 6.4 months (range 3-11 months) to update from receipt of the search strategy to submission for editorial review.

  • The main challenges:

    • lack of familiarity with individual Review Groups’ methods

    • out of date methodology within the existing reviews

    • lack of subsequent author commitment

    • lack of wish for ongoing commitment

  • Would imply the need for a small army of updating officers!!

AHRQ 2012 Annual Conference


Fit for purpose project

“Fit for purpose”project

Three objectives of the project:

  • To work with external stakeholders to develop a customisable prioritisation tool – prioritisation by topic

  • To develop a decision tool for determining whether and when to update Cochrane Reviews – prioritisation by status

  • To explore whether targeted consultancy could facilitate updating and identify process efficiencies

AHRQ 2012 Annual Conference


Fit for purpose project1

“Fit for purpose”project

Three objectives of the project:

  • To work with external stakeholders to develop a customisable prioritisation tool – prioritisation by topic

  • To develop a decision tool for determining whether and when to update Cochrane Reviews – prioritisation by status

  • To explore whether targeted consultancy could facilitate updating and identify process efficiencies

AHRQ 2012 Annual Conference


Methods

Methods

  • Project conducted by Bazian, co-applicants in the NHS engagement award.

  • Stakeholders in the NHS (clinicians, consumers, commissioners etc.) formed a panel.

  • The panel had two meetings, and communicated via email before and after meetings.


Results prioritisation criteria in the tool

Results: prioritisation criteria in the tool


Results testing the tool

Results: testing the tool

  • The tool was tested on 19 Cochrane Reviews, which found:

    • There is a level of judgement when assigning scores.

    • A certain level of knowledge of the field is helpful when assigning scores

    • Needs to be customised (notable absentee parameters..)

  • Recommended that the results of using the tool are discussed with external stakeholders including patient and carer representatives.


Fit for purpose project2

“Fit for purpose” project

Three objectives of the project:

  • To work with external stakeholders to develop a customisable prioritisation tool – prioritisation by topic

  • To develop a decision tool for determining whether and when to update Cochrane Reviews – prioritisation by status

  • To explore whether targeted consultancy could facilitate updating and identify process efficiencies

AHRQ 2012 Annual Conference


Methods1

Methods

  • We refined and amalgamated two complementary methodologies for prioritising systematic review updates:

    • a qualitative tool based on a broad range of updating signals (Loudon 2008).

    • formal statistical methods which assess when the inclusion of new studies is likely to change a review’s conclusions (Sutton 2009).


Output decision tool

Output: Decision tool


Step 1 is the clinical question answered or no longer relevant

Step 1: Is the clinical question answered or no longer relevant?


Step 2 are there any new factors to consider

Step 2: Are there any new factors to consider?


Steps 3 4 are there new studies are the conclusions likely to change

Steps 3 & 4: Are there new studies? Are the conclusions likely to change?


Statistical prediction tool

Statistical prediction tool


About metarank

About metarank

  • Based on minimal information on the new evidence

    • assumes an update strategy is in place such that number of new studies and their sample sizes are known

  • ‘Signals’ of the need to update implemented as a STATA user-written function

  • Performs simulation of several meta-analyses, each with one or more new studies of different sizes


Metarank output

Metarank output

  • Summary of results for each meta-analysis includes details of the original meta-analysis and the signals detected

  • Table with all reviews in the dataset ranked in order of priority

    • by a given signal

    • or the total of all signals triggered

  • Includes descriptive information for the collection e.g.

    • average number of trials and participants in the collection of meta-analyses

    • number of trials in the largest meta-analysis

    • largest number of participants in a meta-analysis


Advantages of metarank

Advantages of metarank

  • Assessment of the likelihood of any definable criteria changing in a meta-analysis

  • Ongoing studies can be taken into account

    • aid in predicting potential ‘shelf life’ of a review in the light of new or accruing evidence

  • To provide information with respect to further research needed such as the number of new trials and the number of participants

    • useful information to end users of reviews in assessing the stability or validity of a review


Limitations of metarank

Limitations of metarank

  • Relies on availability of some information on new trials

  • Simplistic approach used in deriving the predictive distribution for new studies

  • Other issues external to a review may need to be taken into account such as

    • the rate at which new evidence on a topic evolves

    • public heath significance etc.

  • No criteria developed to establish when enough evidence has accrued on a given topic and review that it is deemed decisive and not worth conducting further primary research


Decision tool summary

Decision tool: summary

  • The decision tool provides a set of criteria that can be used to assess whether to update a Cochrane Review.

  • The tool can be applied to a single Cochrane Review or can be used to prioritise a suite of reviews (e.g. those from an individual Cochrane Review Group)


Results testing the tool1

Results: testing the tool

  • One Cochrane Review Group so far

    • Cautiously positive - “helpful structure to assess each review’s eligibility for updating”“a transparent way to explain decisions around updating to stakeholders”

    • Time consuming but might improve with experience

    • Some assumptions around the statistical tool “ too crude”

    • Need to have wider experience and use

AHRQ 2012 Annual Conference


Fit for purpose project3

“Fit for purpose” project

Three objectives of the project:

  • To identify those Cochrane Reviews that NHS stakeholders regard as the most important to update – prioritisation by topic

  • To develop a decision tool for determining whether and when to update Cochrane Reviews – prioritisation by status

  • To explore whether targeted consultancy could facilitate updating and identify process efficiencies

AHRQ 2012 Annual Conference


Meeting the challenge of maintaining cochrane reviews

Aims

  • We aimed to provide short and focused periods of assistance, to incentivise authors, and to address barriers to updating.

  • The intended approach was to retain both the responsibility for producing the update within the authors, and editorial support for within the CRGs.

AHRQ 2012 Annual Conference


Methods2

Methods

  • One-year project (July 2010 to July 2011)

  • Involved 6 CRGs

  • Each CRG offered 37 days’ of assistance for updating 1 to 6 Cochrane Reviews (between November 2010 and April 2011)

  • Tasks performed by reviewers from inside and outside Cochrane without specific topic knowledge

AHRQ 2012 Annual Conference


Tasks offered to crgs and authors

Tasks offered to CRGs and authors

AHRQ 2012 Annual Conference


Results assessment of tasks

Results: assessment of tasks

AHRQ 2012 Annual Conference


Results crg and author feedback

Results: CRG and author feedback

  • Evaluation form: all CRGs responded, most authors responded (1 on holiday, 1 no response)

  • Scale 1-5 (1 greatly improved, 5 greatly worsened)

  • All authors and managing editors who responded to the question on rolling out an updating service for The Cochrane Collaboration were in favour of an updating service (2 CRGs did not respond)

AHRQ 2012 Annual Conference


Results status of updates

Results: status of updates

  • We worked on 14 Cochrane Review updates

  • Mean time spent: 35.5 hours (median 26.71 hours, range 4.5 to 109.75 hours)

  • Lots of activity but....

  • In October 2011, no updates yet published

AHRQ 2012 Annual Conference


Future challenges

Future challenges:

  • Whose priorities?

  • Expectations and the increasing professionalisation of systematic reviews

  • The rise and rise of methods advances

  • How to balance updates versus new reviews?

  • Authorship of updates

AHRQ 2012 Annual Conference


Conclusion

Conclusion

  • Updating is critical to patient safety and credibility

  • Increasing recognition of need to prioritise (but this isn’t as easy as it seems)

  • Transparency to user/reader is critical

  • Different approaches but no “one size fits all” solution

AHRQ 2012 Annual Conference


Conclusion how can we work more effectively together

Conclusion: How can we work more effectively together?

  • Sharing intelligence

    • “Keep up” initiative

    • Shared surveillance

  • Sharing the workload

    • Sharing data

    • “Wiki” approaches

  • Novel approaches

    • Use of data mining and semantic technologies

    • Limited search and “Summary updates”

AHRQ 2012 Annual Conference


Meeting the challenge of maintaining cochrane reviews

thank you for [email protected]

AHRQ 2012 Annual Conference


Authorship of updates

Authorship of updates

  • Scenario:

    A review is updated that originally included 11 studies and now has 17. The text includes 35% the same text as the previous version, across all sections. In addition a summary of findings table has been added.The author team has changed completely and none of the previous authors have contributed to the update.

  • Questions

    How should the work of the previous authors be credited?

AHRQ 2012 Annual Conference


Icmje guidance on contributorship

ICMJE guidance on contributorship

Authorship credit should be based on

  • Substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data;

  • Drafting the article or revising it critically for important intellectual content; and

  • Final approval of the version to be published. Authors should meet conditions 1, 2, and 3.

AHRQ 2012 Annual Conference


Authorship of updates1

Authorship of updates

  • As authors of the updated review

  • Lead author included on the update

  • Should be negotiated on an individual basis

  • Original authors not included as authors but acknowledged within the review

  • Some other approach

AHRQ 2012 Annual Conference


  • Login