How to do cochrane systematic reviews introductory workshop
Download
1 / 96

How to do Cochrane systematic reviews: Introductory workshop - PowerPoint PPT Presentation


  • 130 Views
  • Uploaded on

How to do Cochrane systematic reviews: Introductory workshop. Dr Jeremy Grimshaw Canadian Cochrane Network and Centre University of Ottawa. Introductions Introduction to systematic reviews Introduction to The Cochrane Collaboration Steps in doing a Cochrane review Coffee Break

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' How to do Cochrane systematic reviews: Introductory workshop' - solada


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
How to do cochrane systematic reviews introductory workshop

How to do Cochrane systematic reviews: Introductory workshop

Dr Jeremy Grimshaw

Canadian Cochrane Network and Centre

University of Ottawa


Outline of the course

  • Introductions

  • Introduction to systematic reviews

  • Introduction to The Cochrane Collaboration

  • Steps in doing a Cochrane review

    Coffee Break

  • Advanced methods

    • Appraising the risk of bias

    • Locating and selecting studies for your systematic review

Outline of the course


Introductions

  • Name

  • Background

  • Previous experience of systematic reviews

  • What you want to get out of the training?

Introductions


Session 1 outline

Session 1: Outline


Problems of information management in healthcare

  • Over 20,000 health journals published per year

  • Published research of variable quality and relevance

  • Research users (consumers, health care professionals and policy makers) often poorly trained in critical appraisal skills

  • Average time professionals have available to read = <1 hour/week

Problems of information management in healthcare


Systematic reviews

Systematic reviews


Systematic reviews1

  • A review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review.

    Cochrane Collaboration (2005) Glossary of Terms in The Cochrane Collaboration

Systematic reviews


Meta analysis

Meta-analysis


Meta analysis1
Meta-analysis integrate the results of included studies.


Systematic reviews2

Systematic reviews involve: integrate the results of included studies.

  • stating the objectives of the research

  • defining eligibility criteria for studies to be included

  • identifying (all) potentially eligible studies

  • applying eligibility criteria

  • assembling the most complete dataset feasible

  • analysing this dataset, using statistical synthesis and sensitivity analyses, if appropriate and possible

  • preparing a structured report of the research.

Systematic reviews


Systematic reviews3

  • Reduce bias integrate the results of included studies.

  • Reduce random error

  • Explore variability

  • Provide reliable basis for making decisions

  • Inform and influence future research

Systematic reviews


Traditional vs systematic reviews

Traditional integrate the results of included studies.

lack rigor

methodology not transparent

different reviewers reach different conclusions

become out of date

Systematic

scientific rigor to minimise bias

explicit and reproducible methodology

regularly updated (Cochrane)

Traditional vs systematic reviews


The cochrane collaboration

“It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials”.

Archie Cochrane, 1979

The Cochrane Collaboration


The prototype effective care in pregnancy and childbirth
The Prototype - Effective Care in Pregnancy and Childbirth have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials”.


The cochrane collaboration1

The Cochrane Collaboration is an international organization that aims to help people make well-informed decisions about healthcare by preparing, maintaining, and promoting the accessibility of systematic reviews of the effects of healthcare interventions.

The Cochrane Collaboration


Structure of The Cochrane Collaboration that aims to help people make well-informed decisions about healthcare by preparing, maintaining, and promoting the accessibility of systematic reviews of the effects of healthcare interventions.

Centres

Review Groups

Steering Group

Consumer Network

Fields

Methods Groups


Cochrane Centres that aims to help people make well-informed decisions about healthcare by preparing, maintaining, and promoting the accessibility of systematic reviews of the effects of healthcare interventions.

  • Promote awareness and support of The Cochrane Collaboration on a geographical basis

    • support individual review authors

    • support review groups in that area

    • provide a unique function e.g. software development, consumer support

    • link to government and other agencies

    • not a production house for reviews


Canadian cochrane centre staff
Canadian Cochrane Centre staff that aims to help people make well-informed decisions about healthcare by preparing, maintaining, and promoting the accessibility of systematic reviews of the effects of healthcare interventions.


Cochrane Review Groups that aims to help people make well-informed decisions about healthcare by preparing, maintaining, and promoting the accessibility of systematic reviews of the effects of healthcare interventions.

  • International and multidisciplinary

  • Produce reviews

  • Focused around health problems

  • 51, covering most of health care

  • Editorial base facilitates process


Cochrane review groups

  • Worldwide there are over 11,000 review authors working with 51 registered review groups

  • 5 review groups have their editorial base in Canada

    • Back Group

    • Effective Practice and Organisation of Care Group (EPOC)

    • Hypertension Group

    • Inflammatory Bowel Disease and Functional Bowel Disorders Group

    • Musculoskeletal Group

Cochrane Review Groups


Cochrane Fields 51 registered review groups

  • Represents a population, group, or type of care that overlaps multiple review group areas

    • E.g., primary care, health promotion and public health, health care of older people

    • Child Health Field and the Health Equity Field are located in Canada

  • Ensures that their priorities are reflected in the work of review groups


  • Cochrane Methods Groups 51 registered review groups

    • Develops methodology and advise the Collaboration on how the validity and precision of systematic reviews can be improved

    • Examples: Statistical Methods, Non-Randomized Studies, Information Retrieval

    • Reporting Bias Methods Group is located in Canada


    Cochrane consumer network

    • Facilitates the dissemination of information to patients, their families, friends and advocates throughout the world

    • Plays an integral part to all activities within The Cochrane Collaboration

    • Supports the role of consumers within The Cochrane Collaboration

    • Liz Whamond (NB) current chair of The Cochrane Consumer Network

    Cochrane Consumer Network


    Cochrane reviews

    Cochrane reviews


    Steps of a cochrane systematic review

    STEP 1: formulate the problem and register the title with CRG

    STEP 2: write protocol, submit for peer review and publish

    STEP 3: locate and select studies

    STEP 4: critically appraise studies for risk of bias

    STEP 5: collect data

    STEP 6: analyse and present results

    STEP 7: interpret results and write review

    STEP 8: improve and update review

    Steps of a Cochrane systematic review


    Title registration

    • Decide which Cochrane CRG would be responsible CRG

      • relevant CRG may be obvious

      • most groups are disease (or organ) focused

      • if unsure which CRG, contact CCN/C for advice

    • Information on CRGs

      • The Cochrane Library

      • www.cochrane.org

    • Contact CRG directly

    • Complete title registration form (from CRG)

    Title registration


    Tips for titles

    Tips for titles


    Why have a protocol

    Why have a protocol?


    Format of a cochrane protocol

    • Standard format CRG

      • to help review authors be systematic

      • to help people reading the reviews to find information quickly

    Format of a Cochrane protocol


    Format of a cochrane protocol1

    Format of a Cochrane protocol



    Background handbook section 3

    Background (Handbook Section 3)


    Objectives handbook section 3

    Objectives (Handbook Section 3)


    Lumping and splitting

    • Systematic review question can be framed broadly or narrowly:

      • Effects of audit and feedback to improve chronic diseases across all health care settings and professionals

      • Effects of audit and feedback to improve chronic diseases within primary care

      • Effects of audit and feedback to improve diabetes care within primary care

    Lumping and splitting


    Lumping and splitting1

    • The ‘lumping’ rationale - systematic reviews aim to identify the common generalisable features within similar interventions, minor differences in trial design are not important.

    • ‘Lumped’ reviews allow generalisability and consistency of findings to be assessed across wide range of settings and populations – this reduces risk of bias or chance results

    • The ‘lumping principle’ is that the results of two interventions should be combined unless there are good grounds to believe they will have opposing effects.

    Lumping and splitting


    Lumping and splitting2

    • The ‘splitting’ rationale is that it is only appropriate to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording.

    • Split reviews avoid combining ‘apples and oranges’

    • Reviews can be split by participants, interventions or outcome

    • Very narrowly focused reviews are de facto subgroup analyses

    Lumping and splitting


    Lumping and splitting3

    Practical considerations to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording.

    • Lumped reviews

      • Logistically challenging (large number of included studies)

      • Analytically challenging

      • Heterogeneity expected

      • Interpretation maybe challenging – seeing the woods for the trees

    • Split reviews

      • Easier, quicker, cleaner

    Lumping and splitting


    Lumping and splitting4

    • Effects of audit and feedback (85 studies) to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording.

    • Effects of audit and feedback to improve chronic diseases across all health care settings and professionals (18 studies)

    • Effects of audit and feedback to improve chronic diseases within primary care (14 studies)

    • Effects of audit and feedback to improve diabetes care within primary care (3 studies)

    Lumping and splitting


    Selection criteria handbook section 4

    • Follow naturally from the objectives to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording.

      • type of participants

      • type of interventions

      • type of outcomes

      • type of studies

    Selection criteria (Handbook Section 4)


    Purpose of selection criteria

    • Help you design the search to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording.

    • Selecting studies

    • Start thinking about the analysis

    • One way to minimise bias

    Purpose of selection criteria


    Small group exercises
    Small group exercises to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording.


    Writing the methods and completing the protocol handbook sections 6 7 8

    Writing the methods and completing the protocol to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording. (Handbook sections 6,7,8)


    Purpose of methods handbook sections 6 7 8

    • Plan what you will do before you start to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording.

    • Divide work among review authors and establish timeline

    • Minimise bias

    • Start thinking about the analysis

    Purpose of methods (Handbook sections 6,7,8)


    Methods handbook sections 6 7 8

    • Inclusion of trials to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording.

      - how?

      - by who?

    Methods (Handbook sections 6,7,8)


    Methods handbook sections 6 7 81

    • Assessment of risk of bias to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording.

      - type of assessment

      - by who?

      - how to incorporate in the review?

    Methods(Handbook sections 6,7,8)


    Methods handbook sections 6 7 82

    • Data extraction to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording.

      - what will be extracted?

      - by who?

      - how to deal with missing data?

    Methods(Handbook sections 6,7,8)


    Methods handbook sections 6 7 83

    • Data synthesis to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording.

      - what type of synthesis to do?

      - what summary measures to use?

      - what comparisons to make?

      • plans to investigate for differences in study results (heterogeneity) or potential for missing studies (publication bias)

      • plans for subgroup or sensitivity analyses?

    Methods(Handbook sections 6,7,8)


    Writing your protocol

    • Make sure all sections are completed to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording.

      - title/review authors, background, objectives, selection criteria, search strategy, methods

    • Don’t forget the other bits and pieces

      - contact details of review authors, acknowledgements, conflict of interest, sources of support, references, additional tables

    Writing your protocol


    Resources for cochrane reviewers

    • The Cochrane Handbook to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording.

    • Open learning support

    • Revman

    • Support from CRGs:

      • Trial search co-ordinators

      • Review group co-ordinators

      • Contact editor

    • Support from CCN/C

      • Training

      • Whatever else we can do!

    Resources for Cochrane reviewers


    After protocol is complete

    • Submit to CRG for editorial process to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording.

    • Internal and external peer review 1-2 months

    • May receive critical comments to respond to

    • Protocol published on The Cochrane Library

    • Reward yourself then start the review

    • Note - if you need to change your plan – report it and justify it (be transparent)

    After protocol is complete


    COFFEE! to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording.


    Session 2 outline

    • Locating and selecting studies for your to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording. systematic review

    • Appraising the risk of bias

    Session 2: Outline


    Locating and selecting studies for your systematic review

    Locating and selecting studies for your to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording. systematic review


    Why is a rigorous approach to searching important

    Why is a rigorous approach to searching important


    Finding as much as possible and being strategic

    Use multiple sources and start with the highest yield of the effects of interventions

    • Electronic databases

      • Cochrane Review Group register

      • Cochrane Central Register of Controlled Trials (CENTRAL)

      • MEDLINE

      • EMBASE

      • Other databases – depending on your topic

    • Reference lists of trials and reviews

    • Handsearching journals and conferences

    • Personal communication

    Finding as much as possible and being strategic


    Designing a search strategy

    Designing a search strategy


    Formulate the question

    Formulate the question


    Searching tip

    Searching tip


    Documenting your search

    Documenting your search


    Additional points

    Additional points


    Minimising bias when searching

    • Don’t limit search on resources

      • language

      • source (which journal, database etc)

      • publication status (full article, conference abstract, unpublished etc)

      • publication date

    Minimising bias when searching


    Inclusion and exclusion of studies

    Inclusion and exclusion of studies



    What is bias

    What is bias?


    Bias vs quality
    Bias vs Quality design or conduct of a trial

    Bias

    Quality

    Systematic error influencing

    trial estimate of the ‘true’

    effect of an intervention

    Based on the conduct of the

    trial; did trialists go to sufficient

    lengths to minimise potential bias?


    Why assess susceptibility to bias

    • To determine whether biases could have affected the results design or conduct of a trial

    • Studies with less susceptibility to bias better estimate the ‘truth’ about the effect of an intervention

    • Biased primary studies that provide misleading results could generate misleading systematic reviews

    Why assess susceptibility to bias?



    Biases design or conduct of a trial

    Selection

    Performance

    Detection

    Attrition


    Biases design or conduct of a trial

    Target Population

    Allocation

    Selection

    Intervention group

    Control group

    Performance

    Exposed to

    intervention

    Not exposed

    to intervention

    Detection

    Outcome

    Outcome

    Attrition

    Follow up

    Follow up


    Biases design or conduct of a trial

    Critical appraisal

    Allocation

    Intervention group

    Control group

    Blinding of

    investigator/

    subject

    Exposed to

    intervention

    Not exposed

    to intervention

    Blinding of

    outcome

    assessment

    Outcome

    Outcome

    Intention-to-treat

    Follow up

    Follow up

    Follow up

    Target Population

    Selection

    Random sequence

    Allocation

    concealment

    Performance

    Detection

    Attrition


    Biases design or conduct of a trial

    Allocation

    Intervention group

    Control group

    Exposed to

    intervention

    Not exposed

    to intervention

    Outcome

    Outcome

    Follow up

    Follow up

    Target Population

    Selection

    Performance

    Detection

    Attrition


    Selection bias

    Intervention group design or conduct of a trial

    Control group

    • systematic differences in participantcharacteristics at the start of a trial

    Selection bias


    Selection bias1

    Selection bias


    Selection bias2

    Adequate allocation concealment design or conduct of a trial

    • central computer randomisation

    • on-site computer from which assignment can only be determined after entering patient data

    • serially numbered, sealed, opaque envelopes

      Inadequate allocation concealment

    • non opaque envelopes

    • odd or even date of birth or medical record number

    Selection bias


    Biases design or conduct of a trial

    Allocation

    Intervention group

    Control group

    Exposed to

    intervention

    Not exposed

    to intervention

    Outcome

    Outcome

    Follow up

    Follow up

    Target Population

    Selection

    Performance

    Detection

    Attrition


    Performance bias

    • Occurs during the intervention phase design or conduct of a trial

    • Systematic differences, other than the intervention being investigated, in the treatment of the two groups

    • Participant expectations influencing their response to treatment

    • Avoiding performance bias

      - blinding the provider

      - blinding the participant

    Performance bias

    Schulz KF & Grimes DA 2002 Lancet


    Allocation concealment vs blinding

    Randomisation design or conduct of a trial

    Time

    Concealment

    of allocation

    Blinding

    Selection bias

    Performance bias

    Allocation concealment vs blinding


    Biases design or conduct of a trial

    Allocation

    Intervention group

    Control group

    Exposed to

    intervention

    Not exposed

    to intervention

    Outcome

    Outcome

    Follow up

    Follow up

    Target Population

    Selection

    Performance

    Detection

    Attrition


    Detection bias

    • Occurs during the follow up phase design or conduct of a trial

    • Systematic differences in the outcome assessmentbetween groups

    • Avoiding detection bias

      - blinding the outcome assessor

      - blinding data analyst/investigator?

    Detection bias


    Biases design or conduct of a trial

    Allocation

    Intervention group

    Control group

    Exposed to

    intervention

    Not exposed

    to intervention

    Outcome

    Outcome

    Follow up

    Follow up

    Target Population

    Selection

    Performance

    Detection

    Attrition


    Attrition bias

    • Occurs over the duration of follow up design or conduct of a trial

    • Systematic differences in the loss of participants to follow up between groups

    • Completeness of follow up

      - participants lost to follow up/ not included in outcome assessment, could be different from those who remained in the trial

    • avoiding attrition bias

      - describe proportion of participants lost to follow-up

      - use intention-to-treat analyses

    Attrition bias


    Attrition bias1

    • Intention-to-treat (ITT) analysis design or conduct of a trial

      - “once randomised, analysed”

      - regardless of whether participants:

      • received the intervention

      • deviated from the protocol

      • withdrew from the study

        - maintains randomised allocation of participants

    Attrition bias



    Empirical evidence that bias affects results

    Selection bias results?

    Inadequate/unclear allocation concealment associated with exaggerated treatment effect (average 29% increased benefit)1,2,3,4

    Performance bias

    ‘Open trials’ (no blinding) exaggerate benefits of treatment (average 14% increased benefit) 1,2,3,4

    Empirical evidence that bias affects results

    • Schulz et al, JAMA 1995

    • Moher et al, Lancet, 1998

    • Kjaergard et al, Ann Intern Med, 2001

    • Egger et al, HTA, 2003

    • Juni et al, JAMA, 1999


    Empirical evidence that bias affects results1

    Detection bias results?

    Open outcome assessment exaggerated treatment effects (17%1 and 35%5)

    Attrition bias

    Handling of dropouts did not affect results1,5 however, compared whole group differences (not between group differences). Not an exclusion but important to report.

    • Schulz et al, JAMA 1995

    • Moher et al, Lancet, 1998

    • Kjaergard et al, Ann Intern Med, 2001

    • Egger et al, HTA, 2003

    • Juni et al, JAMA, 1999

    Empirical evidence that bias affects results


    Assessing susceptibility to bias

    Assessing susceptibility to bias


    Beware of quality scales

    • Different scales are available (>50) results?

      - vary in dimensions covered, size, complexity

      - some provide composite numerical score

    • Reliability and validity of many scales not established

    • May lead to different conclusions regarding the susceptibility to bias of the same trial (Juni 1999)

    Beware of quality scales….


    Treatment estimates from results?

    high-quality trials

    Treatment estimates from

    low-quality trials

    Same group of trials

    divided into high and

    low quality using

    25 different Quality

    Assessment Scales

    • Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999;282(11):1054-60


    How to assess susceptibility to bias

    How to assess susceptibility to bias


    Assessing susceptibility to bias in reviews

    • Was the allocation sequence adequately generated (randomised)?

    • Was the allocation adequately concealed?

    • Was there blinding of participants, personnel

    • Was there blinding of outcome assessors?

    • Were dropouts and exclusions adequately addressed?

    • Other (including selective outcome reporting and other criteria for addressing topic-specific or design-specific biases)

    Assessing susceptibility to bias in reviews



    Incorporating susceptibility to bias in meta analysis

    • Exclude trials with high susceptibility to bias meta-analysis?

      - critical a priori cutoff for inclusion (e.g. only trials with adequate allocation concealment or blinding)

      - may be problematic if few trials

    • Sensitivity analysis

      - analyse separately above and below a certain a priori level and compare results (does it change conclusions?)

      - for instance, analyse the impact of allocation concealment, blinding, use of ITT analysis, on results

    Incorporating susceptibility to bias in meta-analysis



    But beware

    • Scant empirical evidence about which biases are important meta-analysis?

    • Reporting of susceptibility to bias is not always the same as the susceptibility to bias in the conduct of the trial

      - allocation concealment unclear in 87% of trials as per published report (Pildal et al, 2005)

      - allocation concealment unclear in about 80% of trials (Middleton 2001)

    • CONSORT statement (1994, 2001) www.consort-statement.org

    • Possible discrimination against older trials

    But beware…


    Important points for your review

    • Plan criteria and how you will meta-analysis?incorporate susceptibility to bias at the protocol stage

    • Assessment undertaken once trials are included and prior to data extraction

    • 2 independent review authors

    Important points for your review


    ad