How to do cochrane systematic reviews introductory workshop
This presentation is the property of its rightful owner.
Sponsored Links
1 / 96

How to do Cochrane systematic reviews: Introductory workshop PowerPoint PPT Presentation


  • 94 Views
  • Uploaded on
  • Presentation posted in: General

How to do Cochrane systematic reviews: Introductory workshop. Dr Jeremy Grimshaw Canadian Cochrane Network and Centre University of Ottawa. Introductions Introduction to systematic reviews Introduction to The Cochrane Collaboration Steps in doing a Cochrane review Coffee Break

Download Presentation

How to do Cochrane systematic reviews: Introductory workshop

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


How to do cochrane systematic reviews introductory workshop

How to do Cochrane systematic reviews: Introductory workshop

Dr Jeremy Grimshaw

Canadian Cochrane Network and Centre

University of Ottawa


Outline of the course

  • Introductions

  • Introduction to systematic reviews

  • Introduction to The Cochrane Collaboration

  • Steps in doing a Cochrane review

    Coffee Break

  • Advanced methods

    • Appraising the risk of bias

    • Locating and selecting studies for your systematic review

Outline of the course


Introductions

  • Name

  • Background

  • Previous experience of systematic reviews

  • What you want to get out of the training?

Introductions


Session 1 outline

  • Introduction to systematic reviews

  • Introduction to The Cochrane Collaboration

  • Rationale for developing a protocol

  • Format of a protocol

  • Resources to support reviewers

Session 1: Outline


Problems of information management in healthcare

  • Over 20,000 health journals published per year

  • Published research of variable quality and relevance

  • Research users (consumers, health care professionals and policy makers) often poorly trained in critical appraisal skills

  • Average time professionals have available to read = <1 hour/week

Problems of information management in healthcare


Systematic reviews

  • Systematic reviews use rigorous scientific methods to identify, assess and synthesise the worldwide available evidence

Systematic reviews


Systematic reviews1

  • A review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review.

    Cochrane Collaboration (2005) Glossary of Terms in The Cochrane Collaboration

Systematic reviews


Meta analysis

  • The use of statistical techniques in a systematic review to integrate the results of included studies.

    Cochrane Collaboration (2005) Glossary of Terms in The Cochrane Collaboration

Meta-analysis


Meta analysis1

Meta-analysis


Systematic reviews2

Systematic reviews involve:

  • stating the objectives of the research

  • defining eligibility criteria for studies to be included

  • identifying (all) potentially eligible studies

  • applying eligibility criteria

  • assembling the most complete dataset feasible

  • analysing this dataset, using statistical synthesis and sensitivity analyses, if appropriate and possible

  • preparing a structured report of the research.

Systematic reviews


Systematic reviews3

  • Reduce bias

  • Reduce random error

  • Explore variability

  • Provide reliable basis for making decisions

  • Inform and influence future research

Systematic reviews


Traditional vs systematic reviews

Traditional

lack rigor

methodology not transparent

different reviewers reach different conclusions

become out of date

Systematic

scientific rigor to minimise bias

explicit and reproducible methodology

regularly updated (Cochrane)

Traditional vs systematic reviews


The cochrane collaboration

“It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials”.

Archie Cochrane, 1979

The Cochrane Collaboration


The prototype effective care in pregnancy and childbirth

The Prototype - Effective Care in Pregnancy and Childbirth


The cochrane collaboration1

The Cochrane Collaboration is an international organization that aims to help people make well-informed decisions about healthcare by preparing, maintaining, and promoting the accessibility of systematic reviews of the effects of healthcare interventions.

The Cochrane Collaboration


How to do cochrane systematic reviews introductory workshop

Structure of The Cochrane Collaboration

Centres

Review Groups

Steering Group

Consumer Network

Fields

Methods Groups


How to do cochrane systematic reviews introductory workshop

Cochrane Centres

  • Promote awareness and support of The Cochrane Collaboration on a geographical basis

    • support individual review authors

    • support review groups in that area

    • provide a unique function e.g. software development, consumer support

    • link to government and other agencies

    • not a production house for reviews


Canadian cochrane centre staff

Canadian Cochrane Centre staff


How to do cochrane systematic reviews introductory workshop

Cochrane Review Groups

  • International and multidisciplinary

  • Produce reviews

  • Focused around health problems

  • 51, covering most of health care

  • Editorial base facilitates process


Cochrane review groups

  • Worldwide there are over 11,000 review authors working with 51 registered review groups

  • 5 review groups have their editorial base in Canada

    • Back Group

    • Effective Practice and Organisation of Care Group (EPOC)

    • Hypertension Group

    • Inflammatory Bowel Disease and Functional Bowel Disorders Group

    • Musculoskeletal Group

Cochrane Review Groups


How to do cochrane systematic reviews introductory workshop

Cochrane Fields

  • Represents a population, group, or type of care that overlaps multiple review group areas

    • E.g., primary care, health promotion and public health, health care of older people

    • Child Health Field and the Health Equity Field are located in Canada

  • Ensures that their priorities are reflected in the work of review groups


  • How to do cochrane systematic reviews introductory workshop

    Cochrane Methods Groups

    • Develops methodology and advise the Collaboration on how the validity and precision of systematic reviews can be improved

    • Examples: Statistical Methods, Non-Randomized Studies, Information Retrieval

    • Reporting Bias Methods Group is located in Canada


    Cochrane consumer network

    • Facilitates the dissemination of information to patients, their families, friends and advocates throughout the world

    • Plays an integral part to all activities within The Cochrane Collaboration

    • Supports the role of consumers within The Cochrane Collaboration

    • Liz Whamond (NB) current chair of The Cochrane Consumer Network

    Cochrane Consumer Network


    Cochrane reviews

    • Cochrane reviews are conducted using standardised rigorous approaches including detailed critical appraisal of all studies

    • Cochrane reviews are reported using standardised structure including abstract and consumer summary

    Cochrane reviews


    Steps of a cochrane systematic review

    STEP 1:formulate the problem and register the title with CRG

    STEP 2: write protocol, submit for peer review and publish

    STEP 3:locate and select studies

    STEP 4:critically appraise studies for risk of bias

    STEP 5:collect data

    STEP 6:analyse and present results

    STEP 7:interpret results and write review

    STEP 8:improve and update review

    Steps of a Cochrane systematic review


    Title registration

    • Decide which Cochrane CRG would be responsible

      • relevant CRG may be obvious

      • most groups are disease (or organ) focused

      • if unsure which CRG, contact CCN/C for advice

    • Information on CRGs

      • The Cochrane Library

      • www.cochrane.org

    • Contact CRG directly

    • Complete title registration form (from CRG)

    Title registration


    Tips for titles

    • Titles for Cochrane reviews have a set format

    • Format‘intervention’ for ‘problem’ in ‘category’

    • ‘Caffeine’ for ‘daytime sluggishness’ in ‘adults’(hypothetical for today)

    • Keep it short

    Tips for titles


    Why have a protocol

    • Systematic reviews are scientific research

    • Plan methods

    • Reduce bias

    • Access to peer review

    • Avoid duplication of effort

    • Rest of review follows

      • focus for thinking about review

      • planning and allocating tasks

    Why have a protocol?


    Format of a cochrane protocol

    • Standard format

      • to help review authors be systematic

      • to help people reading the reviews to find information quickly

    Format of a Cochrane protocol


    Format of a cochrane protocol1

    • Title/review authors

    • Background

    • Objectives

    • Selection criteria

    • Search strategy

    • Methods

    Format of a Cochrane protocol


    Background objectives and selection criteria

    Background, objectives and selection criteria


    Background handbook section 3

    • Description of the condition and its significance

    • Description of the intervention and its role in practice

    • How the intervention might work

    • Why it is important to do the review

    Background (Handbook Section 3)


    Objectives handbook section 3

    • Follow naturally from the background

    • What are the questions?

      • does it work?

      • is the effect consistent or variable?

    • Questions should be clearly framed

    Objectives (Handbook Section 3)


    Lumping and splitting

    • Systematic review question can be framed broadly or narrowly:

      • Effects of audit and feedback to improve chronic diseases across all health care settings and professionals

      • Effects of audit and feedback to improve chronic diseases within primary care

      • Effects of audit and feedback to improve diabetes care within primary care

    Lumping and splitting


    Lumping and splitting1

    • The ‘lumping’ rationale - systematic reviews aim to identify the common generalisable features within similar interventions, minor differences in trial design are not important.

    • ‘Lumped’ reviews allow generalisability and consistency of findings to be assessed across wide range of settings and populations – this reduces risk of bias or chance results

    • The ‘lumping principle’ is that the results of two interventions should be combined unless there are good grounds to believe they will have opposing effects.

    Lumping and splitting


    Lumping and splitting2

    • The ‘splitting’ rationale is that it is only appropriate to combine trials which are very similar in design, patient selection, intervention characteristics and outcome recording.

    • Split reviews avoid combining ‘apples and oranges’

    • Reviews can be split by participants, interventions or outcome

    • Very narrowly focused reviews are de facto subgroup analyses

    Lumping and splitting


    Lumping and splitting3

    Practical considerations

    • Lumped reviews

      • Logistically challenging (large number of included studies)

      • Analytically challenging

      • Heterogeneity expected

      • Interpretation maybe challenging – seeing the woods for the trees

    • Split reviews

      • Easier, quicker, cleaner

    Lumping and splitting


    Lumping and splitting4

    • Effects of audit and feedback (85 studies)

    • Effects of audit and feedback to improve chronic diseases across all health care settings and professionals (18 studies)

    • Effects of audit and feedback to improve chronic diseases within primary care (14 studies)

    • Effects of audit and feedback to improve diabetes care within primary care (3 studies)

    Lumping and splitting


    Selection criteria handbook section 4

    • Follow naturally from the objectives

      • type of participants

      • type of interventions

      • type of outcomes

      • type of studies

    Selection criteria (Handbook Section 4)


    Purpose of selection criteria

    • Help you design the search

    • Selecting studies

    • Start thinking about the analysis

    • One way to minimise bias

    Purpose of selection criteria


    Small group exercises

    Small group exercises


    Writing the methods and completing the protocol handbook sections 6 7 8

    Writing the methods and completing the protocol (Handbook sections 6,7,8)


    Purpose of methods handbook sections 6 7 8

    • Plan what you will do before you start

    • Divide work among review authors and establish timeline

    • Minimise bias

    • Start thinking about the analysis

    Purpose of methods (Handbook sections 6,7,8)


    Methods handbook sections 6 7 8

    • Inclusion of trials

      - how?

      - by who?

    Methods (Handbook sections 6,7,8)


    Methods handbook sections 6 7 81

    • Assessment of risk of bias

      - type of assessment

      - by who?

      - how to incorporate in the review?

    Methods(Handbook sections 6,7,8)


    Methods handbook sections 6 7 82

    • Data extraction

      - what will be extracted?

      - by who?

      - how to deal with missing data?

    Methods(Handbook sections 6,7,8)


    Methods handbook sections 6 7 83

    • Data synthesis

      - what type of synthesis to do?

      - what summary measures to use?

      - what comparisons to make?

      • plans to investigate for differences in study results (heterogeneity) or potential for missing studies (publication bias)

      • plans for subgroup or sensitivity analyses?

    Methods(Handbook sections 6,7,8)


    Writing your protocol

    • Make sure all sections are completed

      -title/review authors, background, objectives, selection criteria, search strategy, methods

    • Don’t forget the other bits and pieces

      -contact details of review authors, acknowledgements, conflict of interest, sources of support, references, additional tables

    Writing your protocol


    Resources for cochrane reviewers

    • The Cochrane Handbook

    • Open learning support

    • Revman

    • Support from CRGs:

      • Trial search co-ordinators

      • Review group co-ordinators

      • Contact editor

    • Support from CCN/C

      • Training

      • Whatever else we can do!

    Resources for Cochrane reviewers


    After protocol is complete

    • Submit to CRG for editorial process

    • Internal and external peer review 1-2 months

    • May receive critical comments to respond to

    • Protocol published on The Cochrane Library

    • Reward yourself then start the review

    • Note - if you need to change your plan – report it and justify it (be transparent)

    After protocol is complete


    How to do cochrane systematic reviews introductory workshop

    COFFEE!


    Session 2 outline

    • Locating and selecting studies for your systematic review

    • Appraising the risk of bias

    Session 2: Outline


    Locating and selecting studies for your systematic review

    Locating and selecting studies for your systematic review


    Why is a rigorous approach to searching important

    • To achieve up-to-date, relevant, unbiased systematic reviews of the effects of interventions

    • To overcome problems with limited searching

      • bias

      • lack of precision and generalisability

    Why is a rigorous approach to searching important


    Finding as much as possible and being strategic

    Use multiple sources and start with the highest yield

    • Electronic databases

      • Cochrane Review Group register

      • Cochrane Central Register of Controlled Trials (CENTRAL)

      • MEDLINE

      • EMBASE

      • Other databases – depending on your topic

    • Reference lists of trials and reviews

    • Handsearching journals and conferences

    • Personal communication

    Finding as much as possible and being strategic


    Designing a search strategy

    • Work with TSC of review group

    • Formulate your question (PICO)

    • Identify your search terms

    • Put it all together

    Designing a search strategy


    Formulate the question

    • Translate the clinical problem into a structured question and identify the key concepts

      • Participants

      • Interventions

      • Outcomes

      • Studies

    • Develop search terms (commonly based on participants, interventions, study design)

    Formulate the question


    Searching tip

    • Searching is often a matter of trying terms and seeing what result you get

    • Find additional terms from the citations you find that are helpful: how are they indexed; what are the keywords used?

    Searching tip


    Documenting your search

    • Important for you, so you can remember what you’ve done

    • Important for users of your review

    • Document

      • what you searched

      • when you searched

      • how you searched

    Documenting your search


    Additional points

    • No right time to stop searching; be guided by returns and resources

    • Ideas for search terms from citations you know of; try them and check returns

    • Use a referencing database to manage citations

    Additional points


    Minimising bias when searching

    • Don’t limit search on

      • language

      • source (which journal, database etc)

      • publication status (full article, conference abstract, unpublished etc)

      • publication date

    Minimising bias when searching


    Inclusion and exclusion of studies

    • Develop and pilot a form based on your selection criteria

    • Scan and retrieve possibly relevant studies

    • Decide if studies meet inclusion criteria and record reasons for exclusion

    • How reproducible are the decisions?

    • Done by two independent review authors

    • Beware of duplicates

    Inclusion and exclusion of studies


    Appraising the risk of bias

    Appraising the risk of bias


    What is bias

    • Bias is systematic (non-random) error arising from the design or conduct of a trial

    • Previously termed ‘methodological quality’; updated to ‘susceptibility to bias’ or ‘risk of bias’

    What is bias?


    Bias vs quality

    Bias vs Quality

    Bias

    Quality

    Systematic error influencing

    trial estimate of the ‘true’

    effect of an intervention

    Based on the conduct of the

    trial; did trialists go to sufficient

    lengths to minimise potential bias?


    Why assess susceptibility to bias

    • To determine whether biases could have affected the results

    • Studies with less susceptibility to bias better estimate the ‘truth’ about the effect of an intervention

    • Biased primary studies that provide misleading results could generate misleading systematic reviews

    Why assess susceptibility to bias?


    How to do cochrane systematic reviews introductory workshop

    • Which biases make a difference to trial results?


    How to do cochrane systematic reviews introductory workshop

    Biases

    Selection

    Performance

    Detection

    Attrition


    How to do cochrane systematic reviews introductory workshop

    Biases

    Target Population

    Allocation

    Selection

    Intervention group

    Control group

    Performance

    Exposed to

    intervention

    Not exposed

    to intervention

    Detection

    Outcome

    Outcome

    Attrition

    Follow up

    Follow up


    How to do cochrane systematic reviews introductory workshop

    Biases

    Critical appraisal

    Allocation

    Intervention group

    Control group

    Blinding of

    investigator/

    subject

    Exposed to

    intervention

    Not exposed

    to intervention

    Blinding of

    outcome

    assessment

    Outcome

    Outcome

    Intention-to-treat

    Follow up

    Follow up

    Follow up

    Target Population

    Selection

    Random sequence

    Allocation

    concealment

    Performance

    Detection

    Attrition


    How to do cochrane systematic reviews introductory workshop

    Biases

    Allocation

    Intervention group

    Control group

    Exposed to

    intervention

    Not exposed

    to intervention

    Outcome

    Outcome

    Follow up

    Follow up

    Target Population

    Selection

    Performance

    Detection

    Attrition


    Selection bias

    Intervention group

    Control group

    • systematic differences in participantcharacteristics at the start of a trial

    Selection bias


    Selection bias1

    • Occurs during allocation into comparison groups

    • Avoiding selection bias

      • random ‘unpredictable’ sequence

      • allocation concealment (protection of the sequence – most important)

    Selection bias


    Selection bias2

    Adequate allocation concealment

    • central computer randomisation

    • on-site computer from which assignment can only be determined after entering patient data

    • serially numbered, sealed, opaque envelopes

      Inadequate allocation concealment

    • non opaque envelopes

    • odd or even date of birth or medical record number

    Selection bias


    How to do cochrane systematic reviews introductory workshop

    Biases

    Allocation

    Intervention group

    Control group

    Exposed to

    intervention

    Not exposed

    to intervention

    Outcome

    Outcome

    Follow up

    Follow up

    Target Population

    Selection

    Performance

    Detection

    Attrition


    Performance bias

    • Occurs during the intervention phase

    • Systematic differences, other than the intervention being investigated, in the treatment of the two groups

    • Participant expectations influencing their response to treatment

    • Avoiding performance bias

      - blinding the provider

      - blinding the participant

    Performance bias

    Schulz KF & Grimes DA 2002 Lancet


    Allocation concealment vs blinding

    Randomisation

    Time

    Concealment

    of allocation

    Blinding

    Selection bias

    Performance bias

    Allocation concealment vs blinding


    How to do cochrane systematic reviews introductory workshop

    Biases

    Allocation

    Intervention group

    Control group

    Exposed to

    intervention

    Not exposed

    to intervention

    Outcome

    Outcome

    Follow up

    Follow up

    Target Population

    Selection

    Performance

    Detection

    Attrition


    Detection bias

    • Occurs during the follow up phase

    • Systematic differences in the outcome assessmentbetween groups

    • Avoiding detection bias

      - blinding the outcome assessor

      - blinding data analyst/investigator?

    Detection bias


    How to do cochrane systematic reviews introductory workshop

    Biases

    Allocation

    Intervention group

    Control group

    Exposed to

    intervention

    Not exposed

    to intervention

    Outcome

    Outcome

    Follow up

    Follow up

    Target Population

    Selection

    Performance

    Detection

    Attrition


    Attrition bias

    • Occurs over the duration of follow up

    • Systematic differences in the loss of participants to follow up between groups

    • Completeness of follow up

      - participants lost to follow up/ not included in outcome assessment, could be different from those who remained in the trial

    • avoiding attrition bias

      -describe proportion of participants lost to follow-up

      -use intention-to-treat analyses

    Attrition bias


    Attrition bias1

    • Intention-to-treat (ITT) analysis

      -“once randomised, analysed”

      -regardless of whether participants:

      • received the intervention

      • deviated from the protocol

      • withdrew from the study

        -maintains randomised allocation of participants

    Attrition bias


    How to do cochrane systematic reviews introductory workshop

    • What is the evidence for these biases influencing trial results?


    Empirical evidence that bias affects results

    Selection bias

    Inadequate/unclear allocation concealment associated with exaggerated treatment effect (average 29% increased benefit)1,2,3,4

    Performance bias

    ‘Open trials’ (no blinding) exaggerate benefits of treatment (average 14% increased benefit) 1,2,3,4

    Empirical evidence that bias affects results

    • Schulz et al, JAMA 1995

    • Moher et al, Lancet, 1998

    • Kjaergard et al, Ann Intern Med, 2001

    • Egger et al, HTA, 2003

    • Juni et al, JAMA, 1999


    Empirical evidence that bias affects results1

    Detection bias

    Open outcome assessment exaggerated treatment effects (17%1 and 35%5)

    Attrition bias

    Handling of dropouts did not affect results1,5 however, compared whole group differences (not between group differences). Not an exclusion but important to report.

    • Schulz et al, JAMA 1995

    • Moher et al, Lancet, 1998

    • Kjaergard et al, Ann Intern Med, 2001

    • Egger et al, HTA, 2003

    • Juni et al, JAMA, 1999

    Empirical evidence that bias affects results


    Assessing susceptibility to bias

    • How do we assess susceptibility to bias?

      - Scales

      - Checklists

    Assessing susceptibility to bias


    Beware of quality scales

    • Different scales are available (>50)

      - vary in dimensions covered, size, complexity

      - some provide composite numerical score

    • Reliability and validity of many scales not established

    • May lead to different conclusions regarding the susceptibility to bias of the same trial (Juni 1999)

    Beware of quality scales….


    How to do cochrane systematic reviews introductory workshop

    Treatment estimates from

    high-quality trials

    Treatment estimates from

    low-quality trials

    Same group of trials

    divided into high and

    low quality using

    25 different Quality

    Assessment Scales

    • Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999;282(11):1054-60


    How to assess susceptibility to bias

    • Component approach: assessment against individual criteria

    How to assess susceptibility to bias


    Assessing susceptibility to bias in reviews

    • Was the allocation sequence adequately generated (randomised)?

    • Was the allocation adequately concealed?

    • Was there blinding of participants, personnel

    • Was there blinding of outcome assessors?

    • Were dropouts and exclusions adequately addressed?

    • Other (including selective outcome reporting and other criteria for addressing topic-specific or design-specific biases)

    Assessing susceptibility to bias in reviews


    How to do cochrane systematic reviews introductory workshop

    • How do we incorporate susceptibility to bias in meta-analysis?


    Incorporating susceptibility to bias in meta analysis

    • Exclude trials with high susceptibility to bias

      -critical a priori cutoff for inclusion (e.g. only trials with adequate allocation concealment or blinding)

      -may be problematic if few trials

    • Sensitivity analysis

      -analyse separately above and below a certain a priori level and compare results (does it change conclusions?)

      -for instance, analyse the impact of allocation concealment, blinding, use of ITT analysis, on results

    Incorporating susceptibility to bias in meta-analysis


    How to do cochrane systematic reviews introductory workshop

    Example of sensitivity analysis


    But beware

    • Scant empirical evidence about which biases are important

    • Reporting of susceptibility to bias is not always the same as the susceptibility to bias in the conduct of the trial

      - allocation concealment unclear in 87% of trials as per published report (Pildal et al, 2005)

      -allocation concealment unclear in about 80% of trials (Middleton 2001)

    • CONSORT statement (1994, 2001) www.consort-statement.org

    • Possible discrimination against older trials

    But beware…


    Important points for your review

    • Plan criteria and how you will incorporate susceptibility to bias at the protocol stage

    • Assessment undertaken once trials are included and prior to data extraction

    • 2 independent review authors

    Important points for your review


  • Login