assessing study quality for a systematic review n.
Skip this Video
Loading SlideShow in 5 Seconds..
Assessing study quality for a systematic review PowerPoint Presentation
Download Presentation
Assessing study quality for a systematic review

Loading in 2 Seconds...

play fullscreen
1 / 37

Assessing study quality for a systematic review - PowerPoint PPT Presentation

  • Uploaded on

Assessing study quality for a systematic review. Trudy Bekkering , PhD Center of Evidence-Based Medicine & Belgian Branch of the Dutch Cochrane Center Centre for Methodology of Educational Research Katholieke Universiteit Leuven, Belgium. Why is it so important?.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Assessing study quality for a systematic review' - lupita

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
assessing study quality for a systematic review

Assessing study quality for a systematic review

Trudy Bekkering, PhD

Center of Evidence-Based Medicine &

Belgian Branch of the Dutch Cochrane Center

Centre for Methodology of Educational Research

KatholiekeUniversiteit Leuven, Belgium

why is it so important
Why is it so important?
  • Meta-analysis aims to increase precision
  • Meta-analysis of studies with bias in results gives very precise but wrong results
  • Garbage in, garbage out
bias versus imprecision
Bias versus imprecision


  • A systematic error in the results or the inferences
  • Methodologicalflaw
  • Overestimation or underestimation
bias versus imprecision2
Bias versus imprecision


  • A random error in the results
  • Sample variation
  • Direction of error is random
bias versus imprecision3
Bias versus imprecision


Ideal study

bias versus imprecision4
Bias versus imprecision


bias versus imprecision5
Bias versus imprecision


risk of bias versus bias
Risk of bias versus bias
  • Clearempiricalevidencethatparticularflaws in study design can lead to bias.
  • Usuallyimpossibletoknowtowhatextentbiaseshave affected the results.
  • Keyconsideration = should the resultsbebelieved
tools for assessing study quality
Tools for assessing study quality

Scales: discouraged

Checklist: between 3 and 57 items

Cochrane tool: “domain based”

Depends on study design

tool for rcts cochrane tool
Tool for RCTs: Cochrane tool

Risk of bias on 6 domaines:

  • Sequencegeneration
  • Allocationconcealment
  • Blinding
  • Incomplete outcome data
  • Selectivereporting
  • Other sources of bias
blinding of intervention
Blinding of intervention
  • Participants (patients, clients)
  • Care providers (doctors, nurses, teachers …)
  • Outcome assessors
incomplete outcome data
Incomplete outcome data

“Attrition” (drop-out): no data

  • Withdrawal
  • Do notattend follow-up appointment
  • Failure to complete questionnaire / diaries
  • Cannotbelocated (lost to follow-up)
  • Decisionby investigator tocease follow-up
  • Data or records are lost
incomplete outcome data1
Incomplete outcome data

“Exclusion”: data available, but excluded from analysis

  • Participants found to be ineligible after enrolment
  • An “as treated” (or per-protocol) analysis:participants are only included if they received the intended intervention in accordance with the protocol
selective outcome reporting
  • = selection of a subset of the variables recorded for inclusion in publication,
  • on the basis of the results
  • For example:
  • Omission of non-significant outcomes
  • Choice of data for an outcome (e.g. osteoporosis)
  • Choice of analysis (e.g. blood pressure)
  • Reporting of subsets of data (e.g. sepsis)
  • Under-reporting of data (e.g. only “not significant”)
presentation in your review
Presentation in your review

“Risk of bias graph”

assessing risk of bias in nrs
Assessing risk of bias in NRS
  • Selection bias (how was groupallocation?)
  • Performance bias (blinding, fidelity of interventions)
  • Attrition bias (completeness of sample & follow-up)
  • Reporting bias (selectiveoutcomereporting)
  • Confoundingandadjustment



Imbalance in prognostic factors

Comparison intervention - control

Intervention versus control

Difference in outcome


Confounding factor

Association between 2 factors

Presence of risk factor

Occurrence of outcome

confounding adjustment
Confounding & adjustment
  • At the stage of protocol: list potential confounding factors
  • Identify the factors the authors have considered and omitted
  • Assess balance between groups at baseline
  • What did authors do to control for confounders (matching, restricting to subgroups, stratification, regression modelling)
tool for nrs
Tool for NRS

Downs and Black instrument

(J Epidemiol Community Health 1998;52:377-84)

27 items:

  • Reporting (10)
  • Externalvalidity (applicability) (3)
  • Internalvalidity - bias (7)
  • Internalvalidity – confounding (6)
  • Power (1)
tool for nrs1
Tool for NRS

Newcastle-Ottawa Scale (NOS)

(Wells 2008)

8 items covering 3 perspectives:

  • Selection of studygroups
  • Comparibility of groups
  • Ascertainement of exposure (case-control) or outcome (cohort)


diagnostic studies
Diagnostic studies


Whiting BMC Medical Research Methodology 2003;3:25

Cochrane version: 11 items (out of 14 original)

Diagnostic Test Accuracy Working Group: handbook


other risk of bias assessment tools
Other risk of bias assessment tools

SIGN (Scottish Intercollegiate Guidelines Network)


in summary
In summary
  • Risk of bias assessment is essential for systematic reviews
  • For RCT: use the Cochranetool
  • For NRS:
    • Higher risk of bias (selection bias & reporting bias)
    • Use the appropriate tool toassess risk of bias
    • Considerhowpotentialconfounders are addressed