Critical appraisal Systematic R eview. กิตติพันธุ์ ฤกษ์ เกษม ภาควิชาศัลยศาสตร์ มหาวิทยาลัยเชียงใหม่. Systematic review vs Meta-analysis. Systematic review: a systematic approach to minimising bias and error
Critical appraisal Systematic Review กิตติพันธุ์ ฤกษ์เกษม ภาควิชาศัลยศาสตร์ มหาวิทยาลัยเชียงใหม่
Systematic review vs Meta-analysis • Systematic review: a systematic approach to minimising bias and error • Meta-analysis: a statistical analysis, which aim to produce a single estimate of a treatment effect • Systematic review may or may not include Meta-analysis
Why do we need systematic review? • Theearly1980suncomplicatedMI “Shouldpt receive a beta-blockerforsecondarypreventionbeforedischarge?” Library:4randomisedcontrolledtrials (RCT)
Beta-blocker vs placebo RCT 1.Mortality and hospital readmission is not different RCT 2. Not conclusive RCT 3. Beta-blocker not shown benefit RCT 4. Long term beta-blocker reduces the mortality and rate of re-infarction
A review in BMJ 1981 • There is no clear evidence that beta-blocker improves long tem survival after MI despite almost 20 yrs of clinical trials • Good enough!!
Another review in European Heart Journal 1981 “itseemsperfectlyreasonabletotreatpatientswhohavesurvivedaninfarctionwithbeta-blocker”
Limitation of a single study • Too small sample size false negative
Problem of Conventional review • Prone to bias and error • Select only evidence support the author’s view • Not specify methodological quality of studies • Finally choose most vote ignore sample size, and design
Meta-analysis = combining all available data • Attractive alternative to such large, expensive and problematic study • Weight average of the result large > small trial
Meta-analysis Beta-blocker trials - MI Beta-blocker better Placebo better
Cumulative meta-analysis Significant effect from 1980 onwards (OR not across 1
Meta-analysis Beta-blocker trials - MI Maybe Unnecessary trials Beta-blockerbetter Placebobetter
Benefit • Estimate the overall effect • Examine different result between studies (heterogeneity) • Identified insufficient data
Cochrane collaboration • International organisation of health care profession • Promoting accessibility of systematic review • Foster development of systematic review • 50 collaborative review groups www.cochrane.org/cochrane/ccweb.htm
Potentials of systematic review • Good • Bad
Systematic review Basic structure and types กิตติพันธุ์ ฤกษ์เกษม ภาควิชาศัลยศาสตร์ มหาวิทยาลัยเชียงใหม่
Like primary research • Why- Introduction, background • How-method • What we found-result • What it mean-discussion
Basic structure • Abstract • Introduction • Background • Objectives • Method “treat a paper like a patient in 1 reseach” • Type of studies • Inclusion criteria type of participants • Exclusion criteria • Type of intervention • Type of outcome measures • Search strategy for identification of study • Method of analysis
Basic structure • Result (special diagram) • Conclusion • Reference
Forest plot Stroke rate ‘LA vs GA in carotid sx trial’ 1966-2001 LA better GA better Blacksquare= OR, horizontalline = 95%confidenceinterval Areaofblacksquare = weight, diamond = combinedORwith 95%CI
Types • Systematic review of primary research • Observational studies • Diagnostic screening • RCT
The process (1) • Research question • Writing protocol • Searching • Article retrieval • Literature review
The process (2) • Inclusion/ exclusion criteria • Validity and quality of articles • Data extraction/ synthesis • Interpretation
The question • Is local anesthesia is better than general anesthesia during carotid endarterectomy?
Writing the protocol • Background • Objectives • Type of studies • Inclusion criteria • Type of participants • Exclusion criteria • Type of intervention • Type of outcome measures • Search strategy for identification of study • Method of analysis • Reference
Searching • Medline • Other database • Hand searching the literature • Writing to people
Getting the article • Which ones to get? • It takes time • Libraries • Inter-library Loans
Literature review • Youdon’thavetoreadthewholepaperyet! • Translation
Validity and quality of articles • Do read the paper and see what the author thought was wrong • Unequal intervention/control size • Hidden loss to follow up
Data extraction • Read method carefully • Design a form
Synthesis/ Interpretation • Estimates and confidence intervals pool effect make by statistic method e.g. Peto method (fix method) give more weight effect for large study than small study (P value) • Difference between studies (Heterogeneity) Chi-squared test (P value)
Critical Appraisal 1. Are the result valid? 2. What are the results?
1. Are the result valid? • Did this review address a sensible clinical question • Was the search for relavant studies detailed and exhaustive? • Were selection and assessment of studies reproducible? • Were the primary studies of high methodological quality?
Publication bias “A (significant) beneficialtreatmenteffectarepublished, butanequalresultremainunpublished” • Ingeneralmedicaljournalandpublicheathjournalreportedstatisticallysignificant 85.4% • Inpsychologicaljournal 95.6%
Time lag bias • “Positiveresultwilldominatetheliteratureforseveralyearuntilthenegativewillreportlater” • HIVtrialinUSA, median timetopublishofpositiveresult 4.2years, butnegativeresult 6.4years
Duplicate publication bias • “ onestudypresentsandreportsseveraltimes” “ includethisleadtooverestimationoftreatmenteffect” • Ondersetrontopreventpostoperativenauseavomitting 16studies 3duplicatedpapers • Sometimesdifficulttosay, sincenotsharesinglecommonauthors!!!!!!
Language bias • “Authors tend to report positive result in international papers, English language journal, but if negative result are published in local journal”
Outcome reporting bias • In trials many outcome is recorded but only favorable finding will be reported • Clinical trials by drug companies, unpublished trials gave information on adverse effect > published trials
Selection Bias • Tend to happen in non Randomised controlled trials (non RCT) • For example select low risk group to new treatment group
2. What are the result? • Were the results similar from study to study? If yes, the credit of single estimates is OK. • Point estimates similar? • Overlapping confidence interval • Test for heterogeneity? (Chi square test) • Percentage of variability (I2 ) good < 20%, concern 20-50%,serious concern > 50%
Look overlapping confidence interval Rerkasem Cochrane Database Syst Rev 2008; (4):CD000126.
Small RCTs show LA is marginal lower mortality than GA Rerkasem Cochrane Database Syst Rev 2008 (4):CD000126.
2. What are the result? • Were the results similar from study to study? • What are the overall results of the reviews? • How precise were the results?