Critical Appraisal

1. Critical Appraisal Vicki Pilkington

2. Why should I care? Why should I care? Interpreting research For academia and clinical practice For your BSc Referencing, Literature Reviews Discussing the limitations of your own project For your career Letter to the editor > PMID AFP interview skills

3. What is Critical Appraisal? 1. What is this study? Question, design 2. What has it found?Results, statistics, applications 3. Can I trust it?Good methods? Does it apply to me?

4. Types of study – The Hierarchy of Evidence Systematic Review + Meta Analysis Experimental Randomised Controlled Trials Observational INCREASING QUALITY OF EVIDENCE Cohort Studies Case-Control Studies Cross sectional Studies Case Reports, Case Series Editorials, Expert Opinion

5. Overview Overview Progress Bar 1 Question and RelevanceWhat is the study question and why are we asking it? Basic Design Who? What? Where? How? Results The data interpretation bit Internal Validity How good are the methods/study design? External Validity How well do the results apply to real life? Ethics Was it right to conduct thus study? Conclusions What can we conclude from this study, bearing in mind the findings, robustness and applicability 2 3 4 5 6 7

6. Overview Overview 1 Question and RelevanceWhat is the study question and why are we asking it? Basic Design Who? What? Where? How? Results The data interpretation bit Internal Validity How good are the methods/study design? External Validity How well do the results apply to real life? Ethics Was it right to conduct thus study? Conclusions What can we conclude from this study, bearing in mind the findings, robustness and applicability 2 3 4 5 6 7

7. 1. Question and Relevance QUESTION Read the title! What is this study asking? RELEVANCE Widespread disease Impactful disease Expensive disease Topical disease 1 2 3 4 5 6 7

8. 2. Basic Design 2. Basic Design P Population Who are we studying? I Intervention What drug/management? C Comparison Control, Comparator drug O Outcome What are we measuring? S Study type Where on the hierarchy of evidence? 1 2 3 4 5 6 7

9. 2. Basic Design – Outcome Types Primary Outcomes Main outcome of interest Secondary Outcomes Other – commonly adverse events Surrogate Markers Easier/quicker/cheaper to measure Should reflect actual outcome well Composite versus SingleSingle = Mortality Composite = Mortality/MI/CVD/TEE 1 2 3 4 5 6 7

10. 3. Basic Design - Study Type What difference is the study looking for? B 1 A 2 3 B A 4 5 6 A B 7

11. Overview 1 Question and Relevance What is the study question and why are we asking it? Basic Design Who? What? Where? How? Results The data interpretation bit Internal Validity How good are the methods/study design? External Validity How well do the results apply to real life? Ethics Was it right to conduct thus study? Conclusions What can we conclude from this study, bearing in mind the findings, robustness and applicability 2 3 4 5 6 7

12. 3. Results – Key Results RandomisationSimilar baseline characteristics between trial arms. Any differences between groups statistically significant? E ffectsHas the intervention had any overall effect on the outcome? Effect estimate – expressed as ARR, RR, OR, NNT etc Significance = p value (or crossed null value) Specificity = Confidence intervals DropoutHow many patients finished compared to planned size? Different between arms? Power? 1 2 3 4 5 6 7

13. 3. Results – Interpreting the Stats P value Probability that the study result has arisen by chance 95% CI Range of values within which one can be 95% certain that the TRUE population value lies Narrow is more precise, wide is less precise Wide tends may suggest there havent been enough cases in the analysis Type 1 error False Positive > affected by high p value Type 2 error False Negative > affected by low sample size Power Probability of correctly detecting a difference between two arms of a trial when such a difference really exists = 1-type 2 error(%). Conventionally 80% = 80% chance of detecting that treatment A is significantly better than treatment B, if treatment A actually is better Power Calculation = to determine how many participants needed for the study to achieve power needed > Depends on expected incidence of outcome of interest + thresholds of statistical significance 1 2 3 4 5 6 7

14. 3. Results – Effect Estimates ARR/RD Absolute Risk Reduction /Risk Difference 1 Mortality Rate from MI 3/100 1/100 2 = 2% (0.02) 3 3% 4 1% Statin Placebo 5 6 7

15. 3. Results – Effect Estimates 6 ARR/RD Absolute Risk Reduction /Risk Difference RRRelative Risk/Risk Ratio > Used in RCT/Cohort 1 Mortality Rate from MI 3/100 1/100 2 3 3% 9 4 1% Statin Placebo 5 6 RR = 0.01/0.03 = 0.33 (33%) = 67% reduction in risk on statins 7

16. 3. Results – Effect Estimates 6 Null value 1 +++ 0 ARR/RD Absolute Risk Reduction /Risk Difference RRRelative Risk/Risk Ratio > Used in RCT/Cohort OR Odds Ratio > Used in Case control 1 2 Positive Association 3 9 4 Negative Association 5 6 No Association 7

17. 3. Results – Effect Estimates 6 ARR/RD Absolute Risk Reduction /Risk Difference RRRelative Risk/Risk Ratio > Used in RCT/Cohort OR Odds Ratio > Used in Case control HR Hazard Ratio > Difference in survival over time 1 2 3 9 4 5 6 7

18. 3. Results – Effect Estimates 6 B ARR/RD Absolute Risk Reduction /Risk Difference RRRelative Risk/Risk Ratio > Used in RCT/Cohort OR Odds Ratio > Used in Case control HR Hazard Ratio > Difference in survival over time NNT Number needed to treat > How many people need to take a drug before one person sees a benefit = 1/ARR 1 2 3 9 4 5 6 7

19. 3. Results – Example Randomisation E ffects Dropout 1 2 3 4 5 6 7

20. 3. Results – Uni/Multivariate Analysis 1 2 3 4 This is Univariate Analysis, and it suggests that treatment A is much better However, this doesn’t account for any difference in the ages of the participants in either treatment arm 5 6 7

21. 3. Results – Uni/Multivariate Analysis 1 2 3 4 5 This is Multivariate Analysis, and it suggests that both treatments are equal The apparent difference found by Univariate Analysis was due to differences in the ages of participants in each arm This is a way of assessing for confounding factors (age) and stratifying the data out to account for differences in baseline characteristics 6 7

22. Overview Overview 1 Question and Relevance What is the study question and why are we asking it? Basic Design Who? What? Where? How? Results The data interpretation bit Internal Validity How good are the methods/study design? External Validity How well do the results apply to real life? Ethics Was it right to conduct thus study? Conclusions What can we conclude from this study, bearing in mind the findings, robustness and applicability 2 3 4 5 6 7

23. 4. Internal Validity – How good is the design? 4. Internal Validity HypothesisShould be clearly predefined with good rationale > avoid data dredging Ideally register the protocol in advance > avoid publication bias Outcome Hard vs soft/surrogate? Clinically relevant? Subjective vs Objective? Population Representative of population of interest? Inclusion/Exclusion criteria – clearly defined and replicable Endemic country? Primary/secondary/tertiary care? Statistics Were appropriate statistical methods used? Significances? Power? LossDropout - How many patients finished compared to planned size? Power? Different between arms – if so, why? ITT = all patients in treatment arm regardless of if they got the treatment OT – only patients in treatment arm who received treatment Ideally <20% LTFU otherwise ask why? Allocation Were patients randomised to different treatments? How - cluster, individual, order of recruitment – any bias? Blinding Blinded? Double blinded? Different arms treated equally throughout trail? 1 2 3 4 5 6 7

24. 4. Internal Validity - Randomisation IndividualFlip a coin, computer based Block Randomisation Random order within a fixed block of size Ensures equal numbers in each study arm throughout study Cluster RandomizationCenters/Whole communities randomised Eg. For community level interventions, education – try to ensure similar characteristics for each center 1 2 3 4 5 6 7

25. 4. Internal Validity - Biases 1 2 3 4 5 6 7 Confounding factors = independent risk factors for outcome of interest

26. 4. Internal Validity – How good is the design? 4. Internal Validity Hypothesis Outcome Population Statistics Loss Allocation Blinding 1 2 3 4 5 6 7

27. Overview Overview 1 Question and Relevance What is the study question and why are we asking it? Basic Design Who? What? Where? How? Results The data interpretation bit Internal Validity How good are the methods/study design? External Validity How well do the results apply to real life? Ethics Was it right to conduct thus study? Conclusions What can we conclude from this study, bearing in mind the findings, robustness and applicability 2 3 4 5 6 7

28. 5. External Validity B roader contextWhat already exists? Compared with placebo or existing standard of care? How important is the disease? How much difference will this make? Applicability How does it apply to clinical practice – will it help patients? Will patients take it/prefer it? Acceptability = adherence + compliance Policy – cost, health systems equipped? All relevant clinical outcomes considered? Cost/benefit: Benefits eg patient health, society (reduced healthcare cost/ economic benefit of healthy population) Costs – financial, environment, safety GeneralisabilityPopulation – Representative eg.Coomorbidities, age etc. Setting - Where is treatment usually given eg. Primary versus tertiary care - Disease endemic regions Intervention – Feasible to give to patients? Eg Logistics and cost 1 2 3 4 5 6 7

29. Overview Overview 1 Question and Relevance What is the study question and why are we asking it? Basic Design Who? What? Where? How? Results The data interpretation bit Internal Validity How good are the methods/study design? External Validity How well do the results apply to real life? Ethics Was it right to conduct thus study? Conclusions What can we conclude from this study, bearing in mind the findings, robustness and applicability 2 3 4 5 6 7

30. 6. Ethics + Funding Reason Was there a good reason to do the study? Was it needed? Is it a question that is relevant? Is the answer important? EquipoiseDid we truly not have evidence of answer without this trial? ApprovalEthics committee – Countries of site and analysis ConsentFreely, given, informed consent Consider perks of being in study eg. Access to healthcare TrackingIs there is data and safety monitoring board Wo can independently stop study early if necessary FundingFinancial conflict of interest eg. Pharmaceuticals, corporations vs NGO – should have no role in design/randomisation 1 2 3 4 5 6 7

31. Overview Overview 1 Question and Relevance What is the study question and why are we asking it? Basic Design Who? What? Where? How? Results The data interpretation bit Internal Validity How good are the methods/study design? External Validity How well do the results apply to real life? Ethics Was it right to conduct thus study? Conclusions What can we conclude from this study, bearing in mind the findings, robustness and applicability 2 3 4 5 6 7

32. 7. Conclusions Summarise This is a ____ study, looking at the effects of _____compared to _____ on ______, which found…. Key strengths Eg. Relevant, large, multicenter Key limitations Eg. Power? Loss to follow up? Specific population? How will you use it? This study alone is unlikely to change clinical practice, but these findings contribute to the larger body of literature in this area, and more research is warranted. 1 2 3 4 5 6 7

33. Overview Overview 1 Question and RelevanceWhat is the study question and why are we asking it? Basic Design Who? What? Where? How? Results The data interpretation bit Internal Validity How good are the methods/study design? External Validity How well do the results apply to real life? Ethics Was it right to conduct thus study? Conclusions What can we conclude from this study, bearing in mind the findings, robustness and applicability 2 3 4 5 6 7

34. 1 Feedback! 2 https://forms.gle/joDvW3qiogTcVQs57 3 4 5 6 7