Critical Appraisal Skills quantitative reviews

1. Critical Appraisal Skillsquantitative reviews Pippa Orr Knowledge Support Librarian With acknowledgements to CASP for their slides North Cumbria Informatics Service

2. Critical Appraisal Skills Programme (CASP) Critical appraisal is the process of weighing up evidence to see how useful it is in decision making http://www.phru.nhs.uk/casp/critical_appraisal_tools.htm North Cumbria Informatics Service

3. Effectiveness of Health Care • doing the right thing • to the right patient • in the right way • at the right time • at the right cost • in the right place North Cumbria Informatics Service

4. Kinds of evidence • Descriptive • cross-sectional, longitudinal • Analytic • case-control study • cohort study • Experimental • randomized controlled trial North Cumbria Informatics Service

5. Hierarchy of evidence North Cumbria Informatics Service

6. Why does good evidence from research fail to get into practice?? - 75% cannot understand the statistics - 70% cannot critically appraise a research paper Using Research for Practice: A UK Experience of the barriers scale Dunn V, Crichton C, Williams K, Roe B, Seers K North Cumbria Informatics Service

7. Critical appraisal helps the reader of research ………... • Decide how trustworthy a piece of research is (validity) • Determine what it is telling us (results) • Weigh up how useful the research will be (relevance) North Cumbria Informatics Service

8. Primary Research Evidence: Randomised Controlled Trials (RCT) • Robust randomisation procedures: • to ensure that the variables are equal in both groups • to remove all bias • to ensure that the results are generalisable North Cumbria Informatics Service

9. Randomised controlled trial new treatment group 1 Outcome population Outcome group 2 control treatment North Cumbria Informatics Service

10. Blinding • Blinding = participants don’t know what intervention they are getting • Double blinding = those giving the intervention don’t know what the participant is receiving North Cumbria Informatics Service

11. Loss to follow-up It is important to ensure that all those that are randomised into the trial are followed up to the trial’s conclusion North Cumbria Informatics Service

12. Intention to treat analysis Analysing people, at the end of the trial, in the groups to which they were randomised, even if they did not receive the intended intervention. North Cumbria Informatics Service

13. Types of review: Reviews Systematic reviews Meta-analysis North Cumbria Informatics Service

14. Publication bias • Papers with "interesting" results are (or may be) more likely to be: • submitted for publication • accepted for publication • published in a major journal and in English Language • quoted by authors • quoted in newspapers North Cumbria Informatics Service

15. Odds Ratio, Relative Risk Measures of risk The likelihood of something happening V The likelihood of something not happening North Cumbria Informatics Service

16. Odds ratio (OR) • The odds of an event happening in the experimental group expressed as a proportion of the odds of an event happening in the control group • The closer the OR is to 1, the smaller the difference in effect, i.e. no effect: OR = 1 North Cumbria Informatics Service

17. Confidence intervals/ limits • Presents the range of likely effects • The 95% confidence interval, for example, includes 95% of results from studies of the same size and design in the same population • This is close, but not identical, to saying that the true size of effect (never exactly known) has 95% chance of falling within the confidence interval • The narrower/ shorter the confidence interval, the more precise/ confident we can be about the estimate North Cumbria Informatics Service

18. Forest plots • Common approach to presenting the results of a meta-analysis • Also known as a ‘blobbogram’ or ‘odds ratio diagram’ • Graphical representation of individual trial results included in a review, together with the combined meta-analysis result North Cumbria Informatics Service

19. line of no effect confidence interval meta-analysis result North Cumbria Informatics Service

20. p-value • The probability (ranging from 0 to 1) that the results observed in a study (or results more extreme) could have occurred by chance if in reality the null hypothesis was true, ie if you did nothing. • If this probability is less than 1/20 (which is when the p value is less than 0.05), then the result is conventionally regarded as being “statistically significant”. North Cumbria Informatics Service

21. The p-value in a nutshell Could the result have occurred by chance? The result is likely to be due to chance The result is unlikely to be due to chance 0 1 p < 0.05 a statistically significant result p > 0.05 not a statistically significant result p = 0.05 or 1 in 20 result fairly unlikely to be due to chance p = 0.5 or 1 in 2 result quite likely to be due to chance 1 20 1 2 North Cumbria Informatics Service

22. Number needed to treat Is the number of people you would need to treat with a specific intervention to see one additional occurrence of a specific beneficial outcome. North Cumbria Informatics Service

23. Critical appraisal:questions to apply to reviews • is it trustworthy? validity • what does it say? results • will it help? relevance North Cumbria Informatics Service