Why do we read research articles?

1. To increase our knowledge in a particular area of interest • To search for evidence for our practice • To review literature in preparation for a research Why do we read research articles?

2. Should we believe everything we read?

3. OPST 199 Methods of Research Critical Appraisal Kristofferson G. Mendoza, PTRP Department of Physical Therapy College of Allied Medical Professions University of the Philippines Manila 27 May 2009

4. Learning Objectives At the end of the session, the learner should be able to: • Identify the level of evidence presented in a research study on the basis of the research design used • Discuss theoretical and methodological aspects of a research study in relation to internal and external validity • Rate the usefulness of evidence presented in a research study using key criteria

5. Critical Appraisal the process of systematically examining research evidence to assess its validity, results and relevance before using it to inform a decision.

6. Levels of Evidence OPST 199 Methods of Research Critical Appraisal

7. Levels of Evidence Various study designs have different levels of rigour

8. Classic Levels of Evidence • By Sackett et al. • For Studies on intervention, prevention, etiology and harm • RCT the most rigorous study design • Modification of the classic level of evidence later on included systematic reviews bec of the proliferation of RCTs

9. (minus) Prospective Data Collection (minus) Randomization (minus) Clinical data (minus) Comparison Groups Level 2 Level 3 Level 4 Level 5 HIGH Level 1 LOW

10. Levels of Evidence • For studies on Prognosis  • optimal individual study – individual inception cohort study with greater than 80% ff up • Where the clinical decision rule has been validated in a single population • Clinical decision rule are algorithms or scoring systems that lead to prognosis estimation or diagnostic category

11. Levels of Evidence • For studies on Diagnosis • optimal individual study – cohort study with good reference standards or clinical decision rule tested within one clinical center

12. Levels of Evidence • For studies on Differential diagnosis / Symptom prevalence • Cohort study with good follow up

13. Levels of Evidence • Despite differences in the optimal study design, certain consistencies are evident across different types of questions • A systematic review of high-quality studies always provides the highest level of rigor • An individual study using the optimal design for that type of clinical question is considered level 1 • Prospective data collection indicates higher study quality than retrospective data collection • Expert opinion, bench research, conceptual framework/theories/first principles are always considered the lowest (level 5) evidence

14. Other Levels of evidence Classification System Greenhalgh (1997)

15. Levels of evidence For quantitative studies Primary Research Experimental Observational For qualitative studies? No consensus about the relative rigour of different methods

16. Research design issues OPST 199 Methods of Research Critical Appraisal

17. Research Design Issues in Quantitative Research • Sampling of Subjects Objective: A sample that represents the target population Optimal Design: A random sample selected from the population Threats to Validity: Differential sampling between groups will affect internal validity; convenience sampling may affect external validity

18. Research Design Issues in Quantitative Research • Sample Size Objective: The optimal number of subjects is large enough to detect important treatment effects but small enough to conduct the study in a timely, efficient manner Optimal Design: Correct number of subjects is determined by an accurate sample size calculation Threats to Validity: Small sample size

19. Research Design Issues in Quantitative Research • Allocation of treatment Objective: Unbiased allocation Optimal Design: Random allocation Threats to Validity: Nonrandom allocation (observational)

20. Research Design Issues in Quantitative Research • Blinding (to treatment allocation and/or outcomes Objective: To minimize sources of bias introduced by study personnel or participants Optimal Design: Ideally everyone involved in the study would be blinded to the full extent possible Threats to Validity: Blinding in rehab is difficult; may result in bias from differential diagnosis, outcome assessments, attention and follow-up procedures

21. Research Design Issues in Quantitative Research • Outcome ascertainment Objective: To accurately determine pretreatment status; To reflect all important changes in outcome post-treatment Optimal Design: Outcomes measured at all likely relevant time points; Outcomes measured using reliable, valid, and responsive measures Threats to Validity: important effects will be missed if relevant time points are not assessed or the outcome measures are not sensitive; outcome measures that reflect only certain domains may be biased towards specific treatments; poor reliability/validity may invalidate the conclusion

22. Research Design Issues in Quantitative Research • Follow-up Objective: To accurately portray the treatment effects obtained by all the participants Optimal Design: 100% follow-up Threats to Validity: differential loss to follow-up can introduce bias in estimate of effects

23. Research Design Issues in Quantitative Research • Statistical analysis Objective: To provide accurate estimates of the size and significance of the observed effects Optimal Design: Accurate and appropriate analysis of all data Threats to Validity: Inappropriate analysis may lead to faulty conclusion

24. Critical review form Law, M., Stewart, D., Pollock, N., Letts, L., Bosch, J., & Westmorland, M., 1998. McMaster University www.fhs.mcmaster.ca/canchild OPST 199 Methods of Research Critical Appraisal

26. Appropriateness of Study Design • Knowledge of topic/issue • If little is known, more exploratory (case study, cross-sectional design) • as level of knowledge increases, study design must be more rigorous (RCT most rigorous) • Outcomes • If outcomes are easily quantifiable and well-developed standardized assessment tools are available, design should be rigorous • If outcomes are not fully understood yet (e.g., quality of life), use design that explore different factors (case control)

27. Appropriateness of Study Design • Ethical Issues: • If there is no ethical issue re: withholding treatment, design should have control groups • Study purpose/questions • Effectiveness of treatment: RCTs, before-after design, single-case studies • Learn more about an issue, pilot study to determine treatment: case control an cross sectional

28. Biases • Sample/Selection Bias • Volunteer bias – usually favors treatment group • Seasonal bias – could work either way • Attention bias – favors treatment group (extra attention)

29. Biases • Measurement/Detection Bias • Number of outcome measures used • Only one outcome measure – could favor either groups • Too many outcome measures for sample size – favors control group • Lack of masked or independent evaluation – treatment group is usually favored • Recall or memory bias – favors treatment group

30. Biases • Intervention/Performance Bias • Contamination – favors control group • Co-intervention – can influence either groups • Timing of intervention • Site of treatment • Different therapists

35. Thank you OPST 199 Methods of Research Critical Appraisal