Cohort Study Evaluation

1. Cohort Study Evaluation Irina Ibraghimova ibra@zadar.net

2. Article • Introducing evidence based medicine to the journal club, using a structured pre and post test: a cohort study • BMC Medical Education 2001;1(1):6 • http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pubmed&pubmedid=11710973

3. Abstract • BackgroundJournal Club at a University-based residency program was restructured to introduce, reinforce and evaluate residents understanding of the concepts of Evidence Based Medicine. • MethodsOver the course of a year structured pre and post-tests were developed for use during each Journal Club. Questions were derived from the articles being reviewed. Performance with the key concepts of Evidence Based Medicine was assessed. Study subjects were 35 PGY2 and PGY3 residents in a University based Family Practice Program. • ResultsPerformance on the pre-test demonstrated a significant improvement from a median of 54.5 % to 78.9 % over the course of the year (F 89.17, p < .001). The post-test results also exhibited a significant increase from 63.6 % to 81.6% (F 85.84, p < .001). • ConclusionsFollowing organizational revision, the introduction of a pre-test/post-test instrument supported achievement of the learning objectives with a better understanding and utilization of the concepts of Evidence Based Medicine.

4. 12 questions • Three broad issues need to be considered when appraising a cohort study: • A - Are the results of the study valid? • B - What are the results? • C - Will the results help locally? • There are 12 questions that are designed to help you think about these issues systematically

5. Screening Questions • The first two questions are screening questions and can be answered quickly. If the answer to those two is “yes”, it is worth proceeding with the remaining questions

6. A.1.Did the study address a clearly focused question? YES – NO – Can’t Tell A question can be focused in terms of: • the population studied • the risk factors studied • the outcomes considered • is it clear that the study tried to detect a beneficial or a harmful effect?

7. Did the study address a clearly focused question? • My answer is - Yes • Study subjects were 35 family residents in a University based Family Practice Program • The study tried to detect a beneficial effect

8. A.2. Did the authors use an appropriate method to answer their question? YES – NO – Can’t Tell • Consider: - is a cohort study a good way of answering the question under circumstances? - did it address the study question?

9. Did the authors use an appropriate method to answer their question? My answer is – YES If it is difficult (impossible) to provide a randomized controlled trial the next best study design is a cohort study

10. Is it worth continuing to read the article? My answer is – YES The next 10 are detailed questions

11. The detailed questions are about: • Study design • Study conduct • Analyses • Interpretation

12. A.3.Was the cohort recruited in an acceptable way? YES-NO-Can’t tell We are looking for selection bias which might compromise the generalisability of the findings: • Was the cohort representative of a defined population? • Was there something special about the cohort? • Was everybody included who should have been included?

13. Was the cohort recruited in an acceptable way? • My answer is – YES • Participants were all family medicine residents in a University-based residency program. The PGY1 information was collected but not included in the report, as they were not available at the initiation of the revised Journal Club in Module 10 of the 13 Module academic year.

14. A.4.Was the exposure accurately measured to minimize bias? • My answers is -Yes • In order to measure the progress in meeting the objectives of the revised Journal Club, a written pre-test and post-test was chosen as a primary method of evaluating each Journal Club session. With the Pre-test there is direct feedback regarding comprehension and competence

15. A.5.Were the outcomes accurately measured to minimize bias ? • My answer is – Yes • a written pre-test and post-test was chosen as a primary method of evaluating each Journal Club session

16. A.6.Have the authors identified all important confounding factors? • Yes • No • Don’t know • My answer is - Don’t know

17. A.7.Was the follow up of subjects complete and long enough? • Yes • No • Don’t know

18. Was the follow up of subjects complete and long enough? • My answer is – No

19. B.8. What are the results of the study? • the introduction of a pre-test/post-test instrument supported achievement of the learning objectives with a better understanding and utilization of the concepts of Evidence Based Medicine

20. B.9. How precise are the results? • The overall Pre to Post-test difference reached significance (F 2.04, p =.046).

21. B.10. Do you believe the results? • Yes • No • Don’t know

22. Do you believe the results? • My answer is - Yes

23. C.11. Can the results be applied to local population ? • Yes • No • Don’t know

24. Can the results be applied to local population ? • My answer is - Don’t know

25. C.12.How do the results compare with previous reports? • When critical appraisal skills among resident physicians have been formally assessed, correct responses have varied between 33 and 42 % suggesting limited integration of these concepts - the same was reported in this institution with an overall average of 32% for incoming PGY1 residents over the preceding three years. The attainment of a Pre-test median score of 79.0 % at the end of the first year of our intervention was certainly comparable and significantly exceeded those results