DISCUSSION

1. Table 1 “The medical school must ensure that before graduation a student will have demonstrated, to the satisfaction of the faculty, the ability to retrieve (from electronic databases and other resources), manage, and utilize biomedical information for solving problems and making decisions ...” Pre-intervention self-assessment of MEDLINE searching skills. Likert scale (1=very low, 5=very high) AAMC Medical School Objectives Project, June 1998 Screenshot: courseware site Table 1 Curricular implementation issues. from the AAMC MSOP report Table 2 Overlaying windows from the internal medicine courseware site. The bottom window shows the “course library” menu, with the options for the PubMed section selected. The top window shows part of a practice exam on PubMed. Use of the accompanying courseware site – number of visits per type of page. Table 2 Instructional implementation issues from the AAMC MSOP report “Residents must be able to investigate and evaluate their patient care practices, …Residents are expected to: …use information technology to manage information, access on-line medical information; and support their own education.” ACGME Outcome Project, February 1999 Teaching and Evaluating Informatics Skills During a Clinical Clerkship Shlomi Codish (codish@bgu.ac.il), Akiva Leibowitz, Baruch Weinreb Faculty of Health Sciences, Ben Gurion University of the Negev, Beer-Sheva, Israel INTRODUCTION Medical students are expected to demonstrate computer literacy and information fluency by the time they graduate. As residents, they are expected to apply these skills to patient care. The Association of American Medical Colleges (AAMC) and the Accreditation Council for Graduate Medical Education (ACGME) have both published objectives relating to medical informatics skills (see boxes). Medical students’ information-management skills are formed during their first clinical clerkships; they research patient-centered information using paper-based and electronic resources. This is a uniquely opportune time to teach medical informatics skills. This poster describes the integration of medical informatics skills in an internal medicine clerkship, applying implementation strategies recommended in the 1998 AAMC Medical School Objectives Program (MSOP) report. RESULTS During the academic year 2004, 64 students participated in internal medicine clerkships. They studied in 8 general internal medicine wards, in three hospitals. Baseline information A questionnaire distributed at the end of the third year found a low self-assessment of MEDLINE search skills, despite fairly high computer literacy and ownership: • AIMS • To implement an educational program to teach medical informatics skills in a clinical clerkship. • To apply the AAMC MSOP recommended strategies as noted on tables 1 and 2. • To develop a multi-dimensional evaluation of the educational program. DISCUSSION Before beginning clinical clerkships, student self-assessment of their MEDLINE search skills was low, despite high self-assessment of other computer skills, and a high computer ownership rate. We did not perform an objective assessment of MEDLINE search skills before the clerkship. Our intervention related directly to many of the 1998 AAMC MSOP Medical Informatics recommendations. Where possible we applied the “ideal state” recommendation. Computer-Assisted Learning interventions are often assessed according to: acceptability, usability, knowledge acquisition or behavioral change. Our results indicate success in the first three of these. The two assessment modes showed higher scores for informatics topics than for other topics, and were non-discriminatory. This could be the result of the topic, which is narrower in scope than general internal medicine, or flawed question design. This study has several limitations: Pre-intervention assessment of search skills was not objectively assessed. The scope of skills assessed was limited to search technique rather than finding the correct clinical answer. This was because of the time constraints of OSCE tests. Behavioral change, arguably the most important outcome of such an intervention, was not assessed. Our study shows that an integrated educational intervention to teach medical informatics skills during a clinical clerkship is effective. Medical informatics skill acquisition can be assessed as part of the general clerkship assessment. METHODS Our intervention consisted of several educational activities: Small group tutorial on the use of PubMed A two-hour tutorial and an additional two-hour practice session on the use of PubMed. Submission of a patient-related MEDLINE search. Critical evaluation and use of Medical web sites A two-hour tutorial on the evaluation of medical information on the Internet (e.g. the HON code and AMA guidelines). Tutorial on the use of medical databases available on the local network (e.g. Micromedex and the Cochrane collaboration) and the computer-assisted learning unit (e.g. DxR and the ACP Clinical Problem Solving Cases). Submission of a written evaluation of a medical web site. Development of a courseware site to accompany the clerkship A courseware web-site was developed to accompany the clerkship (see screen shot). Students were required to maintain a log of clinical encounters using the site. The web site log was analyzed for usage of material and links accessed. Evaluation Acceptability A questionnaire assessing self-evaluation of specific information fluency skills was administered before the clerkship. An additional questionnaire assessing student evaluation of the intervention was administered at the end of the clerkship. Usability Usability was monitored via the courseware site log. Use of educational resources was logged by the specific applications. Knowledge acquisition Questions about the content of the intervention were added to the final exam. Five percent of the final written exam was dedicated to medical informatics. A hands-on assessment of skills was performed by means of a dedicated OSCE station (see below). This comprised 5.3% of the OSCE score. Behavioral change Behavioral change was not assessed in this study. Acceptability Acceptability of the intervention: An questionnaire administered at the end of the course asked students to evaluate all interventions (scores on a 1-5 scale with 1=lowest, 5=highest). The average evaluation of the Medline seminars was 4.6, of the medical web seminars 4.0, of the computer based simulation 4.1, and of the courseware web site 3.9. Students open-ended comments were positive. Usability Usability was assessed by monitoring use of the electronic resources accessed via the courseware site. The courseware web site was accessed, on average, 1.2 times per day by each student during the clerkship. The most commonly used pages on the web site were (in decreasing order): Clinical log (2256 hits), forum (977 hits), clinical seminars (959 hits), recommended web sites (519 hits) and online tutorials (246 hits). BACKGROUND Medical students, residents and practicing clinicians have been shown to perform poorly in applying online medical resources to answering clinical questions (see accompanying web site for an annotated bibliography). Various forms of interventions, from short tutorials to workshops and full curricular change, have been used to improve this. The 1998 AAMC MSOP report includes detailed recommendations for implementation of medical informatics education in medical schools and describes initial and ideal states for educational initiatives (tables 1 and 2). The use of electronic resources for medical research is continuously increasing. There has been a discrepancy between self evaluation of search skills and demonstrated performance. One of the items on the annual AAMC questionnaire asks medical school graduates to rank their agreement with the statement that they “have the skill to carry out reasonably sophisticated searches of medical information databases”. The number of students answering “strongly agree” or “agree” to this question has been steady for the years 2002-4 at over 93%. Studies measuring this skill objectively, consistently demonstrate low success rate in answering clinical questions with the use of medical information databases. At the BGU medical school, the use of MEDLINE is taught in the pre-clinical years, a period during which actual MEDLINE use is limited. During clinical clerkships, students are required to research medical topics several times a day. The Objective Structured Clinical Examination (OSCE) evaluates clinical competence by focusing on outcome via observable behaviors. The OSCE is gaining popularity and is used by the National Board of Medical Examiners as part of the USMLE step 2 test. OSCE have been used to assess clinical skills such as interview, physical examination and test performance and interpretation. In this work we used the OSCE to test proficiency in the use of MEDLINE. • OSCE Station number 9 [to follow interview of an adult patient with abdominal pain] • What is the main problem of the patient you have just interviewed? • What is the corresponding MeSH term for this problem? How many alternative terms does it have? In how many different trees does it appear? • Search for randomized-controlled trials relating to the treatment of pancreatitis with antibiotics. Limit the results to match the patient demographics. Print the search results, including the search strategy. • Evaluation criteria • Correct answers to MeSH questions • Use of MeSH terms and subheadings • Use of Boolean operators • Age and gender limits CONCLUSIONS Development of information management skills has been established as an objective for both medical students (by the AAMC) and for residents (by the ACGME). The first clinical clerkship is a critical time during which these skills are developed. Teaching informatics themes during a clinical clerkship was well accepted by students and faculty. Following our educational intervention, students achieved a high level of information-seeking skills, demonstrated in both a hands-on test and on a written exam. Evaluation of informatics skills can be effectively incorporated into existing evaluation methods. Knowledge acquisition Projects submitted by students (MEDLINE search and web site evaluation) demonstrated good searching and evaluating skills. The five informatics questions on the final exam were non-discriminatory. There was no difference in informatics question scores between the students scoring overall grades in the lowest and highest quartiles. The average score for the informatics questions was 84% while the average score of the entire test was 76%. The OSCE station scores were higher than the average score for other OSCE stations: 88% ± 14.9. The correlation between the OSCE station score and the score of other stations was 0.515 (Pearson’s correlation, p<0.001). This poster, along with an annotated bibliography can be found online at http://medic.bgu.ac.il/homes/AMIA05/