S. Charles, Medical Sciences, KU School of Medicine–Wichita

1. Summary of Results: • Fifty percent of the SPs who completed the certification program increased their inter-rater reliability from the previous year. • Seventy percent of those SPs who did not receive training experienced a decrease in their inter-rater reliability decrease. • For the overall MS III assessment, the inter-rater reliability of the SPs increased by 87% in the second year. • Conclusions: • The inter-rater reliability increased significantly following the implementation oft the SP certification program. • The creation of the SP certification program enhanced the inter-observer reliability and validity of the SP/student encounter. • Future studies will include a self analysis of SP responses to student encounters. • Background: • A trained standardized patient (SP) provides feedback to students enhancing the students’ communication skills. • Inter-observer reliability and validity among SPs in these encounters is key to standardizing feedback. • Objective: • Develop an intervention to increase inter-rater reliability. • Methodology: • During the MS III Clinical Skills Assessment, SPs completed checklists of requirements and provided feedback during end-of-year assessments. • The director of the CACL and trained faculty observers evaluated the feedback from the SPs and completed the same requirements checklist of the students. • We determined inter-rater reliability and validity as being able to compare these checklists within a single standard deviation from the SPs’ original interaction. • Additional inter-rater reliability was validated by industrial psychologists with psychometric training and expertise. • This study tracked the reliability and validity among SPs in MS III end-of-year assessments over a two year period. • Methodology (Cont.): • After determining inter-observer reliability could be improved, a 90-hour certification program was developed: • Educational theory, communication skills and feedback/evaluation methods coursework (30 hours) • Physical examination skills coursework (30 hours) • Film review and evaluation (15 hours) • Mentorship for SPs with less seniority (15 hours) • Each SP was filmed pre- and post-training. • One group of SPs was not trained and 35 SPs were trained. • Analysis of inter-observer reliability was completed by the director of the CACL, trained faculty observers, and industrial psychologists. What is the impact of a certificate program for Standardized Patients (SPs) on the end-of-year assessments? S. Charles, Medical Sciences, KU School of Medicine–Wichita D. Schocken, Center for Advanced Clinical Learning (CACL), USF College of Medicine wichita.kumc.edu