HA Bazan MD 1 , M McDonough BA 1 , AB Marr MD 1 , CC Baker MD 1 , S Chauvin PhD 2

Does Clinical Performance Assessment Grounded in ACGME-Core Competencies Enhance Assessor Discrimination? A Two-Year Comparative Study HA Bazan MD1, M McDonough BA1, AB Marr MD1, CC Baker MD1, S Chauvin PhD2 1Department of Surgery, 2Office of Medical Education Research LSUHSC School of Medicine – New Orleans The Association for Surgical Education Salt Lake City, UT – 28 April 2009

Background Assessment of medical student performance is a complex undertaking: • Many variables involved: • Depends largely on the assessors’ training Differences between junior/senior residents and faculty • Assessor’s perspective and experience (at different times of their career) • Faculty, Resident • Junior/Senior • Type of Evaluation utilized • Context of evaluator’s experience with the student

Background Several studies have struggled with ways to improve the reliability of student ratings • Increasing number of questions in evaluation forms does not result in increased accuracy • Evaluation forms (no matter how sophisticated) do not solve the problem of faculty’s generalized view of a student’s performance • Faculty make an overall decision as to whether a particular student deserves an – H, HP, or a P – and then tend to score the performance variables to reflect this final grade Pulito AR, Donnelly MB, Plymale M. Factors in faculty evaluation of medical students’ performance.Medical Education 2007

Question Can we improve the reliability of student assessments? Assessment based on the six core Accreditation Council for Graduate Medical Education (ACGME) competencies

Six Core ACGME Competencies • Patient Care • Medical Knowledge • Practice-Based Learning and Improvement • Interpersonal and Communication Skills • Professionalism • Systems-Based Practice

Question Can we improve the reliability of student assessments? By adapting a commonly understood construct (ACGME competencies) for medical student evaluations – would this result in more consistent interpretation of the evaluation items and increased assessor discrimination?

Hypothesis A Clerkship performance assessment system based on six ACGME competencies would: • Enhance assessors’ discrimination of performance • Decrease the proportion of students receiving the highest performance level

Methods • Data for analysis: 321 consecutive third year medical students completing the surgery Clerkship at our medical school Two year academic period (2006 – 2007; 2007 – 2008) Group A (2006 – 2007): Non-ACGME based evaluation vs. Group B (2007 – 2008): ACGME-core competencies Performance for Groups A and B were compared using c2-test and the two-tailed unpaired t-test was used to assess each assessment component (significance was set at p ≤0.05)

Clerkship at LSUHSC – New Orleans Clerkship Structure Total time – 12 weeks A) 6 week General Surgery block • Trauma, General Surgery B) 2 Different 3-week Surgical Sub-specialties • Vascular, Pediatric surgery, Urology, Cardiothoracic, Neurosurgery, Orthopedics, ENT, Plastics, Ophthalmology Grading Components Assessments by faculty/senior residents (individual or group) • Evaluations: 60% • Shelf examination: 35% • Ethics: 5%

Decreased Proportion of Students Receiving the Highest Grade – Enhanced Discrimination Group B/ACGME (2007 – 2008) vs. Group A/non-ACGME (2006 – 2007) demonstrates a (more) bimodal distribution of grades

Clerkship at LSUHSC – New Orleans Grading Components Assessments by faculty/senior residents (individual or group) • Evaluations: 60% • Shelf examination: 35% • Ethics: 5%

Decreased Proportion of Students Receiving the Highest Grade – Enhanced Discrimination The evaluation component of the medical student’s grade, and not the ethics component or the shelf exam, account for this difference

Results Evaluation Component Adapting the ACGME-core competencies into the evaluations resulted in a significant decrease in the clinical performance assessment score Group B (ACGME) vs. Group A (non-ACGME) • 56.6 ± 0.16, n=164 vs. 58.3 ± 0.21; n=157 (p<.0001)

Results Adapting the ACGME-core competencies into the evaluations resulted in a significant decrease in the clinical performance assessment score Moreover, significantly less Honors (20.1% vs. 35%) and an increased HP (62.2% vs. 50.3%) and P (14% vs. 11.5%) were noted in Group B compared to Group A (p=.0296)

Limitations of the Study • No data re: Inter-rater reliability • Therefore, cannot address whether there is good/poor agreement among assessors (faculty/residents) in their evaluation of medical students • Increase n • Would be better to have 2-3 year data • Collecting this data prospectively… • In 2nd year, collected data electronically • .pdf file (email) • Not a prospective study • Analyzed two year comparative data

Question Does Clinical Performance Assessment Grounded in ACGME-Core Competencies Enhance Assessor Discrimination? Yes

An evaluation system based on ACGME-core competencies enhances assessor accuracy and discrimination Conclusions • An ACGME-core competencies’ based evaluation decreased the proportion of students receiving the highest performance level • More HPs and less H in Group B vs. A • It is plausible that by adapting a commonly understood construct (eg ACGME competencies) for medical student evaluations, consistent interpretation of items and observed performances will be facilitated

Future • By focusing on ACGME-based competencies, reliability may be increased • Decrease the practice of generalized view of a student’s performance • Faculty make an overall decision as to whether a particular student deserves an – H, HP, or a P – and then tend to score the performance variables to reflect this final grade • Poor understanding why faculty’s evaluation of students apparently represents a simple judgmental process Future research should address this issue. • Validate results over multi-Institutions, more years (increase n) • Because the decisions made on evaluations have serious professional implications, such as future residency match, developing an evaluation system that increases assessor reliability is important

Thank You Audubon Park – New Orleans, LA

HA Bazan MD 1 , M McDonough BA 1 , AB Marr MD 1 , CC Baker MD 1 , S Chauvin PhD 2