Understanding clerkship grading 2019-20

1. Understanding clerkship grading 2019-20 Valerie P. Jameson, MD. Assistant Dean of Clinical Curriculum Office of Medical Education

2. Why grade at all? • Grading is the summative (ultimate) feedback to student performance. • Am I performing at a high, low or acceptable level? • Do I have the skills necessary to be a competent physician? • Do I have the skills necessary to succeed in a highly competitive field? • Grading allows outside entities (residencies) to know how a particular student performance compares with his/her peers in that institution. • Using national norms helps assure that the grading is meaningful outside of the particular institution.

3. Grading in the pre-clinical years • Grading in the basic science courses is mostly based on objective data derived from Multiple Choice Question Testing. • The grade is roughly based on the average of the scores from several MCQT’s. • You can argue about a few questions, but the score probably reflects what a student would do on a similar test on any given day. • Caveat - Measuring true performance is complicated by the additional uneven skill of MCQT Test Taking ability.

4. Grading in the clinical years • More complex • Now students must also be graded on • Clinical skills – patient interaction, communication, eye-hand coordination, reasoning skills, team integration, self-perception, etc. • Professionalism, ethical behavior • While the opinion of an experienced faculty member in judging student performance is considered “objective”, the reality is that subjectivity and perception have an increased influence on feedback. • While this may seem to allow for unfair variability, this is exactly how patients and peers judge physicians in the real world. (Learning how people perceive you is incredibly valuable knowledge and that’s what you get in your clinical grade).

5. Grading in the clinical years • Equally important in describing student performance is knowledge and reasoning ability, otherwise known as expertise. • Patients come to physicians for caring and compassion, but they expect ethical treatment and expertise as well. • Expertise is more easily quantified through MCQT. • This also allows for national comparison.

6. It’s vital that all three aspects contribute to the summative clinical grade • Students often excel in one aspect or the other. • However, the good physician must be competent in all three areas. • Grading therefore must fairly evaluate all three domains. • This allows students to accurately identify their strengths and weaknesses. • It informs residencies which students would best fit their discipline. • It satisfies the public that medical schools are producing competent physicians.

7. How to achieve this goal • Clinical evaluations must provide detailed evaluation of students’ performance and behavior in the clinical setting. • Rather than have faculty assign a grade, faculty are simply asked to choose descriptors of the student’s performance. These are assigned numeric values which added together provide the grade. • This way, students achieving the same descriptors across the hospital, city or state will receive the same grade. • Sites with significantly smaller student numbers may have faculty submit feedback to the clerkship director who then compiles the final evaluation using the same rubric.

8. Measuring expertise • All UT clerkships use the National Board of Medical Examiners Subject exams (known as shelf exams). • Student performance is reported with a raw score(mean equated percent correct score) which resembles a percentage correct but is actually a complexly derived number based on the performance of students answering these questions on previous exams. • Each raw score is reported with the national percentile performance of students in a previous year. • Each discipline has a different set of raw scores and percentiles for their specific exams dictated by the performance of students in their respective disciplines.

9. Choosing passing and honors scores • The NBME gathers information from clerkship directors across the nation to determine acceptable minimum scores for passing and honors designation on shelf exams. • These numbers are reported in ranges. • In an attempt to offset the asymmetric power of the MCQT when grading, UT clerkship directors have typically chosen to use a lower level of these ranges for determining passing and honors.

10. NBME- nationally recommended minimum passing score for the Family Medicine Shelf Exam

11. Minimum passing score for the shelf exam and the clerkship. • Each clerkship chooses their own minimum passing and honors scores as well as weights for each component of their grades which they can explain in detail during their clerkship orientation. • The minimum scores are consistent across the state for each clerkship and for the entire academic year. These percentages and scores are reassessed each year before block 5/6 and kept the same throughout the academic year.  • For example, the Family Medicine clerkship has chosen a raw score of 61, which is the 5th percentile for the national student performance across the 2018-19 academic year. • Although the NBME recalculates these parameters every few years, the clerkships should maintain the same percentile for passing. • So, if a 59 raw score was passing last year and a 61 is passing this year, the percentile is the same in order to remain consistent year to year.

12. Minimum honors score for the shelf exam

13. Minimum honors score for the shelf exam • The Family Medicine clerkship has chosen a raw score of 79, which is the 63rd percentile for the national student performance across the 2018-19 academic year. • 79 is the lowest raw score acceptable to achieve honors in the national recommended range of scores. • In this way, Family Medicine is able to assure that a student receiving an A for the clerkship has achieved a better than average level of expertise in MCQT using nationally derived and accepted standards.

14. Clerkship minimums (academic year 2018-19) • ClerkshipMin PassMin Honors(A) for the Course • Medicine              59(5%ile) 79 (66%ile) • Pediatrics 62(5%ile) 80 (63%ile)    • Neurology 62(2%ile) 82 (62%ile) • Family Medicine   61(5%ile) 79 (63%ile) • Psychiatry             69(6%ile) 84 (63%ile) • Surgery                  60(7%ile) 81 (83%ile) • OB-Gyn                  64(5%ile) 80 (58%ile)

15. Weighting clinical and exam scores • The competent student excels in all three areas – patient care, ethics and expertise. • However, if all student clinical evaluations are glowing, then the shelf exam, by default, becomes the discriminating factor. • This is unacceptable. • We must strive to make the clinical evaluation discriminating and to weight it sufficiently to counterbalance the ruthless objectivity of the shelf score.

16. Correct weighting • Potential outcomes • Super nice, hard working student but average performance on the shelf – Clerkship Grade B • Never available and only putting in the minimal work on the ward in order to be off studying for the shelf which they blow out of the water – Clerkship Grade B • Excellent performance on the wards, intimately integrated into the team, barely makes above the cutoff for honors on the shelf – Clerkship Grade A • The student must excel in both areas to be able to qualify for Honors in the Clerkship. • So just because a student achieved the minimum honors score on the shelf, it does not guarantee that he/she will get an A for the clerkship.

17. Why is this approach necessary? Why grade at all? • Because of a decided lack of grade discrimination across the country, residencies are unable to use GPA or Dean’s letters (MSPE’s) to determine which students would be suitable for their field. • As a consequence, competitive residencies have resorted to using the USMLE Step 1 score as a means of winnowing down the massive numbers of student applying for their positions. • The Step 1 was never designed to be used in this manner and is not statistically valid in identifying the student most likely to be an exceptional physician. • It is unconscionable to allow one day’s performance on a MCQT to determine the career options for the awesome students we have in our college of medicine! • Most importantly, the only way for a student to improve his/her skills is by receiving valid, useful, timely, specific, discriminating feedback.

18. Beta testing • The Internal Medicine and Pediatrics Clerkships are beta testing a variation of this process for the academic year 2018-19. • They have chosen a slightly higher raw score within the minimum honors score range to determine honors for the shelf exams. • However, they are using last year’s minimum honors score to identify the students who are eligible to achieve an A for the rotation. • We will study this method to determine if it is helpful in emphasizing the effect of the clinical grade when computing the composite grade, basically making it possible for a student to get an A for the rotation even if he/she does not get an honors grade on the shelf.

19. What does a grade mean? • Until recently, the Dean’s Letter, MSPE, simply provided a GPA along with the student transcripts. • Lots of A’s looked pretty good. • Now, the MSPE must include the grade distribution for each clerkship. • Residency Program directors will know that getting an A when 80% of the class did as well is meaningless. • Getting an A when only a third of the class did as well is discriminating. • Our job is to assure that that assignment is fair and valid.

20. Lastly, apples and oranges • The NBME reports the raw scores (mean equated percent correct score). • To use them in the calculation with clinical scores, oral and slide exams, it is essential that the raw score is converted to the same format we use for grading. • Clerkship Grading Scale • A 89.5-100B 78.5-89.49C 67.5-78.49F < 67.49

21. So, in Family medicine • If 61 is used as the lowest passing score, then it is converted to 67.5. • If 79 is used as the minimum honors shelf score, then it is converted to 89.5. • All the other grades are adjusted accordingly which results in a number higher than the raw score and is used in the calculation of the grade. (using all apples instead of a mixture of fruit.) • This makes that calculation numerically valid, even at the expense of a bit of confusion when discussing how the grades are derived. • Remember that specific numbers on the shelf are used to determine the minimum passing score and minimum honors score. The adjusted shelf score is used to calculate the final composite grade, but there is no set NBME score for determining between a B or C final grade.

22. UTHSC is not alone • Med schools across the nation are struggling with the same issues. • OME takes student grading very seriously and is always on the lookout for ways to improve. • Significant changes can only be made at the start of a new academic year, although smaller improvements or correction of errors may occur at any time to assure fairness for the students. Mid-year adjustments will only be made in the students’ favor.

23. This powerpoint is in response to feedback • Thanks to all students who take the time to give us constructive feedback. • Just like students, we welcome constructive advice that informs us of issues and provides suggestions for improvement. • UT COM students are truly our greatest resource. • (I admit, I could be biased.)