Joan Anzia , M.D. Tony Rostain , M.D.

1. Clinical Skills Verification rater TrainingMODULE 2Training Faculty Evaluators of Clinical Skills: Drivers of Change in Assessment Joan Anzia, M.D. Tony Rostain, M.D.

2. Outline • Mini pretest! • Brief history of assessment in medical education • Drivers of change in assessment in medical education • Miller’s pyramid • Why is faculty training necessary? • Methods to train faculty to evaluate clinical skills. • Post-test

3. Module 2Pre-Test • A clinical skills exam of a trainee assesses whether he or she “knows how” according to Miller’s Pyramid. a. True b. False

4. Module 2Pre-Test 2. Faculty evaluators in a group are preparing their individual evaluation scores for a videotaped trainee clinical skills exam, and comparing their scores with the scores of “expert” raters. This activity is called: a. Behavioral Observation Training b. Frame of Reference Training c. Direct Observation of Competence Training d. Performance Dimension Training

5. Brief history of assessment in medical education • Through the 1950s: knowledge evaluated through essays and open-ended questions graded by faculty. Clinical skill and judgment tested with live oral examinations, sometimes after bedside data-gathering by the examinee. • 1960s: multiple-choice exams to test knowledge base

6. Clinical Skills Exams vs Multiple Choice Question Exams

7. New technologies come on the scene • Introduction of computers in the 1980s enabled large-scale testing using MCQs that are machine-scanned and scored. • Computers also allow the assessment of clinical decision-making through use of interactive item formats. • Advances in psychometrics allow shorter tests, reduction of bias, and identification of error sources.

8. Since the 1980s • OSCEs (Objective Structured Clinical Exams) have been fine-tuned with improved psychometric qualities. • Assessment of clinical skills and performance has lagged behind – faculty are inexperienced, don’t share common standards, and have not been trained to apply them consistently.

9. Drivers of change in medical education • Outcomes-based education: a focus on the “end product” rather than the process. What should a psychiatrist “look like” at the end of training? • National initiatives in accountability, patient safety and quality assurance: maintaining the public trust in the medical profession and improving the quality of healthcare.

10. Levels of assessment: Miller’s Pyramid

11. Miller’s Pyramid • Knows: what a trainee “knows” in an area of competence. MCQ-based exam. • Knows how: does the trainee know how to use the knowledge (acquire data, analyze and interpret findings). An interactive reasoning exam. • Shows how: can the trainee deliver a competent performance of the skill with a patient. Clinical skills exams. • Does: does the clinician routinely perform at a competent level outside of a controlled testing environment? Performance-in-practice assessment, critical incident systems.

12. Why is faculty training necessary? • Assessment methods based on observation are only as good as the individuals using them. Holmboe and Hawkins, 2008 • Faculty sometimes don’t possess sufficient knowledge, skills and attitudes in particular competencies. • Competencies evolve over time, and faculty may not have been trained in specific competencies.

13. How do we train evaluators?Empirically studied training methods: • Behavioral Observation Training (BOT) • Performance Dimension Training (PDT) • Frame of Reference Training (FoRT) • Direct Observation of Competence Training

14. Behavioral Observation Training • Get faculty to increase the number of their observations of their trainees. • Provide a form of observational aide that raters can use to record observations ( a “behavioral diary”). • Help faculty members learn how to prepare for an observation. (Determining goals, evaluator position, etc.)

15. Performance Dimension Training • Designed to teach the faculty with the appropriate performance dimensions used in the evaluation system. • It is a critical element for all rater training programs: goal is to define all the criteria for each dimension of performance. • Faculty interact to further define criteria (what constitutes “superior performance” etc.) and work towards consensus on framework and specific criteria.

16. Frame of Reference Training • First, Performance Dimension Training must be completed. • FoRT targets accuracy in rating: goal is to achieve consistency. • First, minimal criteria for satisfactory performance defined, then marginal criteria. • Faculty are given clinical vignettes describing performance in different ranges.

17. Frame of Reference Training (cont.) • Faculty use vignettes to provide ratings. • Trainer provides feedback on what the “true” ratings should be, with an explanation for each rating. • Discussion of discrepancy between faculty ratings and “true” ratings from trainer. • Repeated practice: “calibration.”

18. Module 2Post-Test • A clinical skills exam of a trainee assesses whether he or she “knows how” according to Miller’s Pyramid. a. True b. False

19. Module 2Post-Test • A clinical skills exam of a trainee assesses whether he or she “knows how” according to Miller’s Pyramid. b. False A CSV exam assesses whether a resident can “show how.”

20. Module 2Post-Test 2. Faculty evaluators in a group are preparing their individual evaluation scores for a videotaped trainee clinical skills exam, and comparing their scores with the scores of “expert” raters. This activity is called: a. Behavioral Observation Training b. Frame of Reference Training c. Direct Observation of Competence Training d. Performance Dimension Training

21. Module 2Post-Test 2. Faculty evaluators in a group are preparing their individual evaluation scores for a videotaped trainee clinical skills exam, and comparing their scores with the scores of “expert” raters. This activity is called: b. Frame of Reference Training