Authentic Performance Assessment

1. Authentic Performance Assessment Ellen J. Lehning, Ph.D. Department of Anesthesiology

2. Objectives • Define clinical competence and authentic performance • Design an authentic performance assessment • Implement an authentic performance assessment

3. What is the First Principle of Education? Teach to and Test Desired Endpoints: Educational Outcomes

4. Desired Outcome: Assess: Pass Written Boards Pass Oral Boards Achieve Clinical Competence

5. What is Clinical Competence? “Integration of knowledge, skills and attitudes.” Rice et al., Competency-based objectives for clinical training. Can. J. Med. Technol. 57:136, 1995

6. What is Clinical Competence? “The habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served.” Epstein and Hundert, Designing and assessing professional competence. JAMA 287:226, 2002

7. What is Clinical Competence? Miller’s Pyramid of Competence Does Shows how Knows how Knows Miller, The assessment of clinical skills/competencies/performance. Acad. Med. 65:563-567, 1990.

8. What is Clinical Competence? “Competence . . . means being able to function in context.” Chambers and Gerrow, Manual for Developing and Formatting Competency Statements. J. Dental Educ. 58:361, 1994.

9. How is Clinical Competence Assessed? “Competence is concerned with what people can do . . ., that is, potential.” “Performance is concerned with what people do . . ., that is, what is actually done in the real-life context.” Competence is inferred from performance! While, Competence versus performance: which is more important? J. Adv. Nurs. 20:525, 1994.

10. What is Authentic Performance Assessment? • Performance can be simulated or authentic • Simulated performance assessment uses an artificial context: SPs, OSCEs, Long Case - Standardized or Unstandardized • Authentic performance assessment uses a real context: Direct observation of a resident caring for a real patient with a real problem in a real setting – Unstandardized!

11. Why Use Authentic Performance? “Testing should be as close as possible to the situation in which one attacks the problem.” “Ill-structured problems are not found in simulated and/or standardized tests.” “The variation inherent in professional practice will always elude capture by a set of rules.” Wiggins, Assessing Student Performance: Exploring the Purpose and Limits of Testing, Jossey-Bass, Inc. 1993

12. How is it Done? “There are few validated strategies to assess actual clinical practice.” “Assessment at the apex of Miller’s pyramid, the does, is the international challenge of the century for all involved in clinical competence testing.” - Ronald Epstein - Val Wass

13. Design Step 1 – Select Competencies • Breakdown ACGME’s six global competencies into specific competencies • Technical skills, case management skills, clinical decision-making skills, etc. • Avoid too broad • Avoid too specific • Appropriate for the resident’s training level • Sequenced appropriately over the training period

14. Design Step 1 – Select Competencies • Write a competency statement: • Chambers and Gerrow, Manual for developing and formatting competency statements. J. Dental Educ. 58:361, 1994. • Verb • Direct Object • Qualifying conditions

15. Example Competency Statement Induce anesthesia for an ASA I-II patient by the end of the three month period

16. Example Competency Statements • The Global Communication Skills Competency Communicate accurately, efficiently and supportively with the patient, patient’s family and the health care team • Specific Communication Skill Competency Build a Relationship (Rapport Building) • Component Skills used to Build a Relationship Establish initial rapport Support Empathy Partnership Reflection Legitimation Respect Apology

17. The Performance Rating Process Specific performances are judged by expert rater(s) who synthesize multiple impressions in comparison to criteria given in a rating task and filtered through the experience of the rater. Specific performance(s) Judgment Expert Criteria (Criterion referencing) Chambers, A primer on competency-based evaluation. J. Dent. Educ. 61:651, 1997.

18. Design Step 2 – Select Sampling • Which performances or tasks will be observed? • How many observations? • When?

19. Design Step 3 – Select Raters Raters are qualified individuals who have an opportunity to observe and draw conclusions about residents – they judge the presence and the quality of the competency to be assessed How many raters Attendings Other professional staff Patients Self

20. Design Step 4 – Select Rating Criteria • Rating criteria can either be checklists or rubrics • Checklists • Presence or absence of a competency or of the components of a competency • Do not judge quality • May be appropriate for technical procedures or beginning level competencies • May not capture sophistication of complex performances: Hodges, B., Regehr, G., McNaughton, N., Tiberius, R., and Hanson, M. 1999. OSCE checklists do not capture increasing levels of expertise. Acad. Med. 74:1129-1134.

21. Example Checklist • Yes No • Competency: • Built a relationship?  • Specific Skills: • Established initial rapport?  • Employed Legitimation?  • Provided support?  • Demonstrated empathy? • Demonstrated legitimation?  • Demonstrated apology?  • Showed respect? 

22. Design Step 4 – Select Rating Criteria • Rubrics • Guidelines, rules, or principles by which performances are judged • Reflect best thinking as to what constitutes a good performance • Contain multiple quality levels • All rating levels must be written-down, defined and described • Best rubrics contain anchor(s) to illustrate the different points on the quality scale Arter & McTighe, Scoring Rubrics in the Classroom, Corwin Press, Inc. 2001

23. Example Rubric Excellent – Smooth and efficient. Able to use knowledge, judgment and skills to adjust management appropriately to the specific patient and operative procedure. Competent – Lacks smoothness and efficiency but is able to use knowledge, judgment and skills to adjust management appropriately to the specific patient and operative procedure. Beginner – Lacks smoothness and efficiency. Able to manage the case but exhibits limited use of personal judgment and responsiveness to the specifics of the patient and operative procedure. Requires some limited coaching or attending intervention. Novice – Can only manage the case with extensive coaching and attending intervention.

24. What are the Desirable Characteristics of Performance Assessment? Face validity Feasible Low Cost Non-reactive Generalizable Reliable Valid

25. Psychometric Issues • Write good rubrics • Train the raters • Only need 1 rater/performance • Sample over a broad array of cases/patients • Our intercase agreements range from 0.2 – 0.9 • Wass et al., Generalizability in range of the OSCE • ACGME preliminary data shows adequate generalizability, validity and reliability

26. Authentic Performances vs. Authentic Products Assessing an: Authentic Performance versus the Product of an Authentic Performance Authentic Products Patient Care Ethics PBLI Systems-Based Authentic Performances Patient Care Professionalism Communication Skills

27. Why Focus on Authentic Performance Assessment? • ACGME outcomes mandate • Current assessment is rudimentary • Accountability Murray et al., The accountability of clinical education: its definition and assessment. Med. Educ. 34:871, 2000 Bordage et al., Education in ambulatory settings: Developing valid measures of educational outcomes, and other research priorities. Acad. Med. 73:743, 1998