1 / 27

Authentic Performance Assessment

Objectives. Define clinical competence and authentic performanceDesign an authentic performance assessmentImplement an authentic performance assessment. Teach to and Test Desired Endpoints:. What is the First Principle of Education?. . Educational Outcomes. Pass Oral Boards. Pass Written Boards.

MikeCarlo
Download Presentation

Authentic Performance Assessment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

    1. Authentic Performance Assessment Ellen J. Lehning, Ph.D. Department of Anesthesiology

    3. Teach to and Test Desired Endpoints: What is the First Principle of Education?

    5. What is Clinical Competence? “Integration of knowledge, skills and attitudes.”

    6. What is Clinical Competence? “The habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served.”

    7. What is Clinical Competence?

    8. What is Clinical Competence? “Competence . . . means being able to function in context.”

    9. How is Clinical Competence Assessed? “Competence is concerned with what people can do . . ., that is, potential.”

    10. What is Authentic Performance Assessment? Performance can be simulated or authentic Simulated performance assessment uses an artificial context: SPs, OSCEs, Long Case - Standardized or Unstandardized Authentic performance assessment uses a real context: Direct observation of a resident caring for a real patient with a real problem in a real setting – Unstandardized!

    11. Why Use Authentic Performance? “Testing should be as close as possible to the situation in which one attacks the problem.” “Ill-structured problems are not found in simulated and/or standardized tests.” “The variation inherent in professional practice will always elude capture by a set of rules.”

    12. How is it Done? “There are few validated strategies to assess actual clinical practice.” “Assessment at the apex of Miller’s pyramid, the does, is the international challenge of the century for all involved in clinical competence testing.”

    13. Design Step 1 – Select Competencies Breakdown ACGME’s six global competencies into specific competencies Technical skills, case management skills, clinical decision-making skills, etc. Avoid too broad Avoid too specific Appropriate for the resident’s training level Sequenced appropriately over the training period

    14. Design Step 1 – Select Competencies Write a competency statement: Chambers and Gerrow, Manual for developing and formatting competency statements. J. Dental Educ. 58:361, 1994. Verb Direct Object Qualifying conditions

    15. Example Competency Statement Induce anesthesia for an ASA I-II patient by the end of the three month period

    16. Example Competency Statements The Global Communication Skills Competency Communicate accurately, efficiently and supportively with the patient, patient’s family and the health care team Specific Communication Skill Competency Build a Relationship (Rapport Building) Component Skills used to Build a Relationship Establish initial rapport Support Empathy Partnership Reflection Legitimation Respect Apology

    17. The Performance Rating Process Specific performances are judged by expert rater(s) who synthesize multiple impressions in comparison to criteria given in a rating task and filtered through the experience of the rater.

    18. Design Step 2 – Select Sampling Which performances or tasks will be observed? How many observations? When?

    19. Design Step 3 – Select Raters Raters are qualified individuals who have an opportunity to observe and draw conclusions about residents – they judge the presence and the quality of the competency to be assessed

    20. Design Step 4 – Select Rating Criteria Rating criteria can either be checklists or rubrics Checklists Presence or absence of a competency or of the components of a competency Do not judge quality May be appropriate for technical procedures or beginning level competencies May not capture sophistication of complex performances: Hodges, B., Regehr, G., McNaughton, N., Tiberius, R., and Hanson, M. 1999. OSCE checklists do not capture increasing levels of expertise. Acad. Med. 74:1129-1134.

    21. Example Checklist

    22. Design Step 4 – Select Rating Criteria Rubrics Guidelines, rules, or principles by which performances are judged Reflect best thinking as to what constitutes a good performance Contain multiple quality levels All rating levels must be written-down, defined and described Best rubrics contain anchor(s) to illustrate the different points on the quality scale

    23. Example Rubric

    24. What are the Desirable Characteristics of Performance Assessment? Face validity

    25. Psychometric Issues Write good rubrics Train the raters Only need 1 rater/performance Sample over a broad array of cases/patients Our intercase agreements range from 0.2 – 0.9 Wass et al., Generalizability in range of the OSCE ACGME preliminary data shows adequate generalizability, validity and reliability

    26. Authentic Performances vs. Authentic Products

    27. Why Focus on Authentic Performance Assessment? ACGME outcomes mandate Current assessment is rudimentary Accountability

More Related