1 / 29

Comparison: Traditional vs. Outcome Project Evaluative Processes

Comparison: Traditional vs. Outcome Project Evaluative Processes. Craig McClure, MD Educational Outcomes Service Group University of Arizona December 2004. Current Problem. Increasing public concerns with quality and safety. Variable patterns of care that are not based on medical science.

kaida
Download Presentation

Comparison: Traditional vs. Outcome Project Evaluative Processes

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Comparison: Traditional vs. Outcome Project Evaluative Processes Craig McClure, MD Educational Outcomes Service Group University of Arizona December 2004

  2. Current Problem • Increasing public concerns with quality and safety. • Variable patterns of care that are not based on medical science. • Poor quality of interpersonal “service.” • Public encounters difficulty in assessing physician competence (initial and continuing ) and judging quality.

  3. To improve the quality of health care in the United States by ensuring and improving the quality of graduate medical educational experiences for physicians in training. The ACGME Mission

  4. Problem Plus Mission • ACGME responded to the challenge by changing focus to: • How well do we learn what is being taught • How well do we practice what we learn?

  5. Structure & process Competency A new way of thinking How to change the educational and accreditation system from…

  6. OLD: goal was for the Program to comply with the written RRC Requirements NEW: the Program Director must determine if residents achieve the learning objectives set by the Program. Program Goal

  7. Six Domains • Medical Knowledge • Patient Care • Professionalism • Communication and Interpersonal • Practice Based Learning and Improvement • Systems Based Practice

  8. Purpose of Assessment • Assess residents' attainment of competency-based objectives • Facilitate continuous improvement of the educational experience • Facilitate continuous improvement of resident performance • Facilitate continuous improvement of residency program performance

  9. Whatever we measure we tend to improve. David C. Leach, M.D. Executive Director ACGME September 12, 2002

  10. Characteristics of good assessment • Measures actual performance • Identifies areas for improvement • Satisfies reasonable request for accountability • Is practical • Is done over time to discern growth

  11. Types of Evaluation • Formative • Improve performance • Summative • Note achievement Both types of evaluation can be used to evaluate either an individual or a program.

  12. Characteristics of good assessment • Systematic • Dependable • Comprehensive • Congruent • Practical

  13. Characteristics of good assessment (continued) • Makes professional practice more transparent • Deconstructs the role of physician • Clarifies levels of expertise by distinguishing functional levels

  14. Characteristics of good assessment (continued) • Measures actual performance • Identifies areas for improvement, i.e., self, others • Satisfies reasonable requests for accountability

  15. Traditional Evaluation • Global • End of rotation • Subjective • Anchored to norms seen by attending (therefore variable) • “I like/didn’t like the resident” • Focused on rotation goals (not movement toward competency)

  16. Outcome Based Evaluation • Formative, focused on specific competencies required for a physician • Measure the full scope of professional characteristics from very specific procedures to skills involving a synthesis of component abilities • Specific evaluative techniques chosen to match the skill being assessed

  17. Assessment Tools (The Toolbox) • 360° Evaluation Instrument • Chart Stimulated Recall Oral Exam (CSR) • Checklist Evaluation of Live or Recorded Performance • Objective Structured Clinical Exam (OSCE) • Procedure, Operative or Case Logs

  18. The Toolbox (continued) • Patient Surveys • Portfolios • Record Review • Simulations and Models • Standardized Oral Exams • Standardized Patients (SP) • Written Exams (MCQ)

  19. OLD: global checklist format NEW: Type of evaluation chosen specifically to measure the chosen skill drawn from the 6 domains Evaluation Method

  20. OLD: once per rotation NEW: multiple intervals assessing component behaviors as well as the integrated practice of medicine. Frequency of Evaluation

  21. OLD: End of rotation NEW: Timing chosen to facilitate evaluation of a specific competency Timing of Assessment

  22. OLD: Most frequently the preceptor evaluated the resident against the norm of previous residents in that experience NEW: Criteria defining competence are utilized as the standard against which resident performance is measured Anchors for Evaluation

  23. OLD: at best tended to address the resident’s success at the goals for the rotation NEW:Criteria for evaluation describe the qualities of the competent physician, so are more wide ranging or more specific Target of Evaluation

  24. OLD: typically one per rotation NEW: multiple, both physician and non-physician evaluators Number of Evaluators

  25. Other Outcome Characteristics • Authentic • More Individualized • Reflection and Self-knowledge Critical

  26. “Authentic” • Justification for elements included in the curriculum is that competence as a practicing physician requires that skill, knowledge or attitude • Evaluation is of the actual skill, knowledge or attitude used by practicing physicians

  27. More Individualized • A principle of a criteria-driven physician curriculum is that everyone can become competent with sufficient exposure • Residents obtain skills at different rates with requirements for disparate learning experiences • An optimal outcome-driven system would have an intake assessment followed by an individualized program of study

  28. Reflection and Self-knowledge Critical • Criteria for competence are provided to the learner • Impetus for improvement arises from desire to narrow the gap between criteria and performance • Accurate self-assessment is essential to the resident gauging personal performance

  29. In Summary Traditional method: Not systematic Subjective & Normative based Global evaluations @ rotation end Outcomes-based: Systemic and comprehensive Based on criteria defining competence Multiple measures and intervals

More Related