ACGME General Competencies: Evaluation Methods

1. ACGME General Competencies:Evaluation Methods Richard E. Hawkins, MD Deputy Vice President, Assessment Programs National Board of Medical Examiners

2. Overview of Presentation • Introduction to ACGME Competencies • Three important assessment methods • Assessment of the individual competencies • Comprehensive assessment approaches

3. ACGME General Competencies • Medical knowledge • Patient care • Interpersonal and communication skills • Professionalism • Practice-based learning and improvement • Systems-based practice

4. Views on the General Competencies • Not well constructed / not portable or coherent • Not evidence-based • Theoretical basis questionable • ? Influence of content and context • Introduction - important stimulus for action • Two “competencies” essential for practice • Emphasis on outcomes (vs structure & process) • More comprehensive view regarding evaluation

5. ACGME General Competencies = KSA + 2 • Medical knowledge • Patient care • Interpersonal and communication skills • Professionalism • Practice-based learning and improvement • Systems-based practice

6. Selecting Assessment Methods: Factors • Validity • Reliability • Feasibility • Credibility / acceptability • Quality of feedback provided • Behavioral impact

7. Assessing the Competencies • Thoughts on implementation: • Start with an inventory • Refine and/or document existing approaches • Apply new assessment methods to holes • Develop methods collaboratively • Apply locally based upon resources and needs • Critical role of professional organizations – • PD Associations • Understand and take advantage of overlap between competencies and related assessment methods.

8. Three Assessment Tools • Portfolios • Multi-Source Feedback / 360o Evaluations • Chart-stimulated Recall (CSR)

9. Portfolios • “…a collection of products prepared by the resident that provides evidence of achievement related to a learning plan.” (ACGME) • “A collection of material that records and reflects on learning events and processes. A sample of work products that demonstrates accomplishments.” (ABMS)

10. Portfolios • Stimulus for reflection and self-directed learning + repository of evidence • Contents (evidence) depend upon learning and assessment objectives - may include just about anything • Useful for assessing competencies difficult to measure: PBLI, SBP • Reliability, validity, and feasibility issues

11. Portfolios: Critical Components • Learning objectives • “Evidence” • Written work • Video-tapes (mini-CEX, SPs) / Audio recordings (patient communication) • Structured faculty observation / rating • “Reflection on action” – essential • Reflection on past experiences to guide further learning • Summarizes the learning that has occurred with discussion of evidence supporting the attainment of the learning objectives

12. Portfolios: Hybrid Example • Learning objective • Analyze the quality of a recent article using evidenced-based medicine principles • Activities • Learner chooses review article on topic of personal interest • Prepares presentation using EBM criteria (JAMA)

13. Portfolios: Hybrid Example • Activities • “Reflects” on what was learned about topic and EBM criteria for reviews after presentation • May be self-assessment only or with mentor • Contents in portfolio • Copy of presentation • Brief written summary of self reflection

14. Portfolios: Challenges • Well-suited for formative assessment • Summative: need clear definitions of learner goals, content, and explicit evaluation criteria • Must have clear learning objectives • Evidence must be of sufficient quality • Should demonstrate actual learning • Initial development of portfolios challenging

15. Portfolios: Conclusions • Learner centered and learner driven • Emphasizes reflection, self-assessment • Need clear learning objectives • Good evidence for use as formative tool • ? Future as summative “evaluation tool” • Requires fairly significant “up-front” investment • Embraces continuum of education and practice  ? Future role in CPD, MOC, MOL, credentialing and privileging…

16. 360° Evaluations • Definition • Evaluation completed by multiple individuals, usually from different perspectives and based upon observations in different contexts • Raters: self, faculty, peers, nurses, students, patients, other health care providers (medical assistants, social workers, technicians, discharge planners, etc.)

17. 360° Evaluations PATIENTS RESIDENT SELF PEERS ATTENDINGS NURSING

18. 360° Evaluations: Strengths • Captures different perspectives: • Patients and nurses - evaluate humanism, professionalism, communication • Peers – work ethic, team approach, professionalism • Others – unique observations on key attributes • Supplemental approach for other competencies: • Communication / IPS, Patient Care, SBP

19. 360° Evaluations: Limitations • Usual limitations of global rating forms: • Reliability: Patients - need 20 – 80 ratings • Validity: • Nurse ratings – depends upon rating site and rater background • Individual categories highly related • Logistics of data collection, entry, and analysis • Learner resistance and denial issues

20. 360° Evaluations: Conclusions • Uses - professionalism, humanism, team approach and patient focused skills / attitudes • Raters should provide ratings based upon the context of observation and qualifications • Implementation (Norcini, Med Educ 2003;37:539) • Clear purpose, communication of criteria, rater training, monitoring and feedback • Communication of objectives through assessment  importance of team approach and patient-centeredness

21. Chart Stimulated Recall (CSR) • Extension of medical record review • Uses actual patient records as the stimulus for discussion and template for assessment • Expand and elaborate on medical record entries: • Capture information not revealed in routine audits • Explore contextual factors underlying clinical decisions • Further assess FOK, problem solving, clinical reasoning, diagnostic and therapeutic management

22. Chart Stimulated Recall (CSR) • Value points to deficiencies in medical record audit in making judgments regarding physician competence • Filling in missing data or context (that result in underestimation of health care quality) • Rationale for diagnostic and therapeutic decisions

23. ABEM CSR Research Project • Option for recertification • 3 cases / case specific criteria: • Reliability 0.54 • Correlation with real cases 0.70 • Consistent with SP encounters • Eliminated in 1993 • Labor intensive and expensive for ABEM • Time consuming for candidates

24. CPSO Peer Review Program: CSR • 8 records + typed summary + interview • Generic • Knowledge, data acquisition, problem solving, patient management, comprehension of pathophysiology, resource utilization • Inter-rater reliability .75 - .90 • Correlations: MCQ .56; SP .70; oral exam .53; OSCE .31

25. Chart Audit Presenting complaint Differential diagnosis Secondary diagnoses CSR Hx / PE details Tests considered Other treatments Follow-up details Contextual info: Practice factors Patient factors System factors Payment method Trainee present Calgary: Chart Audit + CSR Jennett, JCEHP 1995;15:31

26. Chart Stimulated Recall (CSR) • Advantages: • Relevant - based on examinee’s patients • Addresses higher cognitive skills • Reinforces importance of documentation • Targets difficult to assess competencies: • Patient care, PBLI and SBP • Supplemental information on other competencies: • Medical knowledge – application • Communication and IPS – written communication

27. The Competencies ACGME competencies defined as they apply to Internal Medicine

28. Medical Knowledge • Defined as demonstrating a command of established and evolving biomedical, clinical and social sciences and the application of that knowledge to patient care and the education of others. • Included in this context are:

29. Medical Knowledge • An open minded and analytical approach to acquiring new knowledge • The ability to access and critically evaluate current medical information and scientific evidence

30. Medical Knowledge • Acquisition of applicable knowledge of the basic and clinical sciences that underlie the practice of internal medicine • The application of this knowledge to clinical problem solving, clinical decision-making and critical thinking.

31. Medical Knowledge • Fund of knowledge: • In-training examination / Other written examinations • Application of knowledge / higher cognitive skills • Medical record review – CSR • Mini-CEX • Conference participation • Analytical approach / critical evaluation skills • Overlap with Patient Care and PBLI competencies

32. Interpersonal and Communication Skills • These skills enable physicians to establish and maintain professional relationships with patients, families, and other members of health care teams. • Included are the abilities to:

33. Interpersonal and Communication Skills • Provide effective and professional consultation to other professionals and sustain therapeutic and ethically sound professional relationships with patients, their families, and colleagues; • Use effective listening, nonverbal questioning, and narrative skills to communicate with patients and families;

34. Interpersonal and Communication Skills • Interact with consultants in a respectful, appropriate manner; and • Maintain comprehensive, timely, and legible medical records.

35. Interpersonal and Communication Skills • Direct Observation: • Mini-CEX • Standardized patients • Structured clinical observations • 360o Evaluations: • Incorporates the direct observations of: • Peers, patients, nurses, students… • Written Communication • Medical record reviews • Consultant evaluations • Portfolio task

36. Professionalism • Residents are expected to demonstrate behaviors that reflect a commitment to continuous professional development, ethical practice, and understanding and sensitivity to diversity, as well as a responsible attitude towards patients, the profession and society. • Include are the abilities to:

37. Professionalism • Demonstrate respect, compassion, integrity, and altruism in relationships with patients, families and colleagues; • Demonstrate sensitivity and responsiveness to the gender, age, culture, religion, sexual preference, socioeconomic status, beliefs, behaviors, and disabilities of patients and professional colleagues;

38. Professionalism • Adhere to principles of confidentiality, scientific / academic integrity, and informed consent; and • Recognize and identify deficiencies in peer performance.

39. Professionalism • Performance evaluations (rating scales) • Longitudinal / episodic observations • 360o Evaluations • Patients, peers, nurses, students… • Direct observation • Real or SPs (gender or cultural issues)

40. Professionalism • Recognize and identify deficiencies in peer performance. • Participation in peer assessment and quality of care review (PBLI)

41. Patient Care • Defined as compassionate, appropriate, and effective care which encompasses the promotion of health, prevention of illness, treatment of disease, and end of life. • At the cornerstone of competent patient care are the abilities to:

42. Patient Care • Gather accurate, essential information from all sources, including medical interviews, physical examinations, medical records, and diagnostic / therapeutic procedures; • Make informed recommendations about preventive, diagnostic, and therapeutic options and interventions that are based on clinical judgement, scientific evidence, and patient preference;

43. Patient Care • Develop, negotiate, and implement effective patient management plans and integration of patient care; and • Competently perform the diagnostic and therapeutic procedures inherent to the practice of internal medicine.

44. Patient Care • Data-gathering (Hx, PE, communication) / patient education and counseling / informed decision making: • Direct observation (Mini-CEX, SP) • Patient satisfaction (part of 360o) – preferences • Overlap with Communication, Medical Knowledge • Patient management / use of scientific evidence: • Medical record review +/- CSR • Journal article review / Literature search • Portfolio task • Conference participation • Overlap with PBLI, SBP, Medical Knowledge

45. Patient Care (continued) • Procedural / technical skills: • Direct observation • Performance evaluations • Computer / mannequin simulators • Medical record review (+/- CSR) • Documentation • Indications, Interpretation of results

46. Practice-based Learning and Improvement • The ability to use scientific evidence and methods to investigate, evaluate, and improve patient care practices. • This effort encompasses the abilities to:

47. Practice-based Learning and Improvement • Identify areas for improvement and implement strategies to enhance knowledge, skills, attitudes, values, and processes of care; • Analyze and evaluate practice experiences and implement strategies to continually improve the quality of patient care;

48. Practice-based Learning and Improvement • Develop and maintain a willingness to learn from errors and use errors to improve systems or processes of care; and, • Use information technology and/or other available methodologies to access and manage information, support patient care decisions, and enhance both patient and physician education.

49. PBLI: Important Message • Need for life-long learning • Current practice can be improved • Traditional CME not effective • Value of experiential learning / relevance • Emphasizes the continuum of training and practice • Endorses self-assessment and life-long learning • Facilitates and structures self-assessment • Consistent with models of practitioner learning and behavioral change

50. Desired Practice Actual Practice Validation Assessment Contemplation Definition of Educational Need Educational Action Handfield-Jones, 2002 Reward