Core Competency Assessment in Emergency Medicine from Design to Implementation

1. Core Competency Assessment in Emergency Medicinefrom Design to Implementation Christian Arbelaez, MD, MPH Assistant Residency Director Harvard Affiliated Emergency Medicine Residency Program Brigham and Women’s Hospital CORD Educational Soundbite San Diego, CA 2011

2. Goals • To describe a comprehensive core competencies assessment program designed, implemented, and reviewed after one year of use in our residency program • To share lessons learned from our experience

3. Outline • Design (Oct 2008) • Implementation (Jul 2009 - June 2010) • 1 year Review (Summer 2010) • Lessons Learned

4. Design: Goals • The goals of the program for the residents • Robust feedback loop • Comprehensive yet practical • Multiple and repeated assessments • Assess a resident’s developmental progression • Identify promotion or remediation criteria • The goal of the program for the residency • Identify need for programmatic change

5. Design: Process • Review the ACGME guidelines • Needs assessment and current areas of strengths and weakness • Stakeholder meetings and focus groups • Resource prioritization • Develop action plan with timelines • Competency based curriculum and assessment program • Faculty development program • Concomitant research agenda and groups

6. Design: ACGME Domains • Annual Competency Assessment • Competencies by PG year - expected, measured, remediation • Chief Complaint Competency • 3 chief complaints • Procedural Competency • Pre-determined # of logged procedures • 3 key index procedures • ED bedside ultrasound • Resuscitation Competency • Adult and Pediatric medical and trauma • Off-service rotations

7. Design: 6 Core Competencies • PGY 1 and PGY 2 • Interpersonal and Communication Skills • Professionalism • Medical Knowledge • Patient Care • PGY 3 • Practice-Based Learning and Improvement • PGY 4 • Systems-Based Practice

8. Design: HAEMR • Annual Competency Assessment • Resident portfolio: all below + QI project, Research, PPT • 360-degree eval - Faculty by PGY, Peer, Nursing, Self w/ P-Mex • Chief Complaint Competency • PGY 1) CP 2) Abdominal pain 3) Weakness 4) Oral boards • Mentorship program • Procedural Competency (A and B components) • Logs • Procedure cards: Endotracheal intubation and CVL • U/S program • Resuscitation Competency (A and B components) • Logs • STRATUS Simulation center/Mock codes • Off-service rotations

9. Implementation • Implementation and roll-out plan • Phase 1 and phase 2 • Reasonable set deadlines • Phase 1 start date with new interns • Key component • Clear and transparent communication plan for the residents, faculty, and all staff involved in assessment of the residents

10. Implementation: Communication • Communication Plan • Resident group meetings • Program objectives and expectations • Changes in the electronic data management system • Distribution of semi-annual assessment forms • Examples of portfolio entries and self-reflection pieces • Resident individual meetings • Program manager review portfolios in prep for semi-annual • Faculty meetings • Faculty Primer: NI evals, 3 Procedures, Mentorship program • Procedure cards - direct observation and constructive feedback

11. One-Year Review • Critical review of program • Program evaluation: survey, meeting, retreat • PGY data report • Identified areas for improvement • Action plan for Phase 2

12. One-Year Review: Phase 2 • Annual Competency Assessment • Resident portfolio • Resident learning contract and Professionalism checklist • Promotion criteria • 360-degree eval - Peer and Nursing, Self • Chief Complaint Competency • Mentorship program • Procedural Competency (A and B components) • Logs: Variability but on avg reaching targets by PGY3 • Intubation and CVL part B • Resuscitation Competency (A and B components) • Logs: Variability but on avg reaching targets by PGY3 • Off-service rotations • Agreements w Anesthesia and ICUs

13. Self-Assessment A.P-MEX form in NI B.Core Competency Self Assessment • PGY 1: Interpersonal Communication/Professionalism • Who are you as a clinician? Define your ideal of professionalism in a physician • PGY2: Patient Care • Describe a challenging patient encounter • PGY3: Practice Based Learning and Improvement • Describe a patient encounter where there was a "medical error/Near miss" and what lessons you learned • PGY4: Systems Based Practice • How do you successfully handle patient care and teaching while paying attention to the flow within the department  C.Personal Reflection: • Goals for this year, Academic aspirations/thoughts for scholarly activity • Action plan/Next steps

14. One-Year Review: Promotion Criteria • In order for promotion, at the PGY 3 end of year evaluation, the following are the basic requirements needed in the resident’s portfolio: • Signed resident learning contract • Completed Professionalism checklist • Completed semi-annual evaluations • one Chief Complaint CC assessment per year (including one oral boards) • Procedural/Resuscitation Competency ACGME targets reached by the end of PGY 3 • 360 degree evaluations by the Clinical Nursing Group and Resident Peers • Self-assessment • If need remediation, during PGY 4 • PGY 4 will be promoted with a remediation clause and a signed contract of expectations prior to granting of graduation the following year • Remediation with appropriate mentorship and resources

15. Lessons Learned • Programmatic change • Time • Transparency • Engagement • Expectations and resistance • Action Plans with reasonable deadlines • Communication • Importance of review

16. Lessons Learned • Highlights • Resident Portfolios • Program manager meetings • Adult learner Contract • Professionalism Checklist • PGY 3 Promotion Criteria • 360 degree evaluations • Peer and Clinical Nursing Leadership Group • Self-Assessment • 360 degree evaluation • Self-reflection questions • Goals for year

17. Conclusions • Residency programs are expected to be fully compliant and integrated with the ACGME core competency assessment guidelines by 2011 • Residency programs should be proactive in designing and implementing an evaluation and assessment program that best fits their needs based on institutional strengths and available resources • We offer a comprehensive assessment program designed and implemented at our institution that is sensitive to the residents, faculty, and residency program needs for accountability, growth, remediation, and improvement

18. Questions? Contact Information: Christian Arbelaez, MD, MPH 617-732-8947 carbelaez@partners.org