Orientation, Milestones, and Initial Assessment of New Interns

1. Orientation, Milestones, and Initial Assessment of New Interns Webinar Presentation (June 11, 2014): Cindy Works MD, Brian Veauthier MD Content: Brian Veauthier MD, Beth Robitaille MD University of Wyoming Family Medicine Residency - Casper

2. Objectives • Review benefits of evaluating new interns including ACGME Requirements • Describe a process we started to evaluate our new interns based on the Family Medicine Milestones • Share our successes and pitfalls • Present a case study of a current intern • Open the floor for interactive discussion

3. Why Evaluate New Interns? • Because the ACGME says we need to • Avoid the December shock factor of a poorly performing resident • Variance in medical school education - assess strengths and weaknesses

4. What does the ACGME require? V.A.2.b) The program must: V.A.2.b).(1) provide objective assessments of competence in patient care and procedural skills, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice based on the specialty-specific Milestones; (Core) V.A.2.b).(1).(a) This assessment must involve direct observation of resident-patient encounters. (Detail) V.A.2.b).(1).(b) Each resident should be assessed in each of the six competency areas on entrance into the program. (Detail)

5. Our Process

6. The Tools

7. The Tools

8. The Tools

9. The Tools

10. The Tools

11. The Tools

12. The Tools

13. The Tools

14. The Tools

15. The Tools

16. The Tools

17. The Tools

18. The Tools

19. Sample Evaluations

20. Intern A “The Rock Star"

21. Intern A- Comments/Recommendations “Excellent test scores 440. Mean from last year 386 with std 74” “Professionally appropriate interaction” “Faculty advisor feels like she won the lottery”

22. Intern B “Should be OK"

23. Intern B- Comments/Recommendations “Would benefit from showing more confidence” “Needs significant work on suture skills” “Performed lower than expected on the mock ITE. Score 300 . Mean from 2012- 386 with std 74. consider early IEP” “Professional interactions” “Very quiet and not interactive during orientation-is she fitting in ok?”

24. Intern C “Help"

25. Intern C- Comments/Recommendations “Give literature on common issues that is clinical” “Give specific info on common issues” “Needs organized approach to EKG- Wicks handout” “Scored 220 on mock ITE- mean last year was 386 with STD 74. Strongly recommend early IEP” “Has demonstrated appropriate professional interaction so far” “Some concerns with language barriers” “Interacting well with peers”

26. INTERN c MILESTONE IMPLEMENTATION INTO AN INDIVIDUAL EDUCATION PLAN

27. BIG REVEAL: ORIENTATION MILESTONE EVALUATION • Significant deficits in her medical education leading to obvious need for intervention • Milestone deficiencies most obvious in: • Medical Knowledge • Patient Care • Communication

28. JULY: ROUTINE ADVISOR MEET & GREET • Reviewed Milestone deficits • Outlined her initial improvement plan/IEP • Outlined close monitoring and follow-up

29. OCTOBER: ROUTINE ADVISOR MEETING AND EVALUATION • Some progress noted, but insufficient to become commensurate with her peers • Confirmed continued deficits and need for formal intervention and remediation IEP • General consensus: she was functioning at early MS-4 level

30. OCTOBER: ADVISOR & PROGRAM DIRECTOR MEETING • Intern C formally notified of deficiencies • Remediation IEP developed based on the Milestones, concentrating on the most relevant competencies: • Medical Knowledge • Patient Care • Communication • Practice-based Learning and Improvement • Specific goals: to progress from Level 0to Level 1 by January in the identified sub-competencies • Key away rotation of inpatient pediatrics in the interim

31. JANUARY: ADVISOR & PROGRAM DIRECTOR MEETING • Intern C reassessed on the Milestones • Core program rotational evaluations • Video evaluation • Field notes • ITE • Verbal feedback from away pediatric rotation attending • Senior preparedness OSCE • Demonstrated improvement in several areas but did not reach goal of Level 1 on all identified sub-competencies • Key float rotation in February/March provided next opportunity to achieve the progression required to proceed on as a PGY-2: progress towards Level 2 on identified sub-competencies

32. MARCH: ADVISOR & PROGRAM DIRECTOR MEETING • Intern C reassessed on the Milestones • Field notes • Evaluations • Verbal feedback from faculty • Written feedback from senior float resident • Other IEP requirements (ITE and board preparation) • Demonstrated enough continued progress in the identified sub-competencies to continue at our residency • Remains on remediation IEP

33. MEDICAL KNOWLEDGE

34. PATIENT CARE

35. COMMUNICATION

36. PRACTICE-BASED LEARNING AND IMPROVEMENT

37. CONCLUSIONS • Orientation Milestone evaluation via OSCE and mock ITE helped to identify deficiencies early leading to early implementation of improvement plan • Monitoring Milestone progression throughout the year helped to define specific and objective deficiencies • The Milestone based IEP provided the appropriate detail and language to set specific goals and plans of action • The Milestones played an important role in this successful (although ongoing) remediation

38. How did it work?

39. How did it work?

40. How did it work?

41. Summary

42. Discussion