Author: Dr. Craig Campbell Date: October 4, 2012

1. Incorporating Assessment into Specialist CPD Programs The recommendations of an expert panel for the Royal College Author: Dr. Craig Campbell Date: October 4, 2012

2. Conflict of Interest Declaration • I am a full-time Directorwith the Royal College. • I have no financial relationships with members of pharmaceutical or medical supply companies. • I do not hold any research grants funded by industry. • I do not serve on an advisory board of any for-profit industry. • I have a bias about CPD and Lifelong learning!

3. Learning Objectives At the end of this presentation you will be able to: Discuss the role of self-assessment within continuing professional development Describe the rationale for including assessment strategies within systems of continuing professional development Explain some of the key challenges and barriers to assessment.

4. Role for Assessment in CPD Assessment is an educational imperative Informs: • Provides individuals, groups/teams with data to help identify unperceived needs Guides: • Identification and development of learning opportunities through providing credible feedback

5. Assessment in Practice Two Key Questions • Is there are role for self-assessment within the spectrum of assessment strategies? • What does the literature say about their efficacy of formal assessment in promoting learning, enhancing competence or improving performance?

6. Role for Self-assessment? • Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. Davis et al JAMA 2006 • “I’ll never play professional football” and other fallacies of self-assessment. KW Eva, G Regehr JCEHP 2008

7. Self-Assessment in Practice Key Messages ‘Personal, unguided reflection’ or any global judgment of one’s ability in a particular domain is… • Poorly performed • Essentially flawed • Unlikely to be enhanced through training or education! (Canadian speak for “it’s a waste of time!”)

8. Reflecting “in action”

9. Reflecting “on action”

10. Self-Assessment in Practice How accurate are decisions to pause and learn before acting? “Knowing when to look it up: a new conception of self-assessment ability.” Eva, Regehr, Academic Medicine 2007 Focus on: • Situational awareness when at the limits of knowledge, ability, experience OR • When confidence in ability is lacking.

11. Self-Monitoring in Practice Definition of Self-Monitoring “the moment-by-moment awareness of the likelihood that one maintains the skill/knowledge to act in a particular situation”: Conclusion Greater accuracy in self-monitoring than any global aggregation of performance. Eva and Regehr Adv Health SciEduc Theory Pract. 2011

12. Self-Monitoring of Physicians Focus for Future Research • Exploring the concept of self-guided learning. Brydges: Academic Medicine Oct 2010 • The quality of questions and use of resources in self-directed learning: personal learning projects in the maintenance of certification program. T Horsley: JCEHP 2009

13. Assessment for Learning What is the role for peer or personal assessments of competence or performance within mandatory systems of continuing professional development?

14. Some Definitions…. Competence “The array of abilities across multiple domains or aspects of physician performance in a certain context. Statements about competence require descriptive qualifiers to define the relevant abilities, context and stage of training. Competence is multi-dimensional and dynamic. It changes with time, experience and setting.” ICBME Collaborators Medical Teacher August 2010

15. Some Definitions…. Progression of competence “ For each aspect or domain of competence, the spectrum of ability ranges from novice to mastery. The goal of medical education is to facilitate the development of a physician to the level of ability required for optimal practice in each domain. At any given point in time, and in a given context, an individual physician will reflect greater or lesser ability in each domain” Same group, same journal

16. Some Definitions…. Performance “ An evaluation of a physician’s behaviors or actions in their professional practice context. Performance assessment uses multiple processes (direct observation, chart audit, practice review) and information sources (charts, patient registries, prescribing data) to generate data on adherence or compliance against selected measures or metrics to facilitate the provision of feedback” My thoughts, unpublished

17. Physician Assessment Royal College Strategic Plan Included two important goals: • Promote competence and performance assessment through lifelong learning! • Develop standards for the performance assessment of physicians in practice

18. New MOC Framework

19. Assessment in CPD: Some Key Assumptions Assessment strategies and processes must be: • Embedded within the practice context and supported by the health system • Relevant to each dimension of professional practice and across all content domains • Supported within environments that are safe and divorced from threats of litigation or fear of failure

20. Assessment in CPD: Some Key Assumptions Assessment strategies and processes must be: • Applicable for individuals, groups or teams. • Able to generate data with feedback. • Enable the identification of areas where further learning should be focused

21. Invitational Conference on Assessment in Practice Purpose: • Review the evidence on the impact of assessment strategies on learning and behavior change. • Define the ability of processes and tools to assess competency domains • Develop a set of recommendations for implementation of assessment strategies within the MOC Program

22. Invitational Conference of Assessment in Practice 5 Groups of Experts • Multi-Source Feedback • Simulation • Audit and Feedback • Learning Portfolios • Information Systems

23. Multi-Source Feedback Research has established … • Feasibility and cost effectiveness • Reliability and validity • Promotes change when feedback is • Credible, specific and accurate and/or • When coaching was provided Sargeant JCEHP 2011 Miller BMJ 2010 Violata C. BMJ 2003

24. Multi-Source Feedback Our Expert Panel’s Conclusions • Valuable formative assessment strategy for individual physicians. • Empiric evidence to establish reliability in content areas applicable to all physicians. • Relevant to observable behaviors in three CanMEDS Roles: Communicator, Collaborator, Professional. • Quality of the data / feedback provided influence and guide physician response.

25. Audit and Feedback Jamtvedt G, Young JM, Kristoffersen DT, O'Brien MA, Oxman AD. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews First publication 2003: 88 trials Update in 2006: 118 trials 2nd update June 2012: 140 trials

26. Conclusions Impact on performance and health outcomes is small to moderate Relative effectiveness is enhanced when: • Baseline compliancewith recommended practice was low! • The feedback is provided: • By a colleague or supervisor • More than once • Delivered in both written/oral formats • Identifies targets with an action plan

27. Centrality of Feedback… Impact of workplace based assessment on doctor’s education and performance: a systematic review Miller, BMJ 2012 “Performance changes were more likely to occur when feedback was credible and accurate or when coaching was provided to help subjects identify their strengths and weaknesses”

28. Audit and Feedback Our Expert Panel’s Conclusions • Valuable assessment strategy for individual physicians in their Medical Expert roles. • Validity (face and content) based on trustworthy data that is patient specific. • Multiple systematic reviews to establish the conditions that influence significant behavior change. • Feedback is central and critical • Applicable to multiple sources of data

29. Research on Simulation Issenberg,S.B, McGaghie W.,M Petrusa E.R, Goprdon D.L., Scalese,R.J. Features and uses of high-fidelity medical simulations that lead to effective learning: A BEME systematic review 2005 Medical Teacher 109 research studies from 1969 – 2003 (57% from 2000 – 2003) Approximately 20% were CME/CPD.

30. Virtual Simulation Computerized virtual patients in health professional education: a systematic review and meta-analysis Cook 2010 Academic Medicine 85:1589 • Key is the instructional and narrative design that leads to learning NOT the technology • Technology for VP has grown substantially over the past decade leading to fundamental advances such as: • Virtual Patient interoperability standards • Authoring systems

31. Simulation Our Expert Panel’s Conclusions • Valuable assessment strategy for individuals, groups or teams. • Reliability and validity (face, content, predictive) established for individuals. • Multiple systematic reviews demonstrate simulation with deliberate practice shows better results. • Relevant to multiple competency domains including Medical Expert, Communication, Collaboration

32. Our Current Vision “Informed Self-Assessment” Definition “A set of processes through which individuals use external and internal data to generate an appraisal of their own ability”. Mann K, Sargeant J. Acad Med 2011

33. Informed Self-Assessment Conceptual Model • Sources of information - multiple • Interpretation of information • Response to information • External and internal conditions influencing each of these steps • Tensions arising from competing data and influences Sargeant J. Acad Med 2010

34. Informed Self-Assessment Some key lessons for the future • Work environment must increase the frequency, credibility and safety of seeking and receiving feedback • Informed self-assessment is an essential route to developing, maintaining, and improving performance.

35. Some Conclusions Assessment for Learning Require the intentional integration of multiple strategies that: • Support self-monitoring and the competencies of self-directed learning. • Provide quality data with credible feedback to identify the development of the ‘path to improvement’.

36. Current status… Our Conclusions and Next Steps… • Assessment must be a mandatory component of any lifelong learning strategy anchored in practice. • We are committed to completing a scoping review to inform a set of recommendations for Council on “strategies and options to affirm the continued competence of Fellows of the Royal College”.

37. Thanks for your attention