Polishing the Silver: Competency Assessment for Licensed Professionals

1. Polishing the Silver: Competency Assessment for Licensed Professionals Deanna Williams Della Croteau Nora MacLeod-Glover Anthony Marini Ontario College of Pharmacists Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

2. Program Overview • History of Quality Assurance in Ontario • Creating and maintaining a Competency Assessment Program • Providing meaningful feedback • Evaluating for Impact Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

3. History of Quality Assurance in Ontario Background – Our Context • Professions are self-regulated • Re-licensure is not required • QA legislated by the provincial government • RHPA came into effect in 1993 • ~10,000 pharmacists on the Register Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

4. History of Quality Assurance in Ontario Important considerations about QA • Regulator accountability – mandate is public protection • Program designed to be a pro-active activity • Government requirements to be considered • Must be confidential, selection must be random Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

5. Creating & Maintaining the Program Quality Assurance Program Overview • Two-part Register • Self-Assessment activity • Learning Portfolio • The Practice Review Weekend • Clinical Knowledge Assessment • Standardized Patient Interviews • Practice Enhancement & Follow up Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

6. Creating & Maintaining the Program Acceptance by professionals • Education focus • Involve an assessment consultant • Program development by peers • Communicate constantly Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

7. Creating & Maintaining the Program Relevancy to practice • Ground program in standards of practice • Assess core skills • We focused on: • Managing therapeutic knowledge (CKA) • Communication skills (SPI) Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

8. Creating & Maintaining the Program Provide actionable feedback • Feedback report needs to be easy to understand and reflect opportunities for learning • Focus on both strengths and areas for growth • Consider all remediation options • Catalyst for creating professional development programs Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

9. Providing Meaningful Feedback Accept evaluation limitations • Assessments are really estimates • Successful participants have demonstrated they have skills to be able to practice safely • Unsuccessful participants are expected to address weaknesses identified • Regulator is a partner in remediation Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

10. Providing Meaningful Feedback Feedback report components • Performance Summary Table • Explanation • Self-Assessment Summary • CKA details (clinical content) • SPI details (three charts) Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

11. Providing Meaningful Feedback Practice Enhancement Support Members need to accept & understand learning needs and requirements • Peer Support Group • Professional Development Workshop • International Pharmacy Graduate Program • Ongoing follow up Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

12. Evaluating for Impact Determine focus of evaluation • Driving force is public accountability • Evaluation required by legislation • Understanding the impact on the practitioner is also of interest and can provide useful insights Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

13. Evaluating for Impact Determine measurable impacts • Seek impacts which influence practitioner’s ability to maintain and develop their professional skills • Our focus: • Knowledge improvement activities • Attitude and approach to patients • Changes in Practice* • Attitude toward QA as a whole Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

14. Evaluating for Impact Determined groups to include: • Self-Assessment • individuals selected from 2002 • have not completed the Practice Review • Practice Review • 1997 group helps understand impact of time • Practice Review • 2002 group helps understand immediate impact Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

15. Evaluating for Impact Our Process included: • Hired consultant to facilitate feedback • Conducted focus groups • completed written survey • participated in facilitated discussion • Developed in-depth surveys based on focus group content • print surveys for SAS group • telephone surveys for all others Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

16. What we learned - KIA Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

17. What we learned – A&A Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

18. What we learned - Changes Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

19. What we learned - QA Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

20. Qualitative Observations • Equally positive and negative • Impact not due to QA: already good practice • Open support for QA • Feelings of fear and intimidation • These are decreasing • Worst part is waiting for the day Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

21. Conclusions • PR had the greatest impact on the PR 97/98 group • Compared to the 2002 PR group • PR had greater impact on internationally-trained pharmacists • Increased understanding of expectations • Significant learning: greater support for QA expressed by those who have been through the process compared to those who have not • SAS was effective control group Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

22. Speaker Contact Information Deanna Williams, Registrar Della Croteau, Deputy Registrar Nora MacLeod-Glover, Manager, Continuing Competency Dr. Anthony Marini, Assessment Consultant Ontario College of Pharmacists 483 Huron Street, Toronto, Ontario M5R 2R4 Tel:416-962-4861 Fax:416-847-8282 nmacleod@ocpinfo.com www.ocpinfo.com Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona