Quantifying and Measuring Professionalism: Demonstrating the Professionalism O utcomes of Pharmacy Curricula

2. Quantifying and Measuring Professionalism: Demonstrating the Professionalism Outcomes of Pharmacy Curricula

3. Presenters • Dana Hammer,RPh, PhD • University of Washington, Pharmacy • Kristin, K. Janke, PhD • University of Minnesota, Pharmacy • Katherine A. Kelley, PhD • Ohio State University, Pharmacy • Steven A. Scott, PharmD • Purdue University, Pharmacy

4. Session Goals • Gain an understanding of the unique challenges of measuring professionalism • Explore the unique aspects of professionalism as it relates to interdisciplinary health care teams • Gain and enhanced understanding of how to use professionalism assessment data to inform curricular change

5. Creating a Tool to Measure Professionalism Behaviors: A Multi-Institutional Pilot Steven A. Scott, PharmD Kristin K. Janke, PhD Katherine A. Kelley, PhD

6. Best Practices for Professionalism Assessment • Multiple assessors • Multiple settings • Multiple methods • Formative • Beginning early in the curriculum • Improve existing tools rather than create new Lynch, et al, 2004

7. Activity When working with colleagues, I behave in a ways that communicate my respect for them. • Strongly agree • Agree • Neutral • Disagree • Strongly disagree

8. Barriers to Measuring Professionalism • Definitions tend to be abstract • Context specific • Reluctance to report or address unprofessional behaviors • Ceiling effects Ginsburg et al, 2000

9. Case Example • A student you have on rotation dresses professionally and appropriately everyday. You have noticed that the student tends to talk down to and act rudely toward the office staff members at your practice site. • Identify/name the professional behaviors of this student? • What professional attitudes would you attribute to this student?

10. Professionalism: Internally held vs Externally Exhibited • Attitude or behavior? • I am respectful of others • I dress professionally on rotation • Attitude or behavior? • I am trustworthy • I protect patient confidentiality

11. Challenges Associated with Measuring Professionalism • Often more difficult to measure professional behavior during the didactic curriculum than during the experiential curriculum • Fewer opportunities • Situations are often simulated vs. real-life • Expectations may not be consistent • Faculty deal with many more students than in the practice setting

12. Challenges Associated with Measuring Professionalism • The major focus of assessments and assessors is to evaluate the student’s knowledge and skills, not necessarily behaviors. • Professionalism is more difficult to assess than knowledge or the demonstration of a skill and usually plays a minor role in any course grade. • Most assessors have not been trained to evaluate professional behaviors. (Bryden, et al, 2010)

13. Challenges Associated with Measuring Professionalism • Assessments of professionalism are often sporadic and isolated and thus the professional development of students is difficult to monitor. (Lucey and Souba, 2010) • Assessment of professionalism is often carried out by a variety of individuals who have not observed the student previously. • Isolated incidences of unprofessional behavior may not be communicated and thus undesirable behaviors must be rediscovered as the student matriculates through the curriculum.

14. Challenges Associated with Measuring Professionalism • Volunteer faculty, other healthcare professionals, and patients are often uncomfortable making summative judgments about student professionalism after limited interactions. (Swick, H., 2007, Hickson, et al, 2007) • Other healthcare professionals may focus more on roles than on behaviors. • Peers are often hesitant to provide honest evaluations. (Reed, et al, 2008)

15. Challenges Associated with Measuring Professionalism • Behavioral assessments conducted prior to graduation are prone to being influenced by student motivation to achieve a desired grade, rather than being reflective of the student’s true value system. (Brown and Ferrill, 2009) • Although valid assessment tools have been developed, no standard approach to measuring professionalism has emerged. (Stern DT, et al, 2006)

16. Multi-Institutional Development • 7 institutions* collaborated to create an instrument to measure behavioral aspects of professionalism • GOAL: use previous literature as a foundation to create an instrument that minimizes ceiling effects • Pilot: Spring 2009 in three schools *Big 10: Purdue University, University of Illinois Chicago, University of Iowa, University of Michigan, University of Minnesota, Ohio State University, University of Wisconsin

17. Multi-Institutional Work • 7 institutions* collaborated to pilot the instrument • GOAL: cross-validation of instrument • Spring 2010 with first and third year students *Purdue University, University of Illinois Chicago, University of Iowa, University of Minnesota, Ohio State University, University of Wisconsin, AND University of Kansas

18. The Instrument • Professionalism Assessment Tool (PAT) • Foundational instruments: • Physicianship Evaluation Form (Papadakis, 2001) • American Pharmacists Association white paper on student professionalism (2000) • Hammer et al 2000 • Chisholm et al 2006

19. The Instrument • Domains • Reliability, responsibility and accountability • Lifelong learning and adaptability • Relationships with others • Upholding principles of integrity and respect • Citizenship and professional engagement

20. Self-assessment Ratings • Foundations • Pangaro, 1999 • RIME: Reporter, interpreter, manager, educator • Wanted a basic level • Novice • Miller, 1990 • Knows, knows how, shows, shows how, does

21. The Final Ratings “Know”: I understand these responsibilities, but may perform one or more inconsistently, at times. “Know how”: I understand these responsibilities and perform them in a reliable, consistent and accountable manner. “Show”: Without prompting or support from instructors, preceptors or managers, I determine when and how to engage in these responsibilities. “Show how and Does”:  I am confident in assisting others with these responsibilities or proposing or creating options to fulfill these responsibilities.  “Teach how”: I have mastered these responsibilities and desire to learn more and share my learning with others.  I demonstrate maturity, confidence and an ability to educate others in these areas through the use of evidence and strong interpersonal skills.

22. Kelley KA., et al. A Multi-Institutional Pilot for Cross-Validation of a Tool to Measure Professionalism Behaviors. Amer J Pharm Educ, 2011 article 179.

23. Cross-validation • Internal consistency was good (all 5 factors coefficient alpha >0.9) • Both exploratory and confirmatory factor analysis techniques were used • Factor analysis demonstrated that all items were correctly loaded under their assigned domain

24. Limitations • 155 surveys removed due to incomplete data • Homogeneity of the survey sample (all 7 pilot schools are public research intensive)

25. Future Directions • Longitudinal data collection to determine growth over time • Use of instrument by external evaluators (preceptors and faculty) • Examine possibilities in curricular revision

26. Lessons Learned • Measuring professionalism has value • Assessing professionalism is possible • Building on previous literature is helpful • Multi-institutional work can be productive • There is more work to be done! • Multiple assessors & multiple settings • Longitudinal: beginning early, using multiple measures • Formative (for student and curriculum!)

27. Challenges, Opportunities, and Best Practices for Interprofessional Professionalism Dana Hammer, RPh, PhD

28. Closing the Loop: Using Assessment Data to Improve Professionalism Curricula Group Discussion

29. Discussion Questions • How do you assess professionalism in your curriculum? Or how would you like to assess professionalism? • How does professionalism assessment data feedback to improve the curriculum? Or how would you like to use professionalism data to improve your curricula?