Fundamentals of Assessment

1. Fundamentals of Assessment Todd L. Green, Ph.D. Associate Professor Pharmacology, Physiology & Toxicology PIES Seminar 2-23-11

2. Use of terms • Assessments = Observations /measurements • Evaluation = interpretations / conclusions

3. Uses of assessment • Individual learner • Educational action: Feedback • Administrative action: Judgment on advancement • Program • E.g., Sources of variance  Quality improvement • Accreditation

4. “Unit” of assessment • Single interaction of student and content • Series of encounters (observations by one teacher) • Series of teachers (in a rotation) • Series of teachers + exams (in a clerkship) • Series of clerkship (in a department) • Inter-clerkship (clinical year’s program)

5. PURPOSE (Why Assess) • Meet goals and objectives? • Were innovations successful? • Determine program strengths and weaknesses • Unanticipated strengths and problems • Growing interest in educational outcomes • “Residency programs should use outcome assessment in their evaluation of program educational effectiveness”1 • LCME, RRC require programmatic evaluation1 1Graduate Medical Education Directory, 2001-2002

6. PURPOSE (Why Assess) • Competing goals of stakeholders • “Tension between delivering the service and ensuring time for reflection and learning”2 • Help obtain and/or justify resources for your program 2Hayden, 2001

7. Process: Assessment & Evaluation

8. What are stakeholders and who are they?

9. Marshall University Joan C. Edwards School of Medicine Institutional Objectives Patient Care Students must be able to provide patient care that is compassionate, appropriate, and effectivefor the treatment of health problems and the promotion of health. Medical Knowledge Students must demonstrate knowledge of established and evolving biomedical, clinical, epidemiological and social-behavioral sciences, as well as the application of thisknowledge topatient care. Practice-based Learning and Improvement Students must demonstrate the ability to investigate and evaluate their care of patients, toappraise and assimilate scientific evidence, and to continuously improve patient care based onconstant self-evaluation and life-long learning.

10. Marshall University Joan C. Edwards School of Medicine Institutional Objectives Interpersonal and Communication Skills Students must demonstrate interpersonal and communication skills that result in the effectiveexchange of information and collaboration with patients, their families, and healthprofessionals. Professionalism Students must demonstrate a commitment to carrying out professional responsibilities and anadherence to ethical principles. Systems-based Practice Students must demonstrate an awareness of and responsiveness to the larger context and systemof health care, as well as the ability to call effectively on other resources in the system toprovide optimal health care.

11. Undergraduate Medical Education DURINGAFTER Logbooks Surveys Hours Placing Graduates Critiques Self assessment Exit interviews Student portfolios Review of write-ups Site Visit Written exams Written exams Performance Evaluation Performance Evaluation Grades/narratives (4th year OSCE, mini-CEX) Attitudinal 4th year medicine performance

12. Graduate MedicalEducation BEFORE DURING AFTER Self Assessment Logs Surveys Hours Placing Graduates Learner Critiques 360 Degree Evaluation Chart Review Exit interviews Attitudinal Written Exams Written Exams Written Exams Performance Eval Performance Eval Performance Eval GPA Disciplinary Action Narratives Narratives Research/Teaching Global Evaluation National Society Participation

13. Undergraduate Medical Education

14. Undergraduate Medical Education

15. Graduate Medical Education

16. Graduate Medical Education

17. Principles of Assessment Fairness Consistency Simplicity Pangaro HMI 07

18. Principle 1 : Fairness • to society : competence (P/F) • to students : transparency and feedback • to teachers : faculty development and protection Pangaro HMI 07

19. Principle 2 : Consistency • Validity • Are you measuring what you want? • Reliability - Inter-teacher and inter-clerkship: suitable for high-stakes decisions? Pangaro HMI 07

20. Reliability versus validity X X X X X X X X X X X X X X X Reliable Valid

21. Principle 3: Simplicity • How we frame the question? • How we ask the question? • How do we use the answer? Pangaro HMI 07

22. Strategy • Simplicity leads to acceptance and use • Acceptance leads to consistency • Consistency = fairness Pangaro HMI 07

23. Framing the question simply • What do we expect of learners (students)? • How do we communicate this to them and to their teachers? Pangaro HMI 07

24. In order to get the information you need from an assessment and evaluate the results, you have to ask the right questions.

25. Exercise • Find the person nearest you. Person to the left is “A”. • Person A: In 30 seconds tell the person nearest you what it takes to pass a course/clerkship you are familiar with. How is this assessed? • Person B: In 30 seconds tell the person nearest you what it takes to get the highest grade in a course/clerkship you are familiar with. How is this assessed? Pangaro HMI 07

26. Discussion Based on the information you have: • Are you able to determine your assessment system’s strengths and weaknesses? • Can you determine whether the system has met the required levels of success for the purpose? • Is it clear how your own program evaluation system will be assessed?