A Practical Framework for Integrating Assessment, Learning Curriculum

1. 03 May 2006 IV National Congress Medical Education A Practical Framework for Integrating Assessment, Learning & Curriculum Stewart Mennin, PhD Professor Emeritus University of New Mexico School of Medicine

2. May 2006 IV National Medical Education Congress Overview Planning curriculum assessment Competence & capability Matrices � Blueprinting Assessment of Competencies Preparing teachers & students for assessment Governance & assessment

3. May 2006 IV National Medical Education Congress Planning Assessment & Curriculum National & regional priority health needs Capabilities & Competencies Program (curriculum) evaluation Student assessment Objectives Curriculum design & implementation Continuous evaluation Curriculum is too important to be planned first. Curriculum is too important to be planned first.

4. Fraser and Greenhalgh BMJ 2001 323 Oct 6 Traditional education and training largely focuses on enhancing competence (knowledge, skills, and attitudes) In today's complex world, we must educate not merely for competence, but for capability (the ability to adapt to change, generate new knowledge, and continuously improve performance) Capability is enhanced through feedback on performance, the challenge of unfamiliar contexts, and the use of non�linear methods such as story telling and small group, problem based learning Education for capability must focus on process (supporting learners to construct their own learning goals, receive feedback, reflect, and consolidate) and avoid goals with rigid and prescriptive content In complex adaptive systems the behaviour of the individual agents, and therefore of the system of which they are part, evolves in response to local feedback about the impact of actions. Similarly, the basis of transformational learning is the information that is fed back to learners about the impact of their own actions and those of others. An education process that provides feedback about performance as it takes place will enhance capability. O Fraser and Greenhalgh BMJ 2001 323 Oct 6 Traditional education and training largely focuses on enhancing competence (knowledge, skills, and attitudes) In today's complex world, we must educate not merely for competence, but for capability (the ability to adapt to change, generate new knowledge, and continuously improve performance) Capability is enhanced through feedback on performance, the challenge of unfamiliar contexts, and the use of non�linear methods such as story telling and small group, problem based learning Education for capability must focus on process (supporting learners to construct their own learning goals, receive feedback, reflect, and consolidate) and avoid goals with rigid and prescriptive content In complex adaptive systems the behaviour of the individual agents, and therefore of the system of which they are part, evolves in response to local feedback about the impact of actions. Similarly, the basis of transformational learning is the information that is fed back to learners about the impact of their own actions and those of others. An education process that provides feedback about performance as it takes place will enhance capability. O

5. Competency Comparisons

7. May 2006 IV National Medical Education Congress Competent Capable

9. May 2006 IV National Medical Education Congress Integrated Curriculum

10. May 2006 IV National Medical Education Congress Assessment Methods Selection Supply Performance

11. May 2006 IV National Medical Education Congress Selection Methods MCQ Matching Extended Matching True False Multiple True False Extended matching good for more complex and diagnostic questionsExtended matching good for more complex and diagnostic questions

12. May 2006 IV National Medical Education Congress Supply Methods MEQ Essay (long & short) Practical Short Answer Written simulations

13. May 2006 IV National Medical Education Congress Performance Assessment Procedures � skills lab OSCE (with skills stations) Mini CEX Long Case Logs Portfolio Clinical Notes Observation Other

14. May 2006 IV National Medical Education Congress

15. May 2006 IV National Medical Education Congress

16. May 2006 IV National Medical Education Congress

17. May 2006 IV National Medical Education Congress Competencies Critical Reasoning & Integration of Knowledge Patient Care Communication Interpersonal Skills Professional Attitudes, Values, Ethics & Self Assess Systems & Community Based Practice Permanent Education Humanization

18. May 2006 IV National Medical Education Congress

19. May 2006 IV National Medical Education Congress Phase I

20. Assessment As Learning How much When Who Culture of formative assessment How much When Who Culture of formative assessment

21. May 2006 IV National Medical Education Congress

22. May 2006 Critical Integration of Knowledge Skills Problem Identification Hypothesis Generation Testing Hypothesis Evaluating Data Describing Mechanism Research Analysis Communication

23. May 2006 IV National Medical Education Congress CRIK Skills Spidergram

24. May 2006 IV National Medical Education Congress CRIK Skills Spidergram

25. May 2006 IV National Medical Education Congress CRIK Skills Spidergram

26. May 2006 IV National Medical Education Congress Assessment Item banks Progress tests

27. May 2006 IV National Medical Education Congress

28. May 2006 IV National Medical Education Congress

29. Exam Standardized patient stations Content Areas Interview PE Verbal feedback Write-up Comments Phase ISummative 1One 15 minute, interviewHistory/Communication SkillsBasicInformedFacultySummative 2One 50 minute, complete H&P End of clinical skills courseBasicInformedFacultyxSummative 3Two 20 minute, focused H&PNeurology & Psychiatry examBasicInformed-Summative 4Five 20 minute, focused H&PCV & Pulm exams & other examBasicDecide-Summative 5One 50 minute, complete H&POne 20 minute, focused H&PReview complete exam Abdominal examPatient agendaDecide-xSummative 6Three 20 minute, focused H&PEndo/Renal/Repro Patient agendaDecide-Exam Standardized patient stations Content Areas Interview PE Verbal feedback Write-up Comments Phase ISummative 1One 15 minute, interviewHistory/Communication SkillsBasicInformedFacultySummative 2One 50 minute, complete H&P End of clinical skills courseBasicInformedFacultyxSummative 3Two 20 minute, focused H&PNeurology & Psychiatry examBasicInformed-Summative 4Five 20 minute, focused H&PCV & Pulm exams & other examBasicDecide-Summative 5One 50 minute, complete H&POne 20 minute, focused H&PReview complete exam Abdominal examPatient agendaDecide-xSummative 6Three 20 minute, focused H&PEndo/Renal/Repro Patient agendaDecide-

30. May 2006 IV National Medical Education Congress Tutorial Assessment: Criteria Observable behaviors Change & growth Helping behaviors Learning contract

31. May 2006 IV National Medical Education Congress Tutorial Assessment Knowledge base Reasoning Process/Decision-Making Communication Assessment Professional behavior

32. May 2006 IV National Medical Education Congress

33. May 2006 IV National Medical Education Congress Phase II Subject examinations OSCEs

34. May 2006 IV National Medical Education Congress Phase II OB/Gynecology & Surgery Pediatrics & Neurology � Psychiatry Internal Medicine & Family Medicine

35. May 2006 IV National Medical Education Congress Clinical OSCEs 5 std pts stations 15 min each three, 10 minute clinical write-ups in Calibrated Peer Review� two, 5 minute feedback sessions Skills stations Up to 14, eight minute skills stations A student will take this Phase II Performance Examination after completing 8 weeks of Internal Medicine and 8 weeks of Family medicine. This student, one of up to 23 students who have just completed the same rotations, will spend a morning seeing five standardized patients. Some of the patients will present with the sorts of clinical problems seen on the Internal Medicine and Family Medicine services and there will be patients with presentations that are not necessarily related to the rotations just completed. The student will write clinical notes after three of the patient evaluations. These notes will be entered into the Calibrated Peer Review� computer program. A student will take this Phase II Performance Examination after completing 8 weeks of Internal Medicine and 8 weeks of Family medicine. This student, one of up to 23 students who have just completed the same rotations, will spend a morning seeing five standardized patients. Some of the patients will present with the sorts of clinical problems seen on the Internal Medicine and Family Medicine services and there will be patients with presentations that are not necessarily related to the rotations just completed. The student will write clinical notes after three of the patient evaluations. These notes will be entered into the Calibrated Peer Review� computer program.

36. May 2006 IV National Medical Education Congress Assessment Tools Checklists Global ratings Direct observation Self assessment Patient assessment Supply Tests (MCQs)

37. Phase II: Blueprint Clinical Skills

38. May 2006 IV National Medical Education Congress

39. Phase II: Blueprint Clinical Skills

40. May 2006 IV National Medical Education Congress

41. Phase II: Blueprint Clinical Skills

42. May 2006 IV National Medical Education Congress

43. May 2006 IV National Medical Education Congress

44. May 2006 IV National Medical Education Congress Phase III: Clinical Skills Blueprint

45. May 2006 IV National Medical Education Congress Standard Setting Modified Angoff Method

46. May 2006 IV National Medical Education Congress Faculty Development in Assessment Test item writing Formative feedback Tutorial � classroom � clinic Clinical assessment Community � based assessment How people learn Collaborative Teaching/learning

47. May 2006 IV National Medical Education Congress Governance of Assessment Curriculum committee Assessment review committee Test review Complaints % new items Accountability Responsibility & authority

48. May 2006 IV National Medical Education Congress GOALS OF ASSESSMENT Enhance student learning Guide faculty teaching efforts Inform curricular development & revision Fulfill responsibilities to students, institution & society Contribute to research

49. May 2006 IV National Medical Education Congress

50. May 2006 IV National Medical Education Congress

51. May 2006 IV National Medical Education Congress

52. May 2006 IV National Medical Education Congress

53. May 2006 IV National Medical Education Congress

54. May 2006 IV National Medical Education Congress Checklist/Global Rating Scale Comparison Checklists Dichotomous scores (yes/no) Relatively objective Record rather than interpret Station specific Rating Scales Ordinal level measures (poor, good, excellent) Relatively subjective Station independent

55. May 2006 IV National Medical Education Congress Global Rating Scale Scoring Methods Holistic Overall measure of the performance Analytic Discrete dimensions separated

56. May 2006 IV National Medical Education Congress Psychometric Comparison Global rating scales Equally or more reliable across stations Better construct validity Better concurrent validity

57. May 2006 IV National Medical Education Congress Phase I