1 / 26

Life Beyond MCQs: Performance-Based Assessments for Biomedical Sciences

Life Beyond MCQs: Performance-Based Assessments for Biomedical Sciences. Edward C. Klatt MD Professor of Pathology Biomedical Education Program Director Department of Biomedical Sciences Mercer University School of Medicine Savannah, Georgia, USA. Objectives.

finian
Download Presentation

Life Beyond MCQs: Performance-Based Assessments for Biomedical Sciences

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Life Beyond MCQs: Performance-Based Assessments for Biomedical Sciences Edward C. Klatt MD Professor of Pathology Biomedical Education Program Director Department of Biomedical Sciences Mercer University School of Medicine Savannah, Georgia, USA

  2. Objectives • Identify forms of performance-based assessment applicable to biomedical science courses • Outline strategies for deployment of performance-based assessments with optimal use of faculty resources • Describe how performance-based assessments are utilized to promote attainment of multiple student competencies • Define administrative oversight methods for tracking performance and outcomes

  3. Competencies • Medical Knowledge • Interpersonal Skills and Communication • Professionalism • Patient Care • Practice Based Learning and Improvement • Systems Based Practice

  4. Performance-Based Assessments • Students must attain multiple competencies. • Biomedical science courses often focus on the knowledge competency (and so do the students). • It is possible to incorporate performance-based assessments that promote communication, interpersonal skills, practice skills, and professionalism.

  5. Performance-Based Assessments • Written assignments • Subject matter expert and public evaluations of a veterinary toxicology course brochure-writing assignment. J Vet Med Educ. 2013 Spring;40(1):19-28. • Oral examinations • Research projects • Student presentations • JIAMSE 2009;19(4):150-154.

  6. Practice-Based Assessments: Examples • Prescription writing • Meeting the challenge of prescribing and administering medicines safely: structured teaching and assessment for final year medical students. Med Educ. 2003 May;37(5):434-7. • Drug discovery process • Simulated drug discovery process to conduct a synoptic assessment of pharmacy students. Am J Pharm Educ. 2014 Mar 12;78(2):41. • Performing a gram stain • Delivering bad news

  7. Teaching and Learning in Medicine. 2001;13(2):74-79.

  8. Student Perceptions • Students may view knowledge assessments as the rate-limiting step to a career. • Students may discount performance assessments as not essential. • Anything not assessed can be safely ignored. • Are there passing standards? • Does a minimum level of competency need to be demonstrated?

  9. Assessment Comparisons • In general, a global assessment of a performance is as valid as any number of performance components you score separately. • takes less time to score fewer components • Is feedback given on the components? • More faculty viewing a performance provide more valid and reliable results, but more time is required.

  10. Assessment Outcomes • Assessing students' communication skills: validation of a global rating. Adv Health Sci Educ Theory Pract. 2008 Dec;13(5):583-92. • As detailed checklists have been shown to be not well suited for the assessment of communication skills for different reasons, this study aimed to validate a global rating scale. • Results show that communication skills can be validly assessed by trained non-expert raters as well as standardized patients (increases the available pool of faculty).

  11. Assessment Outcomes • Activities of daily living: reliability and validity of gross vs specific ratings. Arch Phys Med Rehabil. 1981 Apr;62(4):161-6. • ... when activities can be broken down into independent task components, specific protocols are the assessment method of choice. When activities are made up of highly interdependent task components, a behaviorally anchored gross rating protocol may be more effective.

  12. Assessment Outcomes • Reproducible and accurate recognition of presence and severity of ataxia in horses with neurologic disease is important when establishing a diagnosis, assessing response to treatment, and making recommendations that might influence rider safety. • Clinicians should be aware of poor agreement even between skilled observers of equine gait abnormalities, especially when the clinical signs are subtle. • J Vet Intern Med. 2014 Mar-Apr;28(2):630-8.

  13. But It’s Not Fair ! • Review of grades awarded .... to the written reports of 4th year medical students showed a highly significant difference of approximately half a grade between supervisors and second markers with supervisors marking higher. • ...use of structured marking sheet for assessment of undergraduate medical students, supervisors marks are not associated with a halo effect, but leniency does occur. • BMC Med Educ. 2004 Nov 29;4(1):28.

  14. Halo Effect • Grades awarded by 2 independent graders to undergraduate projects were analyzed with a correlated uniqueness model. Grades showed substantial halo despite being awarded by expert assessors at the time of reading the work. • Halo effects in grading student projects. J Appl Psychol. 2007 Jul;92(4):1169-76.

  15. Peer Evaluation • Advanced Pharmacy Practice Experience Course, inpatient and outpatient settings, each 5 weeks long. Students were evaluated using a 19-item grading rubric designed to assess 5 components: general objectives/professional performance, general knowledge, clinical skills, communication skills, and formal presentations. Each item was rated on a 5-point categorical scale. • Self-reported student grades were lower than the faculty-reported grade overall. • Students graded their peers higher than did faculty members. • Am J Pharm Educ. 2011 Sep 10;75(7):130.

  16. Faculty Perceptions • Why are we doing this? • I’m already too busy. • We don’t agree on the standards of performance. • We’ll pass the students along anyway. • The students will get that later.

  17. Student Perceptions • It’s too subjective. • It’s not important. • I’ll spend too much time practicing for it. • I’ll not have the opportunity to practice for it. • I won’t do this again.

  18. Response Process • Response process refers to the cognitive processes involved in responding to the examination as both a learner and an evaluator. • Students must be familiar with the type of testing. • Raters must be accurate in their judgments. • Calculation and determination of scores must be appropriate. • Choice of tools used for rating (e.g., checklists) must be defensible. • Mt Sinai J Med. 2009;76(4):365-71.

  19. Performance-Based Assessment Issues • Testing examinees in realistic situations does not make test design and domain sampling simple and straightforward. • Even a realistic simulation is still a simulation and examinees do not behave the same as in real life. • High-fidelity methods yield rich examinee behavior that can be difficult to score fairly.

  20. Performance-Based Assessment Issues • Performance in one context (patient case) does not predict performance in other contexts well. • Validation is difficult when using other assessment methods targeting different skills. • The difficulty in development of test materials means that small numbers of exercises are used, making security an issue.

  21. Performance-Based Assessment Issues • Assessment methods can have an unpredictable impact and side-effects upon teaching and learning. • Selection of assessment methods should depend on the skills to be assessed, and a blend of methods is desirable. • Swanson DB, Norman GR, Linn RL. Educ. Researcher. 1995;24(5):5-11.

  22. Administrative Issues - Resources • What resources are needed? • Facilities • Staffing - Proctoring • Faculty numbers and hours • Research & Development • Deployment and Feedback • Grading and Reporting • “Using direct observation is not for the faint of heart or wallet.”

  23. Example – Oral Examination • 100 students at 2 campuses • Minimum 13 faculty required • Minimum 10 hours per faculty member • 12 oral exams per year (1st and 2nd years) • 13 x 10 x 12 = 1560 hours • Students must separately pass each set of 6 per year. • >80% agreement on scores (1 – 5 scale) with review of 25% of the exam presentations.

  24. Administrative Issues - Tracking • Are the assessments reliable and valid? • Harder to generate statistical reports from subjective ratings • Do the outcomes impact future performance? • What are the measures? • Evaluations • Students • Faculty (peer)

  25. Group Discussion Tasks • Purpose of performance-based assessments • Factors affecting their use, including available resources • Personal experiences and insights regarding their use • Administrative oversight and outcome

More Related