1 / 48

Performance-Based Assessment In Clinical Training

Performance-Based Assessment In Clinical Training. Susanto Nugroho Department of Child Health Faculty of Medicine University of Brawijaya Dr. Saiful Anwar General Hospital. BACKGROUND. Competence: what an individual is able to do in clinical practice

woods
Download Presentation

Performance-Based Assessment In Clinical Training

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Performance-Based Assessment In Clinical Training Susanto Nugroho Department of Child Health Faculty of Medicine University of Brawijaya Dr. Saiful Anwar General Hospital

  2. BACKGROUND • Competence: what an individual is able to do in clinical practice • Performance: what an individual actually does in clinical practice • Dimensions of medical competence including: • The scientific knowledge base • Professional practice elements: history taking, communication, physical examination, procedural and management skills, problem solving ability, relationships with colleagues & ethical behaviour Faculty of Medicine University of Brawijaya

  3. Learning Domains • Cognitive:intellectual skills (facts, procedures, concepts, rules, principles) • Psychomotor: complex perceptual – motor skills • Affective: emotional control, stress –coping, attitudinal predisposition to respond Faculty of Medicine University of Brawijaya

  4. Does Shows how Knows how Knows Miller’s Model of Clinical Competence Performance-based testing: 360 degree, Portfolio, Mini-CEX Competency-based testing: OSCE, Short case, long case Behaviour Clinical contact-based testing: MEQ, EMQ, PMPs, SAQ, SEQ Factual testing: MCQ, Essay, Oral Cognition Miller’s pyramid: a framework for assessing clinical competence Faculty of Medicine University of Brawijaya

  5. Learning Stages forComplex Cognitive Skills Novice: rigid adherence to rules; poor situational perception; no experience Advanced beginner: uses guidelines for action, still limited situational perception Competent: better view of big picture; plans ahead; but still procedure driven Proficient: holistic view and able to extract most important elements of situation; uses maxims (generalizable) Expert: no longer relies on rules, guidelines or maxims; intuitive grasp and recognition of patterns for ease of decision action Dryfus& Dryfus (1980) Faculty of Medicine University of Brawijaya

  6. Learning Stages For Complex Perceptual Motor Learning • Cognitive: knowledge based (thinking, verbal) understanding of task and strategies • Associative: procedural (rule-based) understanding of steps or task sequence • Autonomous: (unconscious) automatic execution of skilled performance with high accuracy and precision Fittsand Posner (1967) Faculty of Medicine University of Brawijaya

  7. Performance assessment: the assessment of skills and behaviour, both in academic & workplace setting Performance assessment: building in a degree of complexity to Miller’s pyramid, recognising both the development of performance expertise, and the need for skills and behaviour maintenance through deliberate practice. Through the skills complexity triangle, some of the contextual factors which impact on performance measures, both individual and systems related, including taking experience into account. Faculty of Medicine University of Brawijaya

  8. Miller’s Model of Performance Assessment Deliberate practice Integrated team performance Does Ericsson Integrated skills Shows how Integrative phase Task training Skills complexity triangle Knows how Cognitive phase Knows Miller Fitts and Posner Faculty of Medicine University of Brawijaya

  9. ASSESSMENT • Integrated assessment  3 components: • Applied knowledge test (AKT) • Clinical skill assessment (CSA) • Workplace-based assessment (WPBA) • Each of these components is assessed independently and will assess different skills,but together the 3 components will cover the curriculum. Faculty of Medicine University of Brawijaya

  10. Assessment tools for clinical competence OSCE Objective structure clinical examination Objective structured long case examination record OSLER Objective structured assessment of technical skills OSATS These are undertaken outside the “real” clinical environment but have many aspects of realism of workplace incorporate into them and are assessed at the “shows how” level of Miller’s pyramid. Faculty of Medicine University of Brawijaya

  11. Assessment tools for clinical performance Mini-CEX Mini clinical evaluation exercise DOPS Direct observation procedural skills CbD Case-based discussion Mini-PAT Mini-peer assessment technique COT Consultation Observation Tool PSQ Patient Satisfaction Questionnaire Faculty of Medicine University of Brawijaya

  12. WORKPLACE-BASED ASSESSMENT (WPBA) • Knowledge is only one aspect of being a doctor  how about competencies & performance in the professional practice? • Competencies can be assessed through relevant clinical tasks  were often simulated & occasionally distant from the workplace • Assessment of competence – performance misses routine behavior • WPBA tools assess at the “does” level of Miller’s pyramid Faculty of Medicine University of Brawijaya

  13. Definition of WPBA • A form of authentic assessment testing of performance in the real environment facing doctors in their everyday clinical practice • Boursicot K, Etherridge L, et al. 2010 • Assessment of day to day working practices undertaken in the working environment • PMETB Workplace – Based Assessment January 2005 • The evaluation of a doctor’s performace progress over time in professional practice areas best tested in the workplace • http://www.1scp.ac.uk/Assessment/WBA/Intro.aspx Faculty of Medicine University of Brawijaya

  14. Reasons for using WPBA • Structured & continuous: not only to form judgement on competence • Repeated assessment  assessor has the opportunity for gathering documentary evidence of the student’s actual performace progression • Can identify gaps in practice  allow the assessor & student to mutually plan student development needs • Helps identify difficulties, strength & weakness of student in different areas of practice e.g. technical skills, professional behaviour & teamworking Faculty of Medicine University of Brawijaya

  15. Reasons for using WPBA..... • Best assessment practice in medical education rather than traditional assessment  allow the assessment of professional behavior aspects • Allows the assessment to get as close as possible to the real situations in which doctors work • Based on contemporary educational design • Drive learning in important areas of competency Faculty of Medicine University of Brawijaya

  16. Classification Scheme for WPBA Method of collecting data Clinical records Basis for judgment Administrative data Outcomes of care Process of care Practice volume Diaries or log Observation Faculty of Medicine University of Brawijaya

  17. WPBA Tools • Mini-CEX: Mini-Clinical Evaluation Exercise • DOPS: Direct Observation of Procedural Skills • CbD: Case-based Discussion • Mini-PAT: Mini-Peer Assessment Tool Faculty of Medicine University of Brawijaya

  18. Mini-CEX: Mini-Clinical Evaluation Exercise

  19. Mini-Clinical Evaluation Exercise • A method to assess clinical skills that doctors most often use in real clinical encounters • Direct observation of the student’s performance in real clinical encounters in the workplace • The student is judged in ≥ 1 of 6 clinical domains and overall clinical care, using 9-10 point rating scale  then the assessor gives feedback • Clinical domains: history taking, physical examination, communication, clinical judgment, professionalism, organization/efficiency Faculty of Medicine University of Brawijaya

  20. Mini-Clinical Evaluation Exercise..... • Is performed on multiple occasions with different patients and different assessors • Encounter takes around 15 minutes • Not only helps identify strength and weakness  immediate feedback & helps improve skills • Reliability: • There is good evidence of the reliability • Inter-rater variations in marking  : more assessors rating fewer encounters • Good reliability: 10-14 encounters Faculty of Medicine University of Brawijaya

  21. Mini-Clinical Evaluation Exercise..... • Validity: • Able to measure clinical performance over a wide range of clinical complexity & level of training  to discriminate between junior & senior student • Educational impact: • Can monitor progress & identify educational needs • Appropriate feedback allows student to correct the weakness & to mature professionally • Reassures student of satisfactory performance & increases the interaction with senior doctors Faculty of Medicine University of Brawijaya

  22. Mini-CEX: Step by Step Student immediately discuss the clinical encounter with assessor to get feedback and to plan the next clinical encounter The clinical encounter & process with patient should be done by the student and assessor will assess ≥ 1 of 6 clinical domains and overall performance Assessor and student discuss the Mini-CEX process before do the real clinical encounter with patient Faculty of Medicine University of Brawijaya

  23. Mini-CEX form

  24. Mini-CEX form

  25. Mini-CEX form

  26. DOPS: Direct Observation of Procedural Skill

  27. Direct Observation of Procedural Skills • A method of assessment focused on procedural skills • It requires assessor to: • Directly observe the student undertaking the procedure • Make judgement about specific components of the procedure • Grade the student’s performance • Strengths: fair,direct observation, allow global evaluation, direct feedback, practical and easy to use • Weakness: it may be necessary to have an expert observer or assessor Faculty of Medicine University of Brawijaya

  28. Direct Observation of Procedural Skills..... • Reliability: • A reliable measure: repeated on several occasions • Poor reliability & unknown validity: subjective assessment of comptences done at the end of a rotation • Specifically focuses on procedural skills and pre/post procedure counselling carried out on actual patients • Validity: • Validity indicator: score increase between the first and next level of training Faculty of Medicine University of Brawijaya

  29. Direct Observation of Procedural Skills..... • Acceptability: • Acceptable: needs only 15 minutes/procedure (including observation & feedback) & 4-6 times/year  a practical method • Educational impact: • Feedback gives it the potential for high educational value Faculty of Medicine University of Brawijaya

  30. DOPS form

  31. DOPS form

  32. CbD: Case-based Discussion

  33. Case-based Discussion • A variation ofChart Stimulated Recall (CSR) • Essentially case reviews: discuss particular aspects of a case  involved to explore underlying reasoning, ethical issues & decision making • Can be used in a variety of clinical settings e.g. clinics, wards, ICU, etc • Assesses 7 clinical domains: medical record keeping, clinical assessment, investigations and refferals, treatment, follow-up and future planning, professionalism, and overall clinical judgement Faculty of Medicine University of Brawijaya

  34. Case-based Discussion..... Reliability: a good reliability for CSR Validity: able to differentiate between doctor in good & poor performing  correlates with other forms of assessment Acceptability: assessor bias reduces the validity  training of all assessors  high cost? Educational impact: feedback on clinical reasoning and decision making  to be valuable in helping students progress Faculty of Medicine University of Brawijaya

  35. CbD form

  36. CbD form

  37. Mini-PAT: Mini-Peer Assessment Tool

  38. Mini-Peer Assessment Tool • Amodified version of Sheffield Peer Review Assessment Tool (SPRAT) reliable & feasible • A method to collate the judgements of peers  systematic & broad sampling across different individuals (legitimate to make judgements, e.g. colleagues, nurses, residents, administrative staff, medical students and also patients) • The objective and systematic collection and feedback of performance data  to assess behaviours and attitudes, e.g. communication, leadership, team working, punctuality and reliability Faculty of Medicine University of Brawijaya

  39. Mini-Peer Assessment Tool..... • Astructured quesionnaire consist of 15 questions on the individual student’s performance • Needs 8-12 assessors and the time taken to assess is approx. 10-30 minutes • Reliability: • Inter-item correlation is good, but correlation between assessment is lower • Inter-rater variance: consultants tending to give a lower score except they have known the student Faculty of Medicine University of Brawijaya

  40. Mini-Peer Assessment Tool..... • Validity: • Strong concurrent between Mini-PAT, Mini-CEX & CbD • Evidence: senior doctors achieving a small, but statistically significantly higher overall mean score compared to more junior doctors • Acceptability: • A high response rate of 67% was achieve • Main advantage: anonymous  assessors to be more honest about their reviews Faculty of Medicine University of Brawijaya

  41. Mini-Peer Assessment Tool..... • Main critism: feedback is often delayed and difficult to attribute unsatisfactory performance to specific clinical placements due to anonymous feedback • Educational impact: • Has been used as a formative assessment  provide an honest and balanced view of student • Allows the student to reflect on the feedback  to improve clinical performance • Disadvantage: risk of victimization and potentially damaging harsh feedback Faculty of Medicine University of Brawijaya

  42. Mini-PAT form

  43. Mini-PAT form

  44. SUMMARY • Dimensions of medical competence including the scientific knowledge base & professional practice elements • There were assessment tools for assess clinical competence and clinical performance • WPBA tools assess at the “does” level of Miller’s pyramid  clinical performance • WPBA allows assessors & student to get as close as possible to the real situations in workplace • WPBA tools that are commonest used: Mini-CEX, DOPS, CbD, and Mini-PAT Faculty of Medicine University of Brawijaya

  45. THANK YOU

  46. CURRICULUM VITAE • Susanto Nugroho, MD., Paed (C) • PLACE/DATE OF BIRTH: Semarang, 4 March 1968 • EDUCATION & TRAINING: • Medical Doctor – University of Diponegoro 1993 • Pediatrician – University of Diponegoro 2003 • Fellowship in Pediatric Hematology Oncology Consultant, University of Airlangga 2006-2007 • ONDT in St. Georges University of London 2011 • INSTITUTION: • Department of Child Health Faculty of Medicine • University of Brawijaya/Dr. Saiful Anwar Hospital

More Related