1 / 23

The Royal College of Pathologists – the overall assessment system

The Royal College of Pathologists – the overall assessment system. Dr. Trevor Gray Director of Examinations and Assessment. The Assessment System. Overview of PMETB and assessment strategy How WBA fits with other assessments Advantages and disadvantages of WBA compared with other assessments

kenny
Download Presentation

The Royal College of Pathologists – the overall assessment system

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. The Royal College of Pathologists – the overall assessment system Dr. Trevor Gray Director of Examinations and Assessment

  2. The Assessment System • Overview of PMETB and assessment strategy • How WBA fits with other assessments • Advantages and disadvantages of WBA compared with other assessments • Local organisation of WBA and how it is administered by the College • Future prospects

  3. Overview • PMETB (and the MMC process) has laid down significant challenges to medical postgraduate training (elegantly dissected in the Tooke report) • PMETB principle is that training and assessment should be much more integrated – an assessment system • Workplace-based assessment is an essential element of the assessment of trainees undertaking specialist medical training • The specialty curriculum, training, exams and assessments are core College business (in the charter) • The College’s aim is to develop a validated assessment system which is fit for purpose

  4. PMETB strategy on assessment • PMETB principles for assessment 2004 • Nine principles for assessment systems • Now backed up with technical documentation • Principles based on “overarching strategy” • Workplace-based assessment • Examinations of knowledge • Examinations of Clinical Skills (“Workplace-based assessment” PMETB, Jan 2005) • Specialty curriculum and assessment system now approved by PMETB • (i.e. match strategy)

  5. Workplace-based Assessment • Assessment based on performance in the workplace and measured there • Assessed locally by supervisors as well as staff of all grades in contact with the trainee • Assessment occurs during normal work immediately followed by feedback so trainee receives continuous feedback on their routine performance • Results put into trainee portfolio for their annual review and hence important for trainee development and ultimately their employment contract

  6. College Assessment Systemapproved by PMETB • Workplace-based assessment • Year 1 assessment • MRCPath Part 1 (test of knowledge) • MRCPath Part 2 (test of skills and applied knowledge) • Feedback on regular basis (WBA) • Educational supervisor’s report • ARCP (Annual Review of Competency Progression)

  7. The College Assessment system: Miller’s pyramid WBA - MSF Does WBA e.g. Mini-CEX, DOPS Shows MRCPath Part 2 Practical Knows how MRCPath Part 1 Knows

  8. Value of multiple assessments • Key is several modes of assessment – triangulation • A single exam system (MRCPath) does not provide adequate coverage of clinical practice • WBA is also not sufficient in itself (Assessment System Guidance – PMETB)

  9. Examinations and Assessment • Examinations are simply one particular form of assessment • Assessments are: • formative (involve feedback and are intended to guide training) or • summative (assess whether reach given standard) e.g. driving test, MRCPath • All assessments should be “blueprinted” to the curriculum to demonstrate that the most appropriate method is being used

  10. How Workplace-based Assessment fits with MRCPath • Workplace-based assessment depends on assessment by local staff, examinations rely on assessment by those who have had no contact with the trainee • Examinations provide an independent assessment of knowledge and practical skills to national standards • Workplace-based assessment provides a local insight into candidates’ performance in routine tasks and interaction with staff and patients

  11. How Workplace-based Assessment fits with MRCPath • Workplace-based assessment is a local process governed by deaneries albeit to PMETB standards • Examinations are organised nationally by the College • Comments, complaints and appeals for WBA go to deanery, for exams go to College

  12. Workplace-based Assessment – Advantages • Measurement occurs in more authentic environment than the simulated one of the examination room • Trainees can take assessment in their own time and are not limited to exam timetable • The assessment tools are set up to provide extensive feedback to the candidate (formative) • Use of computerised forms makes them easy to record

  13. Workplace-based Assessment – Disadvantages • To avoid time-consuming “busy” assessment forms, the assessments are often simplified if not simplistic • A robust rating can only occur if there are a number of independent observers – which can be difficult in small specialties • They have only been validated in a number of clinical settings but not in all pathology settings • Computerised forms can over-simplify complex judgements and may not reward excellence appropriately

  14. Year 1 Assessment (OSPE) • Developed to provide evidence of progression in run-through training in the absence of further recruitment and selection • Designed to assess progress in first year and be taken after 7 months training • It is a test to “indicate the capability and potential of a trainee through a test of applied knowledge and skills relevant to the year 1 curriculum”

  15. Training and Assessment Plan ARCP F1 F2 ST1 ST2 ST3 ST4 ST5 Foundation Stage A Stage B Stage D Stage C WBA Workplace-based assessment MSF Y1A MRCPath CCT Part 1 Part 2

  16. The training environment • Workplace-based assessment utilises the skills of a wide variety of professional groups as assessors: medical, nursing, BMS, clinical scientist, ancillary staff • The College cannot train or quality assure all assessors • Training centres are recommended to develop a training environment where those with the skills needed for assessment are recognised and training is cascaded from those attending events such as this

  17. Workplace-based Assessments and the College • WBA is part of assessment system for Specialist Trainees from 1st August 2007 • Assessments need to be undertaken from early in training and recorded for ARCP • 2007 – Launch of tools and paper-based assessment forms • College is researching web-based systems similar to the ones used in foundation

  18. Workplace-based Assessments and the College • April 2008 – pilot web-based system • By August 2008 – launch of web-based system • Use data generated by all assessments including MSF and OSPE for evaluation of assessment strategy • Ensure compliance with MMC Gold Guide and planned ARCP

  19. Vision for College Assessment system • College will have integrated specialty curriculum and assessment system endorsed by PMETB • College will have fully validated workplace-based assessment tools delivering feedback on progress to trainees throughout course • Valid and reliable tests will underpin MRCPath examination – Part 1 test of knowledge, Part 2 test of applied knowledge and skills

  20. Factors affecting future assessment strategy • Selection into specialty may not be an annual process in future compromising ability to deliver year 1 assessment (except maybe in Histopathology) • Multiple recruitment may require more flexibility in arrangements for WBA • Selection into specialty may involve a national set of tests (Tooke and Academy) displacing Year 1 assessment

  21. Factors affecting future assessment strategy • Workplace-based assessment is likely to continue but its use must be evidence-based • Workplace-based assessment is not cast in stone – work in progress and subject to improvement following evaluation • Place of individual components of assessment in MRCPath, Year 1 assessment and workplace-based assessment may change

  22. Summary • The College’s education and training, assessment and examinations departments have worked hard to produce curricula and assessments acceptable to PMETB • This training day is the first for workplace-based assessment and is intended to assist training centres to develop assessment locally • Workplace-based assessment will be subject to a full programme of validation and optimisation to ensure fitness for purpose • The College is looking to source a web-based system to support workplace-based assessment

  23. Glossary ARCP Annual Review of Competence Progression CbD Case-based Discussion DOPS Direct observation of practical skills ECE Evaluation of clinical events MSF Multi-source feedback (ePATH-SPRAT) Mini-CEX Mini clinical evaluation exercise OSPE Objective structured practical examination PMETB Postgraduate medical education and training board WBA Workplace-based assessment

More Related