Formative assessment of classroom competencies (FACCs) for postgraduate medical trainees

1. Formative assessment of classroom competencies (FACCs) for postgraduate medical trainees Professor Chris Gray on behalf of Department of Medical Education, City Hospitals Sunderland

2. Competence: ability of a professional to combine knowledge, skills and behaviour to perform a specific role Definition McClelland, D. C. (1973). American Psychologist, 28, 1-14.

3. The legal standard • ‘The ordinary reasonable professional exercising skills appropriate to the role and profession’ (Bolam test) • No concessions for inexperience • Competence assumed • But…competence acquired Bolam v Friern Hospital Management Cttee 1957. 1. WLR 583

4. Problem • Assumption of competence in core clinical skills • Core skills defined by GMC • Variation between graduates within and between institutions • Trust request to provide evidence for all trainees

5. Aims • To determine the feasibility of undertaking a routine assessment of core competencies as part of the routine induction process for postgraduate medical trainees. • To provide an overview of GMC defined core competencies in medical trainees. • To develop a classroom based assessment of core clinical competencies as a tool for assessing and developing clinical skills.

6. MethodsUndergraduate experience in core skills Questionnaire to all FY1 doctors (June/July 07) N=18 • Self reported frequency core skills undertaken prior to graduation (patients/mannequins). • Self reported confidence in skills prior to FY1 and end FY1. 0 no confidence - 10 no concerns.

7. Self reported frequencies (median IQR) of undergraduate experience in the core clinical skills (18 FY1 doctors)

8. FY1 doctors’ self confidence ratings for core competencies prior to (retrospective) and final 4 weeks (current) of F1 training. Likert scale 1= no confidence and 10 = confident, no concerns. N=18.

9. Development of FACCs • Consensus panel , 22 GMC core skills • Tasks undertaken as routine or emergency, unsupervised, generic not speciality • 8 station OSCE (7 observed, 1 written) • 7mins stations, max for circuit 1 hour • Venesection, nasogastric tube insertion, male catheterisation, cannula insertion, prescribing, intermediate life support (including defibrillation), ECG recording, arterial blood gas sampling

10. Assessment of performance Within each station: • Pre defined performance domains: infection control techniques, ability to adequately select, prepare and safely dispose of equipment, and the ability to sequence and complete the procedure • For each domain, candidates were scored as having completed the tasks correctly or not. • Pre-specified ‘critical’ domains those tasks which if not completed correctly or omitted would expose the patient, staff or the host organisation to clinical risk

11. Statistics • For each station a percentage score was derived for each candidate’s performance. (overview: all candidates, medians and distributions) • Within each station: Frequency of critical errors/omissions achieved by all candidates recorded

12. Feasibility • 106 doctors (87.6% of 121 overall) completed assessment (compulsory!) • 91 [85.5%] doctors completed in first three days as part of routine induction process. • 14 circuits during the first 14 days of employment in the trust.

13. Distribution of FACCs scores (%) for each station (Median and IQR 1st and 3rd quartiles (all candidates N=106)

14. % Number of candidates failing criticals at each station (N=106)

15. Arterial blood gas 57% Failed to check syringe expiry 17% failed express air/heparin pre 21% failed express air post Male catheterisation 7.5% failed reposition foreskin Resuscitation 72% one or more errors/omissions 33% Failed opening airway 21% failed remove oxygen 7% omitted stand clear warning IV cannulation 38.7% failed to confirm content of flush Criticals

16. Performance scores (%) for each clinical station (median, IQR) by training grade. *p=0.012 **p<0.001 (KW Anova)

17. Interpretation • Pre FY confidence in practical skills low (exc. venesection, ECG) • Pre FY experience in practical skills limited • Post FY1 confidence high (exc resuscitation) • FACCs feasible, majority (88%) assessed during induction (feasibility) • Overall median station scores >90% suggest competency (overview) • Critical errors analysis highlight problem areas (development)

18. Take away message! Get FY1 to do…. your ABGs and NG tubes!!

19. Limitations • Appropriateness of classroom based assessment of clinical skills • Circuit construct – relevance across all specialities • OSCE familiarity (training grade effect, variation) • Definition of competence, hierarchy of criticals • Integration of assessment performance into trainee’s personal development plan • Managing the criticals / poor performer

20. Acknowledgements • Undergraduate and postgraduate teams • Examiners • Participants (not voluntary but in spirit of collaboration)

21. Prescribing

22. Assessment • What is competence ? • Assessment against ‘standard’ not definable • Diversity of grades and specialities • Formative (strengths/weaknesses) • No pass fail threshold • Identify critical errors omissions • Direct remedial learning and development

23. ‘Do you feel your undergraduate training adequately prepared you for undertaking the GMC core practical procedures as listed?’ N=18 FY1 doctors 0= totally unprepared, 10 = totally prepared.