Changes in selection to the Foundation Programme

1. Changes in selection to the Foundation Programme Professor Paul O’Neill Chair, ISFP Project Group Member UKFPO Rules Group Lead for Research and Evaluation Selection

2. Plan for Talk Background to change – robust & numbers Evidence around selection SJTs • Evidence • Piloting EPM Algorithm Academic FP

3. Selection (appointment) Best prediction of the right person to do the best job Should be done fairly

4. PON experience: House jobs (Foundation) CVs. application forms and letters References & ‘putting a word in’ Interview panels – numbers, questions Presentations Occupational pyschologists Personality testing

5. Now: Mostly national Application - anonymous, standardised and assessed against criteria Use of standardised tests (e.g. SJT) Selection centres – standardised, multimodal, some competency testing Interview panels more & more standardised

6. Evolution 2009 DH commissioned a review of selection methods. The Improving Selection to the Foundation Programme project was set up and overseen by the Medical Schools Council 2010/2011 New selection methods were piloted successfully FP 2012 Full-scale Parallel Recruitment Exercise (PRE) FP 2013 New selection methods implemented

7. A collaborative venture between Scottish Foundation Board Foundation School Directors Educational Supervisors

8. Selection in Medicine Undergraduate Single School UKCAT Postgraduate Foundation Postgraduate – Birmingham Review Substantive – Consultant & GP

9. Literature Reviews 3 commissioned Warwick Newcastle Durham Overlapping Specify methods used Identify gaps and where judgement will be needed

10. Durham: McLachlan & Turnbull 236 References Behaviour predictive of future behaviour Conscientiousness significant component of concerns

11. Warwick: Thistlethwaite et al 197 papers 1990-2009, medical school and residency. Most from USA Mostly looked at face validity and reliability NOT predictive validity Lack of consensus Non-medical literature – focus on employability skills (job analysis)

12. Newcastle: Illing et al 202 references Foundation programme – narrow range of applicants, very few cannot do the job Selection methods must satisfy their stakeholders (employers, students etc) Cognitive tests are moderately predictive of later cognitive tests Non-cognitive elements need to be considered to ensure that a doctor is able to perform well Assessment centres allow for a range of methods to be used

13. Selection Methods Combination of: Cognitive ability (academic) Non-cognitive Academic range very narrow High ‘justice’ Incorporate measures of big 5 personality applied to medicine Not IQ or general aptitude tests

14. Selection Methods Big 5 Model of Personality is predictive of job and academic performance Extraversion (outgoing, sociable, impulsive) Emotional stability (calm, relaxed) Agreeableness (trusting, co-operative, helpful) Conscientiousness (hardworking, dutiful, organised) Openess to experience (artistic, cultured, creative)

15. Assessment Criteria Reliability Validity Consistency Granularity Longevity Will the process pick the ‘best’ applicants across the UK? Will the process be good for many years?

16. Assessment Criteria Will the process support or undermine educational objectives? Educational Impact Applicant Burden How onerous is the process for applicants ? Clinician Time To what extent does the process distract from service delivery?

17. Assessment Criteria Will the process pick minimise cheating or malpractice? Compliance Can the applicants see exactly where the goal posts are? Transparency Fairness (Justice) Is there a level playing field? Public Opinion Person in the street thinks that it is fair

18. Summary of Results Lower 5-year cost Better score

19. Selection to FP 2013 Situational Judgement Test (SJT) • SJTs will replace the ‘white space’ application form questions • This is an invigilated, machine markable test in exam conditions • The SJT will consist of 70 questions in 2 hours 20 mins Educational Performance Measure (EPM) • The EPM will replace the academic quartile scores. The EPM score is comprised of three elements: • medical school performance in deciles • additional degrees • academic achievements The EPM and SJT will each be worth 50 points from a 100 point application score

20. The case for change Concern SJT and EPM

21. The case for change Concern SJT and EPM

22. Situational Judgement Test

23. What is a Situational Judgement Test? SJTs are: • a test of aptitude • designed to assess the professional attributes expected of a Foundation doctor • based on a detailed job analysis of an FY1 doctor SJT questions assess your judgement by presenting you with challenging situations you are likely encounter at work during the first year of an integrated Foundation Programme

24. Example SJT Questions There are two question formats: • Rank the five responses in the most appropriate order • Choose the three most appropriate responses from eight You should answer what you ‘should’ do in the scenario described, not what you ‘would’ do

25. Example Question 1 – ranking Mr Reese has end-stage respiratory failure and needs continuous oxygen therapy. While you are taking an arterial blood gas sample, he confides in you that he knows he is dying and he really wants to die at home. He has not told anyone else about this as he thinks it will upset his family, and the nursing staff who are looking after him so well. Rank in order the following actions in response to this situation (1= most appropriate; 5= least appropriate). Tell Mr Reese that whilst he is on oxygen therapy he will need to stay in hospital Reassure Mr Reese that the team will take account of his wishes Discuss his case with the multi-disciplinary team* Discuss with Mr Reese's family his wish to die at home Discuss Mr Reese's home circumstances with his General Practitioner

26. Answer to Question 1 B. Reassure Mr Reese that the team will take account of his wishes C. Discuss his case with the multi-disciplinary team* E. Discuss Mr Reese's home circumstances with his General Practitioner D. Discuss with Mr Reese's family his wish to die at home A. Tell Mr Reese that whilst he is on oxygen therapy he will need to stay in hospital This question is focusing on effective communication with patients. Ensuring that patients’ informed wishes are met in relation to their care is central to your approach to patient care and this needs to be communicated to the patient in a reassuring manner even in situations relating to end of life care (B). These wishes should have been sought when addressing the management plan for Mr Reese and once identified the multidisciplinary team needs to be made aware of them in order to ensure that as far as possible Mr Reese’s views in relation to his end of life care are implemented (C). The management of Mr Reese will require the active involvement of his GP and communication with the GP is therefore of importance (E). Any decision to discuss Mr Reese’s wishes in relation to his end of life care with his family can only be made with the full agreement of Mr Reese (D). It would not be appropriate to give the patient inaccurate information in order to engineer a different medical pathway (A).

27. SJTs (Literature Review 77 papers) Management, university, police, engineers Large scale selection – short-listing Construct validity not clearly identified • Single construct (e.g. ‘practical intelligence’) • Can be designed to measure differing constructs • Predictive validity will depend on what criterion is targeted • SJT designed to test interpersonal skills will more likely predict inter-personal orientated performance

28. SJTs Used nationally to select GP registrars and other ‘high stakes’ occupations significant validity in predicting job performance incremental validity over methods such as ability tests and personality questionnaires typically relate to general experience and ability, rather than job-specific knowledge or experience tend to show smaller differences between candidate groups (e.g. based on race) than cognitive ability tests

29. Job analysis of FY1 doctor Commitment to professionalism Coping with pressure Effective communication Learning and professional development Organisation and planning Patient focus Problem solving and decision-making Self-awareness and insight Working effectively as part of a team

30. Job analysis of FY1 doctor

31. Item development

33. Example Question 2 – multiple choice You have been prescribed codeine for persistent back pain which has become worse in the last few weeks. You have noticed that during shifts you are becoming increasingly tired, finding it difficult to concentrate and your performance, as a result, has been less effective. Choose the THREE most appropriate actions to take in this situation Ask a colleague to assist with your workload until you finish your codeine prescription Make an effort to increase the number of breaks during your next shift Stop taking the codeine immediately Make an appointment to see your General Practitioner Seek advice from a specialist consultant about your back pain Arrange to speak with your specialty trainee (registrar)* before your next shift and make them aware of your situation Seek advice from your clinical supervisor* regarding further support Consider taking some annual leave

34. Answer to Question 2 D. Make an appointment to see your General Practitioner F. Arrange to speak with your specialty trainee (registrar) before your next shift and make them aware of the situation G. Seek advice from your clinical supervisor regarding further support This question looks at how you demonstrate commitment to professionalism and self-awareness. The essential problem is that as an FY1 doctor the level of your clinical performance is dropping. This constitutes a risk to the patients you are caring for and will impose a greater workload on your colleagues. In this circumstance you should inform and seek the advice of the senior clinician responsible for your work (G) and alert your colleagues (F). This matter is most likely to be related to your prescribed medicine and you should therefore consult with your GP (D) rather than any other specialist (E). It is not your place to re-allocate workload (A). Increasing the number of breaks is unlikely to improve a situation that is likely to be due to an adverse effect of a drug (B). You should not make any unilateral decisions about your medical treatment (C) and should seek the advice of others (D). You should not be seeking to use your annual leave (H) to compensate for a medical problem.

35. Scoring of the SJT Scoring key, determined through: Consensus at item review stage (item writers, SMEs) Expert judgement in concordance panel review Review and analysis of the pilot data Scoring not “all or nothing”, but based on how close to scoring key

36. Scoring of the SJT – ranking • Up to 20 marks available • Up to 4 marks available for each response (points for “near misses”) • No negative marking

37. Scoring of the SJT – multiple choice • Points for each correct answer • No negative marking • 4 points for each correct answer • Up to 12 points per item

38. Parallel Recruitment Exercise (PRE) New selection methods trialled alongside the normal selection methods during 2012 FP applicationround Aims: logistics, awareness, pilot new SJT content All 31 medical schools involved SJT – 1 hr, 30 questions EPM – each medical school consulted study body on ‘basket of assessments’ to be used

39. Parallel Recruitment Exercise (PRE) 90+% of FP applicants participated in the SJT Valuable learning experience ahead of live implementation Feedback to inform live implementation: • Applicants • Medical schools Was the final step in ensuring the selection methods can be consistently and robustly applied for implementation for FP2013

40. PRE - SJT • 30 item, one hour SJT • 6,842medical students took part in the PRE • Participants included: • Final year medical students • Students who had been pre-allocated to the Defence Deanery • Students who had chosen to take a year out post-graduate • International students returning overseas after graduation • 30 medical schools (plus two centres for Eligibility Office applicants) delivered the SJT in 72 venues • Psychometric analysis shows that a 60 item SJT is a reliable measurement Sheffield SJT pilot

41. Descriptive Statistics • Internal Reliability: • Adjusted for a 60 - item test that included only robust items (such as would be used in an operational paper), all papers had an estimated reliability of α = 0.80 or above (α = 0.80 to α = 0.87) • Demonstrates that the SJT is a reliable test in this context with items testing different things

42. Histograms showing the distribution of scores for Paper One (left) and Paper Two (right)

43. Descriptive Statistics Mean: • Overall Mean = 79.5%; Range from 78.0% to 80.6%. So not too easy – differentiating appropriately Standard deviations: • Mean SD = 18.6; Range from 17.3 to 20.0. Distributions: • 305 to 468 (out of a maximum of 512) – as expected given length of paper • Appears to be slightly negatively skewed, (more people towards top end) although results do show a close to normal distribution

44. Item Facility (difficulty) • Ranked Items • Maximum score 20 • Score 18 = ‘very easy’; • Score 11.6 = ‘very hard’ • Mean facility similar across all papers (approx 16) • Range of facility values differed across papers • SD range similar for all papers, except for Paper 1 where one item had a very high SD • Multiple Choice Items • Maximum score 12 • Score 10.8 = ‘very easy’; • Score 3.6 = ‘very hard’ • Mean facility across papers ranged from • 7.9-9.0 • Range of facility values differed across papers • SD range was similar for all papers

45. Student views The content of the assessment seemed relevant to the Foundation Programme The scenario content seemed appropriate for my training level

46. Educational Performance Measure

47. Why change to EPM? A clear framework with agreed principles used to calculate the EPM, ensuring that it is fair, transparent and consistent across the schools of the UK Splitting cohorts into deciles rather than quartiles provides a wider spread of scores, which makes it easier to differentiate between applicants, and will be more fair for applicants at the margins It makes more sense for all the academic components of the application to be one part of the application

48. How is the EPM calculated? Score produced by applicant’s medical school to reflect achievement and performance compared to rest of cohort EPM = 3 parts (maximum 50 points): • Medical school performance in deciles (34 – 43 points) E.g. Top 10% = 43; Top 20% = 42; etc • Additional degrees (max 5 points) • Educational achievements (presentations, prizes and publications (max 2 points) Schools have consulted with students about how the decile points for the EPM will be calculated.

49. PRE - EPM 27 of 30 medical school initiated a new consultation and review of framework • (3 schools consult annually and framework aligns) • Other schools do consult annually – but undertook new consultation with students for the PRE Majority of schools pleased with student engagement, especially amongst later years • Benefit of raising awareness with students & staff • Students felt sense of ownership

50. Consultation on frameworks