THE OSCE 30 YEARS ON James Ware Kuwait

1. THE OSCE 30 YEARS ON James Ware Kuwait

2. It is a pleasure to acknowledge some of the thoughts and facts freely offered me from: • Ronald Harden, Scotland • Dale Dauphinee, Canada • Cees van der Vleuten, Holland • John Norcini, USA • David Newble, Australia • Peter Devitt, Australia • Neil Paget, Australia • Sam Leinster, England

3. The organisation of this brief history of the OSCE milestones is like this: 1964-1974 Pre-OSCE era, simulated patients introduced and traditional methods questioned 1975-1984 Early OSCE days 1984-1992 Research and refinements improve our understanding and the first conferences 1993-2006 National bodies introduce OSCE to test competence to practice

4. 1964 HS Barrows, a neurologist, with S Abrahamson, an anatomist, introduced the use of programmed patients as a teaching method and then later in assessments at the University of Southern California, reported in 1968 Barrows HS, Abrahamson S. The programmed patient: a technique for appraising student performance in clinical neurology. J Med Educ. 1964, 39:802-805

5. 1964-68 Barrows and Abrahamson introduced the use of actors to play the roles of patients – calling them programmed patients, while today we would probable name simulated or standardised patients. Barrows HS, Abrahamson S. The programmed patient: a technique for appraising student performance in clinical neurology. J Med Educ. 1964, 39:802-805

6. 1968 John Evans, the first dean at McMaster, Canada, Enticed Barrows away and what was accomplished by this move was the creation of the first PBL school. He was later to recruit Tamblyn, who was another OSCE innovator.

7. 1970s sometime An anecdote has it that the Irish Constabulary introduced role play for their objective structure practical examination sometime in the 1970s, but we shall hear of this later. The source of their simulated rogues and citizens was not given!

8. 1968 After graduating at Glasgow University Ronald Harden practiced as a general physician with a special interest in endocrinology, his interest in medical education began already while in Glasgow

9. 1969 Ron Harden started creating the beginnings of the OSCE already in Glasgow and was carrying out research to investigate areas which were to raise considerable interest much later. They reported the variation in rating the clinical competence of candidates using direct observation and videos of candidates. The wide variation (huge in the case of the videos) in determining the fate of these candidates was the challenge they clearly identified The original OSCEs were different from what we do today and yet in several aspects what was done will resonant with some, such as feedback to the student. Harden R et al. Two systems of marking objective examination questions. Lancet 1969, 1:40-42 Harden R et al. Examination of clinical examiners. Lancet 1969, 1:37-40

10. 1970 Ron harden moves to Ninewells Hospital and becomes Head of the Division of Clinical Medical Education, University of Dundee. Later he was to set up the highly successful Centre of Medical Education, Dundee. 1975 The first description of the OSCE Harden R McG et al. Assessment of clinical competence using objective structured examination. BMJ, 1975, 1:447-451

11. Inspected and scored by an examiner History from patient* * These were real patients MCQ*** ** were these OSPE stations? Inspection of slide** *** MCQs or open ended questions MCQ*** Inspected and scored by an examiner Neurological exam* MCQ*** Inspection of an ECG** Harden R McG et al. Assessment of clinical competence using objective structured examination. BMJ, 1975, 1:447-451

12. STATION 1 STUDENT’S NAME INSTRUCTIONS TO THE EXAMINERS Yes No The candidate Felt the radial pulse Counted the rate with a watch Elevated the limb to detect a collapsing pulse Located the 2nd R space properly Auscultated the neck Auscultated down the left sternal border Sat the patient up to auscultate Auscultated in full inspiration Used diaphragm for auscultation Check lists versus qualified marking (global ratings) Initially (Glasgow) check lists were used to rate the performance, later to be modified to allow expert judgment. Harden R McG et al. Assessment of clinical competence using objective structured examination. BMJ, 1975, 1:447-451

13. 1979 In January 1979, 30 years ago, Ron Harden publishes the seminal paper describing the OSCE, he also introduced the idea of the OSPE. After 10 years of refinement Harden and Gleeson (his research fellow) are able to describe the OSCE much as we know it today. Others had begun to use the OSCE and would publish their findings Many still remember the classical long case, unobserved, as much a test of knowledge as one of clinical competence. But interestingly what goes around . . . . . . Harden RM, Gleeson FA. Assessment of Clinical Competence using an Objective Structured Clinical Examination (OSCE). ASME Medical Education Booklet No, 8. Med Educ, 1979, 13:41-54

14. 1979 Professor Alfred Cuschieri, a distinguished hepato-biliary surgeon, introduces the OSCE in the Final Surgery Examinations in Dundee and reports what happened. MCQs were used to check the knowledge base of clinical competence and check lists for Observer-Examiners Cuschieri A et al. A new approach to a final examination in surgery, use of the objective structured clinical examination. Ann R Coll Surg Engl, 1979, 5:400-4005

15. 1980s About this time Ian Hart spent a sabbatical in Dundee with Ron Harden resulting in two landmarks, the introduction of the real OSCE to North America and also the beginnings of the Ottawa Conferences on the Assessment of Clinical Competence

16. 1980 Tamblyn assisted the first nursing OSCE in Canada, since which many health sciences have used this examination format. She was later to be heavily involved in the training of simulated patients and supporting the introduction of the OSCE in the MCCQE

17. 1984 Sam Leinster ran several OSCEs in Cardiff during the 1970s before coming to Liverpool where he ran the first OSCE in the Department of Surgery there, James Ware was one of his assistants. It certainly was a learning experience but took me six more years to do my next one under the guidance of Ron Harden in 1990!

18. 1984 David Newble was one of several who attended the first Cambridge Conference and left with renewed enthusiasm to set about investigating and testing new methods of measuring clinical competence, among them the OSCE. He was to provide David Swanson with much data used to analyse the psychometric properties Newble DI, Swanson DB. Psychometric characteristics of the objective structured clinical examination. J Med Educ. 1988, 22:325-334

19. 1985 The first Ottawa Conference for the Assessment of Clinical Competence, attracting 230 participants from 19 countries who presented sixty-five papers. This was arguably the springboard for which the OSCE took off in many ways, research and the acceptance that this was a realistic way to measure the elusive test of clinical competence – but nothing is perfect. Hart I, Harden RM, Walton HJ. Newer developments in assessing clinical competence, Proceedings of the first Ottawa Conference, 1986, Ottawa, Canada

20. 1989 Dale Dauphinee, who worked for years with the MCC reminds us that the first licensing examination using the OSCE format was held in Canada by the College of Family Physicians. Dauphinee then chaired the Future Directions Committee of the MCC to prepare the OSCE-SP MCCQE Part II examination which eventually was delivered in 1993

21. 1980s David Swanson started an in depth look at the OSCE, initially with David Newble’s data. This was to provide many insights and provocative thoughts about the strengths and weaknesses of the OSCE format. He was also preparing the way for the NBME to finally introduce the OSCE twenty years later. Newble DI, Swanson DB. Psychometric characteristics of the objective structured clinical examination. J Med Educ. 1988, 22:325-334

22. 1980-1990s David Swanson and Newble’s research with the Adelaide data showed that reliability was a problem compared to written components of Finals Examinations. They concluded that if test length was increased to six hours, criteria of reliability would be met. It was also found that careful station selection improved reliability – and sent a strong message to train OSCE examiners. They also suggested combining OSCE scores with other sources, a message sent later by van der Vleuten (OSCE + OSPE + CBKA) Newble DI, Swanson DB. Psychometric characteristics of the objective structured clinical examination. J Med Educ. 1988, 22:325-334

23. Acknowledgement to Cees van der Vleuten for kindly allowing his assembled tabled to be used

24. 1989 Comparisons of different SPs playing the same role may result in significantly different examinee scores but even out if many encounters are included, while they suggested other sources are controllable. Norcini J, Swanson D. Factors influencing testing time requirements for simulation-based measurements: do simulations ever yield reliable scores. Teaching and Learning in Medicine 1989; 1:85-91

25. 1989 Swanson and Norcini concluding their evidence that it is better to have more stations with the available examiners rather than use paired examiners. Norcini J, Swanson D. Factors influencing testing time requirements for simulation-based measurements: do simulations ever yield reliable scores. Teaching and Learning in Medicine 1989; 1:85-91

26. 1989 Cees van der Vleuten with David Swanson and co-workers studied the impact of training and showed that: need and effectiveness of training was least for physician raters, more needed and effective for medical students and most needed but most effective for non-physicians (including SPs). Van der Vleuten CPM et al. Training and experience of medical examiners. Med Educ. 1989, 23:290-296

27. 1990 Cees van der Vleuten and David Swanson published an influential review describing the use of simulated patients. Many would say that SPs are heart and soul of the OSCE, while others would argue that simulation is not the real thing and so the debate goes on. Van der Vleuten CPM, Swanson DB. Assessment of clinical skills with standardised patients: State of the art. Teaching and Learning in Medicine. 1990, 2:58-76

28. 1990 Geoff Norman was also contributing with his data on reliability : reporting that inter-rater reliability is high, 0.80 – 0.99; as was test-retest of the same skill, ICC=0.60-0.88, but across different skills correlations were low, alpha = 0.198. This study was with Baccalaureate nursing students. Roberts J, Norman G. Reliability and learning from the objective structured clinical examination. Med Educ. 1990, 24:219-223

29. 1990 The UK police were not far behind introducing an OSPRE one year later in their national examinations for promotion to sergeant, inspector and detective. Again, I wonder the source of their simulated rogues and citizens.

30. 1993 Dale Dauphinee after tireless work was able to mount the first licensing OSCE 1993 in six centres across Canada. By 1997 the MCCQE Part II exam was delivered in 17 centres with 2000 candidates on a single weekend, but of course this was a team effort, so crucial for the success of any OSCE . . . . . .

31. Some of the collaborators on this project, interestingly propelled by patient complaints both in Canada and the US. This resulted in a long term study: the outcome proved that scores in the communication and clinical decision making components of the OSCE licensing exam predicted complaints to medical regulatory authorities – the paper is worth a read! Tamblyn R et al. Physician scores on a national clinical skills examination as predictors or complaints to medical regulatory authorities. JAMA. 2007, 298:993-1001

32. 1997 The collaborators from MCC describe their innovative Borderline Group Method for standard setting OSCE. It should be added that this method works best when there are many candidates and many standard setters – but many of us use it still in our schools. Dauphinee WD et al. Using the judgments of physician examiners in setting standards for a national multi-center high stakes OSCE. Adv Health Sci Educ Theory and Pract, 1997; 2:201-211

33. 1998 The AMC Held its first IMG OSCE to license foreign medical graduates for practice in Australia in 1998. There were seven centres catering for about 700 candidates with five exam sessions each year. The OSCE replaced a clinical exam with two VIVA stations covering diagnostic and management skills of twenty minutes each.

34. 1998 ECFMG held its first SP-OSCE. Throughout the investigations carried out by the National Boards it had been found that American medical colleges were resistant to the idea that candidates could pass a knowledge test only to be prevented from being licensed by their marks in a skills test using the OSCE. So it was that ECFMG pioneered the way and now NBME CS is held in the same five centres as those set up by Commission.

35. 2003 John Norcini had for some time been influential for standard setting, an issue which has attracted a considerable body of literature for the OSCE. Most are based on either a Comparative Group method (NBME) or the Borderline Group Method (MCC), but regression methods have considerable merit as well, while Angoff is considered too intensive. Norcini JJ. Standard setting on Educational tests. Med Educ 2003; 37:464-469

36. 2000 The General Medical Council introduces the SP-OSCE as part of the licensing process, PLAB, for foreign medical graduates wishing to practice in the UK

37. 2004 Emil Petrusa has been a consistent contributor to what we know about the use of simulated patients and has taken a pragmatic view, which is important about the way we set up our OSCEs. OSCEs scores shall have some predictive value about what will happen in future practice. Petrusa ER. Taking standardised patient-based examinations to the next level. Teaching and Learning in Medicine, 2004; 16:98-110

38. 1998 Evidence for the predictive value of current assessments is relative small. If, as there seems little correlation between scores obtained when demonstrating the sequence of breast examination maneuvers with the ability to detect a lump, yet good evidence that detecting a lump in a silicon breast simulation, why do we still use the former method for assessment? Chalabian J, Dunnington G. Do our current assessments assure competency in clinical breast evaluation skills?. Am J Surg, 1998; 175:497-502

39. 2006 National Board of Medical Examiners hold their first USMLE Step 2 CS examination under the auspices of the Educational Commission, using the same five sites as for ECFMG. The exam lasts eight hours at the centre, that sounds familiar, but there are only twelve stations, compare with 14 in Canada. Each encounter with an SP is videoed and 34-35,000 go through this process annually. The NBME use the contrasting group method to set their standards.

40. 2009 The last word goes to Cees van der Vleuten: OSCEs are not inherently reliable. Neither objectivity or standardisation improve this situation. OSCEs do not test the whole picture. Global rating scales are superior to check lists. Computers are efficient replacements for written stations and offer other possibilities. The Borderline Group Method remains a cheap and valid way to set standards with. But Maastricht use real life work based assessments to measure clinical competence today.

41. Thank you