1 / 26

The OSTE: Objective Structured Technical Examination for Engineering students

The OSTE: Objective Structured Technical Examination for Engineering students. Nandini Alinier, Guillaume Alinier University of Hertfordshire, U.K. ICEE 2005. Overview. Background & origins Application in an Engineering context Stages of the OSTE Development of stations

orenda
Download Presentation

The OSTE: Objective Structured Technical Examination for Engineering students

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The OSTE: Objective Structured Technical Examination for Engineering students Nandini Alinier, Guillaume Alinier University of Hertfordshire, U.K. ICEE 2005

  2. Overview • Background & origins • Application in an Engineering context • Stages of the OSTE • Development of stations • Preparation for sessions • Pilot • First series • Second series • Analysis N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  3. Background • Objective Structured Clinical Examination (OSCE) (Harden 1975) • Originally developed in the mid-seventies in Dundee, Scotland • Used in healthcare education • Method to objectively assess the skills of healthcare trainees • Used in a project funded by the British Heart Foundation at UH during 2001-03 (Alinier et al 2004) • Perceived positively by students & lecturers (Alinier 2003) N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  4. OSCE • Comprises 15 to 20 stations • Station: short task performed in 5 mins • Practical exercise • Requires an assessor • Theoretical exercise • Pen & paper • Gap period between stations (1 min) • Students rotate individually through the stations • Allows time for assessors to complete marking & reset any equipment • Timing of session by an electronic clock N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  5. OSCE (continued) • Students given explicit written instructions • Objective marking sheet for each station N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  6. Example of a station from an OSCE N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  7. Applied to Engineering • Project undertaken in the School of Electronic, Communication & Electrical Engineering at the University of Hertfordshire, UK • Funded by the Higher Education Academy Engineering Subject Centre (Learning & Teaching Support Network(LTSN)Engineering) • Objective Structured Technical Examination (OSTE) N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  8. Study about the OSTE • Used for formative assessment • Piloted with level 2 BEng students • Conducted with level 1 BEng students (1st + 2nd semesters) • Each student did the OSTE twice • Circulated feedback questionnaires for students + assessors N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  9. Stages of the OSTE N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  10. Stage 1: Development of OSTE stations • Gathering a variety of ideas about essential electrical engineering skills: • Seminar within School • Call for ideas via Email • Identified appropriate range of exercises doable in 5 mins • 16 stations were short-listed • Developed instruction sheets & marking criteria for each station N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  11. Stage 1: Development of OSTE stations (continued) • Types of stations: • Theoretical: pen & paper • Practical: assessor required • Computer-based: assessor required N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  12. Stage 2: Preparation for OSTE sessions • Development of PC-based clock with an integrated voice prompt • Briefing of students • Training of staff to act as assessors • Designing of student & assessor feedback questionnaires N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  13. Stage 3: Pilot of OSTE • Group of 15 volunteer 2nd year BEng students • Resulted in changes in: • Tasks to be carried out • Marking sheet • Instruction sheet N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  14. Example of a computer-based station N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  15. Stage 4: First series of OSTE • 1st year BEng students • During 1st semester (Dec 04) • Formative OSTE without feedback • To determine the level of students’ performance at the start of the course without preparing them too much for their 2nd OSTE • Feedback collected from students + assessors N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  16. Stage 5: Second series of OSTE • Same group of 1st year students • During 2nd semester (Mar 05) • Run as “mixed-mode” OSTE: (Alinier 2003) • Collection of data about students’ performance • Feedback to students by assessor • Feedback collected from students + assessors N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  17. Results: students’ perspective • Out of 46 student feedback questionnaires • The OSTE was beneficial: 93.5% • The OSTE helped them developed confidence in using equipment: 73.9% • 52.2%: rated the OSTE as very good or good (5-point Likert Scale with 1=very good, mean=2.4 with 95% C.I. 1.4–3.4) • The OSTE should be used as a summative assessment tool: 28.3% • Number of times OSTE should be repeated per year: 2.9 (95% C.I. 1.4 - 4.4) N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  18. Results: assessors’ perspective • Out of 33 assessor feedback questionnaires • The OSTE should be incorporated into the 1st year BEng curriculum: 93.1% • The OSTE helped students develop their confidence in using equipment they were exposed to: 75.8% • 90.6%: rated the OSTE as very good or good (5-point Likert Scale with 1=very good, mean=1.5 with 95% C.I. 0.8 – 2.2) • The OSTE should be used as a summative assessment tool: 75.9% • Number of times OSTE should be repeated per year: 2.2 (95% C.I. 1.4 - 3.0) N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  19. Further analysis • Identify key weaknesses among students • Any correlation between performance in OSTE and performance as per in-class assessment N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  20. OSTE results N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  21. Conclusions • OSTE: perceived positively by students and assessors • Formative assessment  • Summative assessment ? N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  22. Thanks for your attention Any questions? N.D.Alinier@herts.ac.uk G.Alinier@herts.ac.uk N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

  23. References • Harden R M, Stevenson M, Downie W W, Wilson G M, "Assessment of clinical competence using an objective structured clinical examination", Medical Education, Vol 1, 1975, 447-451. • Alinier G, "Nursing students’ and lecturers’ perspectives of OSCE incorporating simulation", Nurse Education Today, Vol 23, No 6, 2003, 419-426. • Alinier G, Hunt W B, Gordon R, "Determining the effect of realistic simulation in nurse education: Study design and initial results", Nurse Education in Practice, Vol 4, No 3, 2004, 200-207. • Ross M, Carroll G, Knight J, Chamberlain M, Fothergill-Boubonnais F, Linton J, "Using the OSCE to measure clinical skills performance in nursing", Journal of Advanced Nursing, Vol 13, 1988, 45-56. • Hulett S, Gilder N, "The application and development of the Objective Structured Practical Examination in evaluating physiotherapy students’ performance at the University of Cape Town", South African Journal of Physiotherapy, Vol 42, No 2, 1986, 40-43. • Marshall G, Harris P, "A study of the role of an objective structured clinical examination (OSCE) in assessing clinical competence in third year student radiographers", Radiography, Vol 6, 2000, 117-122. • Mossey P A, Newton J P, Stirrups D R, "Scope of the OSCE in the assessment of clinical skills in dentistry", British Dental Journal, Vol 190, No 6, 2001, 323-326. • Townsend A H, Mcllvenny S, Miller C J, Dunn E V, "The use of an objective structured clinical examination (OSCE) for formative and summative assessment in a general practice clinical attachment and its relationship to final medical school examination performance", Medical Education, Vol 35, 2001, 841-846. • Harden R M, "Twelve tips for organising an OSCE", Medical Teacher, Vol 12, No 3/4, 1990, 259-264. • Sloan D A, Donnelly M B, Schwarts R W, Strodel W E, “The Objective Structured Clinical Examination: The new gold standard for evaluating postgraduate clinical performance”, Annals of Surgery, Vol 222, No 6, 1995, 735-742. N Alinier & G Alinier, ICEE 2005, July 25-29, Gliwice, Poland

More Related