Technology-enhanced examinations at the University of Bradford

1. Technology-enhanced examinations at the University of Bradford John Dermo, Centre for Educational Development, University of Bradford University of Edinburgh 23rd April 2012

2. Technology enhanced exams at UoB • A strategic approach to technology enhanced assessment • Drivers: including the student perspective • Developing a facility for e-assessment • Innovative e-assessment at UoB

3. Pathfinder Project 2007-8 • e-Learning Pathfinder Programme • “Embedding Support Processes for e-Assessment” http://www.bradford.ac.uk/educational-development/educational-research/pathfinder-caa/

4. ITS4SEA Project 2007-9 Institutional Innovation Programme • JISC Institutional Exemplar Project • “Integrating thin client systems and smart card technology for flexible, accessible and secure E-assessment.” http://www.jisc.ac.uk/whatwedo/programmes/institutionalinnovation/its4sea.aspx

5. Drivers: reasons for focus on online assessment • University e-Strategy (2004-9) • LTA strategy and Academic Framework (2009-14) • National Student Survey • Increased use of e-learning • Student expectations

6. In 20 years HE has seen radical and unprecedented change (Education Act, 1992; Dearing Report, 1997; Roberts Report, 2003; Leitch Report, 2006; Browne, 2010; CSR, 2010) • Learners have different expectations and assumptions about their HE experiences • The student body has become dramatically more heterogeneous • Impact of student fees

7. NATIONAL UNION OF STUDENTS’ PRINCIPLES OF EFFECTIVE ASSESSMENT (2009) • Should be for learning, not simply of learning • Should be reliable, valid, fair and consistent • Should consist of effective and constructive feedback • Should be innovative and have the capacity to inspire and motivate • Should measure understanding and application, rather than technique and memory • Should be conducted throughout the course, rather than being positioned as a final event • Should develop key skills such as peer and reflective assessment • Should be central to staff development and teaching strategies, and frequently reviewed • Should be of a manageable amount for both tutors and students • Should encourage dialogue between students and their tutors and students and their peers

8. “Please, Sir, may I have some more exams?” With apologies to Charles Dickens and Ronald Searle Becka Colley, Dean of Students, University of Bradford. Keynote speech at eAssessment Scotland, 2011 http://www.slideshare.net/rcurrant/please-sir-may-i-have-some-more-exams

9. Student Union Feedback There was a strong preference to retain two exam sessions. This was felt to be more manageable in terms of revision. Students were not in favour of one exam session at the end of the academic year. A staged approach with summative assessment was viewed as the most appropriate. Also, January exams give an opportunity for students to ‘trial’ their exam revision and writing skills. [UBU feedback on review of the academic year, June 2011]

10. Developments at UoB • Dedicated support • Move towards integrated processes and systems • Workflow model and clear roles and communication channels • Approved policies • Training materials and sessions, briefings • Develop new flexible space for e-assessment

11. 100 seat e-assessment cluster

13. VMware and Sun Ray servers Smartcards: link each exam to a different card

14. Benefits of e-assessment suite for online assessments • Can re-image software available very quickly between exams • Easy to control software available, flexible using Appsense • Can link assessment to card to allow multiple simultaneous exams in single location • Easier to invigilate – more secure • Room easier to find • More pleasant environment • Easier to manage extra time students

15. First used January 2009, since when the use for summative e-assessment has gradually increased:Usage: 2500 different students per year, from more than 30 modules covering range of subject areas across disciplines “I’ve already told the DVC that we will need another cluster like this within 2 years” Associate Dean (Teaching and Learning) Life Sciences “This is just what I always wanted but we couldn’t do on the PCs!” Lecturer in School of Management

16. Growing interest in e-assessment • Further demand on facilities, especially for summative assessment • More efficient use of space (smartcards, flexible scheduling) • “Overflow” room to increase capacity to 130 • Single rooms available when necessary

17. Some Student Feedback

18. Varied use of e-assessment suite • exam periods: block booked by exams office for summative e-assessment • term time – managed by room bookings for • one off e-assessments • regular teaching sessions • otherwise: open 24/7 as self access cluster for students

19. 24/7 Cluster Use • Room in regular use • “vibrant environment” • Pleasant learning space • Encourages group work • Demand during exam period • But… limitations of thin-clients?

20. ChallengesHow to manage the resource 365 days a year?Who owns the facility?How to get the most out of the investment?How to manage larger groups?Booking the room?Server capacity to allow 100+ logins? Expanding the use of the room to include essay exams (cf interest in e-submission and e-marking)

21. Integrating Institutional Processes

22. But don’t forget the human processes which go with the technical systems

23. Innovation requires commitment and acceptance of responsibility across the institution instructors course teams invigilators students department admin exams office disabilities office senior management IT services support teams room bookings line managers estates

24. Lessons Learned at Bradford? Challenges are not just technology-based Communication is key Your systems are only as good as the data in them Automated systems require humans to play their part

25. We all know what an exam looks like, don’t we?? • …to this. • From this…

26. Examples of authenticity Department of Biomedical Sciences: • Combination of web-based genetic research tool (BLAST) with MCQs to deliver authentic, real-life tasks which require application of knowledge and understanding in a real-life professional context • Can be assessed automatically = great for large numbers of students

27. Examples of authenticity School of Medical Sciences: • Use of formative “e-assessment for learning” (in F42 and via VLE) to deliver automatic generic feedback to students. • Work has been carried out on delivering question level feedback and topic based feedback – students would appear to best appreciate a combination of these.

28. Examples of authenticity School of Optometry: • Use F42 for “Active problem-based learning” • Formative group based case studies • Information is automatically and gradually released during process • Students make joint decisions, then receive email feedback. • Group focus initially then individual summative process

29. Other Developments • “Hybrid” assessments • Some questions automatically marked, others done by a human • More efficient use of human resources to allow time to be spent on students and feedback processes.

30. Other Developments • School of Health Studies, School of Management and School of Lifelong Education and Development: • Using e-peer assessment • Students grading performance and/or give feedback collaborative group tasks. • This assessment can take place either all together in the e-assessment suite or remotely via the VLE.

31. Other Developments “Activity-based Student-centred” (AbSc) teaching and learning: an Engineering perspective at Bradford • 8 e-Assessments in 12 weeks in F42 (including 6 bi-weekly assessments, one e-assessment on lab work and one mid-term e-assessment) • Supported by online quizzes – self assessment with feedback done in own time in VLE • Final summative e-assessment in F42

32. F42

33. Thank You Any questions?j.dermo@bradford.ac.uk