ICT for assessment and feedback on undergraduate accounting modules

1. HEA Social Science Workshop • and Seminar Series • Winchester Business School • 30 January 2013 • ICT for assessment and feedback on undergraduate accounting modules Pru Marriott and Lim Keong Teoh University of Winchester

2. Acknowledgement We are grateful for the funding received from the Higher Education Academy to conduct this research. Aim To obtain a snapshot of current usage of ICT for assessment and feedback on undergraduate Accounting modules in a UK higher education institution using a case study approach.

3. Contents Introduction Phased Online Assessment: Case study 1 – Summative Assessment Case study 2 – Formative Assessment Screen-cast Feedback: Case study 3 Conclusion

4. Introduction Assessment and feedback are fundamental in the teaching and learning process. In the UK, assessment and feedback have been regularly highlighted by the National Student Survey as a critical aspect that requires improvement (Surridge, 2006; Williams and Kane, 2009; HEFCE, 2009, 2010).

5. Case Studies Three case studies are presented and analysed in relation to the effectiveness of using ICT for assessment and feedback in accounting education as measured by students’ views on the technologies.

6. Case Study 1 – Summative Assessment • Compulsory 20-credit First Year Financial Accounting module • Delivered over 24 weeks

7. Case Study 2 – Formative Assessment • Compulsory 20-credit Second Year Business and Personal Tax module • Delivered over 24 weeks • 30% coursework and 70% examination • Weekly formative on-line assessments • Indicative mark awarded to enable students to track progress

8. Format of Case Studies

9. Students’ Responses to Questionnaire Over 60% of both first and second year students regard on-line summative assessment as adding value to their learning, rating highly the immediate provision of results at a time when the test is “fresh in their minds” (≥95%). On-line tests and assessments are very good as you receive an immediate grade and answers so you haven’t forgotten what the test was even on by the time you receive your marks. (CS1) On-line feedback that is instant means that I can clearly see where I have gone wrong soon after the exam has been sat whereas when there is a lengthy wait for exam results it is hard to see where you have gone wrong as it is more difficult remembering the questions. (CS2)

10. Students’ Responses to Questionnaire The detailed and personal feedback provided (≥80%) and its legibility (≥83%) are also perceived as benefits of on-line assessment. I can make my own corrections on difficult questions for me rather than go through each question with all the group. (CS1) I really agree with on-line feedback, it helps me to understand where I personally went wrong. (CS2) On-line tests were very useful and the on-line feedback removes the problems of [tutor’s] illegible handwriting. (CS1) The immediate release of results is very helpful, and certainly at school, handwritten feedback was not always very legible. (CS2)

11. Students’ Responses to Questionnaire Over 80% of all students consider on-line assessment to help their revision and prepare for subsequent assessments. On-line tests are easier to complete and quicker to get the results back so you have more time to practice and do better for the future. (CS1) The online test were a very good barometer of my learning, this coupled with the fact that they are still available to use as revision has made them an invaluable tool. (CS2) I strongly agree that on-line feedback is useful to revise from and prepare for subsequent assessments. (CS2)

12. Students’ Responses to Questionnaire

13. Students’ Written Comments • The majority of students (85%) do not have a problem with working on-line with under a quarter of students indicating a preference for paper-based rather than on-line tests. However, a significant minority of students have concerns with on-line assessments with 26% of students regarding on-line marking to be less reliable than human marking. • The instant feedback from online exams is useful although not always as in depth as needed. (CS1) • Regarding questions that require calculations, on an on-line test, you will only receive • full marks if your end question is correct, which is not a fair reflection of a true test. (CS1) • The on-line feedback is good, although sometimes it does not go into • enough detail and it is still hard to understand where I went wrong on • a question. (CS2)

14. Over 40% of both cohorts view on-line tests as favouring some students over others with 20% of first year and 30% of second year students finding it difficult to concentrate on questions when doing tests on-line. [On-line assessment] may benefit students who are more familiar with computers and also benefits students with poor handwriting. (CS1) The main problem with online working is that different students might have different IT skills. A student who might have very good subject knowledge may be disadvantaged if his IT skills are not very good. (CS2) I found that working online on computers was quite detrimental as it was hard to not get distracted by face book. (CS1) I agree that online tests make feedback more immediate than paper based, however, I find it hard to concentrate on a PC long enough to do an online test of longer than 30 mins. (CS2)

15. Despite no technical issues arising in any of the on-line tests 42% and 52% of first and second year students respectively considered technical problems to be an issue, making on-line assessment impractical. • I agree with the on-line feedback because it will be easier to read and hopefully results will be given out quicker, however, I feel doing a test online may complicate things and technical problems may occur. (CS2)

16. Students’ Responses to Questionnaire

17. Majority of students expressed a preference for on-line exams/tests to be a combination of formative and summative. Students’ experience of on-line assessment appeared to influence their preference. Majority regard on-line assessment to be more appropriate for numerical modules and inappropriate for complex questions.

18. The frequency of on-line assessments appears to be influenced by the nature of the assessment. Ideal duration of each test to be between 15 and 30 minutes.

19. Case Study 3 – Screen-cast Feedback • Compulsory 15-credit First Year non-specialist Managing Finance module • Delivered over 12 weeks • Assessment – 50% Coursework / 50% Examination • Mixed methods approach to data collection • On-line survey • Focus groups

20. Students’ Responses to questionnaire For all four themes over 85% of students agreed or strongly agreed that there are benefits to be received from screen-cast feedback.

21. The students indicated that feedback from the tutor should be a combination of audio and visual with 72% selecting screen-cast feedback as their preferred option.

22. Focus Group Comments Theme 1 – Feedback is Clear and Easy to Follow Last semester we had one lecturer that I couldn’t actually read the writing, so I didn’t actually know like what the feedback was, so in a way it sort of seemed pointless. With this tutor I know exactly what he meant because he was explaining it, showing it to us, it was a lot more useful than just getting sheets of paper with writing next to it … that you can’t read. (Group 5) I’d want the screen cast because I find it a lot easier. I think it’s due to my dyslexia, the writing is a lot harder to read but hearing it is a lot better for me. (Group 3) I prefer to be spoken to than just reading, so it [screen-cast feedback] has helped me learn because it’s spoken as well as pinpointing what was good, what was bad, but I think it would be much better than me just sitting here and reading it in black and white, because I take in information better if I can see it visually and I can hear it, so I think it’s definitely a plus. (Group 4) I like the way the arrow kept on moving and highlighting the bits he was talking about because … you know if it’s written then it’s not always like next to what you’ve done wrong and you can’t quite understand it, but because he highlights every bit as it goes, as he’s talking, it’s more understandable. (Group 2)

23. Focus Group Comments Theme 2 – Feedback Identified Students’ Strengths and Weakness You get more of a step by step analysis as to what you did right and what you did wrong whereas like the traditional way we get feedback … they’ll [the tutors] say a couple of things that probably really stood out to them but we don’t get a chance to really go through our assignments with our lecturers… whereas he [the tutor] went through it question by question which was quite handy, so that we knew where and when we’d gone wrong. (Group 2) … because you actually heard him [the tutor] talking to you, and you saw him like going through your essay and saying … oh this you did wrong here, but I can see why you’ve done that, and he’s actually like addressing the problem that you’ve done, and the fact that you can hear him saying it, you can see him going through. This really helps with my understanding. (Group 5) It’s quite motivational as I’m not sure what other lecturers are like but [the tutor] would say like good comments if you’ve done something well, whereas with the written one they don’t usually say whether you’ve done it well or not… (Group 3)

24. Focus Group Comments Theme 3 – Feedback is Discreet and Personal I just think it looked like he [the tutor] did take a lot of time over it, it just seems like a bit more personal because you’re getting an actual … this is what you did wrong, this is where you went wrong, rather than just the writing … when someone writes your assignment you just get … oh this is wrong, but you can’t sort of explain why with a written feedback really. (Group 1) I think it’s a bit more personalised as well because it’s directed to the group members and actually he [the tutor] was quite enthusiastic when he was talking back to the group members, so it made you listen a bit more than you may actually want to read the writing on the assignment. (Group 4)

25. Focus Group Comments Theme 4 – Amount of Feedback (thoroughness and ability to review) It’s more thorough as well, like the way it explained it was like … it’s also telling you what you should have done or what you could have done as well instead of just telling you what you did wrong, it’s like showing you a way that you could have done it. (Group 1) … to me to get the feedback from my teacher like face to face, I’m going to have to make an appointment and I go and see him and he doesn’t have time, he’s just going to have to rush through it sort of thing, but now I have it all the time and can watch it whenever I want, you know that’s much easier for me, I prefer doing it that way to be honest. (Group 2)

26. Conclusions – Case Studies 1 and 2 Continuous assessment important and powerful assessment practice that: • Supports high-quality learning and teaching • Measures student development • Provides appropriate support through timely feedback • Facilities self-reflection and personal development • Enables students to become actively involved in their own learning

27. Conclusions – Case Study 3 An innovative approach to delivering feedback using screen-cast technology has: • Increased the value placed on feedback by students. • Provided feedback that is considered by students to be • Personal, • Understandable • Supportive • Increased student engagement

28. Limitations and Scope for Further Work Limitations Case studies at one UK institution, one Faculty Anecdotally – student success rates increased but not an experimental design to test for causality. Experimental design possible – but ethical problems Where Next Study of what and how is being done with ICT and assessment at other institutions across the UK Students views on ICT and Assessment – across the UK