University of York May 2008 Using assessment to support student learning Graham Gibbs

1. University of York May 2008 Using assessment to support student learning Graham Gibbs

2. Personal Background Open University 1975-80 and 1997-2003 • Top in National Student Survey, especially assessment and feedback ratings Oxford Brookes University 1980-1997 • Most ‘coursework’ assessment • Systematic course design University of Oxford 2004- • Least ‘coursework’ assessment • Top in National Student Survey, especially assessment and feedback ratings • ‘No course design’

3. Personal Background Practical books and articles about assessment ‘53 Interesting ways to assess your students’ ‘Assessing Student Centred Courses’ ‘Assessing Large Classes’ Consultancy to universities on strategic decisions about assessment policy Research into the impact of assessment on student learning

4. Student experience of assessment

5. “I just don’t bother doing the homework now. I approach the courses so I can get an ‘A’ in the easiest manner, and its amazing how little work you have to do if you really don’t like the course.”

6. “I am positive there is an examination game. You don’t learn certain facts, for instance, you don’t take the whole course, you go and look at the examination papers and you say ‘looks as though there have been four questions on a certain theme this year, last year the professor said that the examination would be much the same as before’, so you excise a good bit of the course immediately…”

7. “The feedback on my assignments comes back so slowly that we are already on the topic after next and I’ve already submitted the next assignment. I just look at the mark and throw it in the bin”

8. “The tutor likes to see the right answer circled in red at the bottom of the problem sheet. He likes to think you’ve got it right first time. You don’t include any workings or corrections – you make it look perfect. The trouble is when you go back to it later you can’t work out how you did it and you make the same mistakes all over again”

9. “One course I tried to understand the material and failed the exam. When I took the resit I just concentrated on passing and got 98%. My tutor couldn’t understand how I failed the first time. I still don’t understand the subject so it defeated the object, in a way”

10. “I do not like the on-line assessment method…it was too easy to only study to answer the questions and still get a good mark … the wrong reasoning can still result in the right answer so the student can be misled into thinking she understands something … I think there should have been a tutor-marked assessment part way through the course so someone could comment on methods of working, layout etc.”

11. “We were told this course was going to be an opportunity to be creative, to take risks. Then in week five we were hit with a multiple choice question test and we realised what it was really all about.”

12. Summative assessment that is redundant • Most students can, in their first year, predict their final results with some accuracy • As few as 5% of assessments are necessary to produce the same overall grades

13. Assessment that improves learning • The case of the Engineer • The case of the Manager • The case of the Pharmacist • The case of the Psychologist • The case of the Accountant

14. The case of the engineer • Weekly lectures, problem sheets and classes • Marking impossible • Problem classes large enough to hide in • Students didn’t tackle the problems • Exam marks: 45%

15. The case of the engineer • Course requirement to complete 50 problems • Peer assessed in six ‘lecture’ slots • Marks do not count • Lectures, problems, classes, exams unchanged

16. The case of the engineer • Course requirement to complete 50 problems • Peer assessed in six ‘lecture’ slots • Marks do not count • Lectures, problems, classes, exams unchanged • Exam marks increased from 45% to 85% Why did it work?

17. The case of the engineer • time on task • social learning and peer pressure • timely and influential feedback • learning by assessing • error spotting • developing judgement (internalisation of standards) • self-supervision(meta-cognitive awareness)

18. Assessment that improves learning • The case of the Engineer • The case of the Manager • The case of the Pharmacist • The case of the Psychologist • The case of the Accountant

19. “Conditions under which assessment supports student learning”

20. Quantity and distribution of student effort 1 Assessed tasks capture sufficient student time and effort 2 These tasks distribute student effort evenly across topics and weeks

21. Quality and level of student effort 3 These tasks engage students in productive learning activity 4 Assessment communicates clear and high expectations to students

22. Quantity and timing of feedback 5 Sufficient feedback is provided, both often enough and in enough detail 6 The feedback is provided quickly enough to be useful to students

23. Quality of feedback 7 Feedback focuses on learning rather than on marks or students themselves 8 Feedback is understandable to students, given their sophistication

24. Student response to feedback 9 Feedback is received by students and attended to, and is acted upon by students to improve their work or their learning

25. Effective assessment tactics • Bioscience: poster reports • Engineering: sampling lab reports + cheap feedback • Law: essay requirements + sampling + models • Estates management: project exams • French Literature: critiquing texts under examination conditions

26. Assessment Experience Questionnaire Measures extent to which the ‘conditions’ are perceived to be met • Quantity and distribution of effort • Quality, quantity and timeliness of feedback • Use of feedback • Impact of exams on quality of learning • Deep approach • Surface approach • Clarity of goals and standards • Appropriateness of assessment

29. ‘University A’ is the Open University • 8 assignments per course • Detailed written feedback on every assignment • Quality assurance of feedback • Less funding per student than any other university, mainly spent on feedback • Best student ratings nationally • Much pedagogic research • Formative-only early assignments improve retention • Computer-based assignments reduce retention and performance

30. …then I went to Oxford • Oxford responds in a limited way to most national quality assurance guidelines • learning outcomes • assessment criteria • alignment of assessment with aims • Oxford has not ‘modernised’ its assessment • reliance on examinations, little assessed coursework, little summative assessment of any kind, no modularisation

31. …then I went to Oxford • Oxford responds in a limited way to most national quality assurance guidelines • learning outcomes • assessment criteria • alignment of assessment with aims • Oxford has not ‘modernised’ its assessment • reliance on examinations, little assessed coursework, little assessment of any kind, no modularisation • Outstanding quality of student experience at Oxford • student retention of 98% (1st in UK) • Oxford ranked 1st for teaching in UK (Times, Guardian) • better CEQ scores than elsewhere in world • better NSS ratings than the Open University

32. Research questions • What are the characteristics of programme level assessment environments that are associated with positive student learning responses? • Are the characteristics of programme level assessment environments that are most closely associated with positive student learning responses those that quality assurance regulations emphasise?

33. Research design • Three contrasting universities (Oxford, pre-1992, post 1992) • Three contrasting programmes in each (Humanities, Science, Applied Social Science) • Characterise assessment environments • Read documentation (all modules) • Interview programme leader, lecturers and students • Administer AEQ • Explore relationships between characteristics of programme level assessment design and qualities of student learning • with Harriet Dunbar-Goddet, Chris Rust and Sue Law • funded by the Higher Education Academy

34. Coding characteristics of programme level assessment environments • % marks from examinations • Volume of summative assessment • Volume of formative only assessment • Volume of (formal) oral feedback • Volume of written feedback • Timeliness: days after submission before feedback provided • Explicitness of criteria and standards • Alignment of goals and assessment

35. Coding characteristics of programme level assessment environments • % marks from examinations • High: more than 70% • Med: between 40% and 70% • Low: less than 40%

36. Coding characteristics of programme level assessment environments • Explicitness of criteria and standards • High: clear criteria for most assignments & exams; link to grades; effort made to enable students to internalise criteria & standards • Low: explicit criteria and standards rare and/or nebulously formulated; marks/grades arrived at through global judgment in tacit way; no effort to enable students to internalise criteria and standards

37. Range of characteristics of programme level assessment environments

38. Range of characteristics of programme level assessment environments • % marks from exams: 17% - 100%

39. Range of characteristics of programme level assessment environments • % marks from exams: 17% - 100% • number of times work marked: 11 - 95

40. Range of characteristics of programme level assessment environments • % marks from exams: 17% - 100% • number of times work marked: 11 - 95 • number of times formative-only assessment: 2 - 134

41. Range of characteristics of programme level assessment environments • % marks from exams: 17% - 100% • number of times work marked: 11 - 95 • number of times formative-only assessment: 2 - 134 • number of hours of oral feedback: 3 - 68

42. Aspect of assessment Oxbridge Post-1992 Pre-1992 Volume/frequency of formative only assessment High Low Med % marks from exams High Low Med % marks from coursework Low High Med Alignment of learning activity with assessment Low High Med Explicitness of goals/outcomes and criteria Low High Low Institutional assessment environments

43. Patterns of assessment features within programmes • every programme that is high on the volume of formative assessment is low on the volume of summative assessment • no examples of high volume of summative assessment and high volume of feedback

44. Patterns of assessment features within programmes • every programme that is low on the volume of summative assessment is high on the volume of formative assessment • no examples of high volume of summative assessment and high volume of feedback • there may be enough resources to mark student work many times, or to give feedback many times, but not enough resources to do both

45. Relationships between assessment characteristics and student learning

46. Assessment characteristics and student learning response: 1 When the level of explicitness of criteria and standards is high, students’ experience is characterised by: Less coverage of the syllabus Less and poorer quality feedback Less use of feedback Less learning from the examination Less deep approach

47. Assessment characteristics and student learning response: 2 When the level of alignment of goals and standards is high, students’ experience is characterised by: Less coverage of the syllabus Less and poorer quality feedback Less use of feedback Less appropriate assessment Less clear goals and standards Less learning from the examination Less deep approach

48. Assessment characteristics and student learning response: 3 When the level variety of assessment methods is high, students’ experience is characterised by: Less and poorer quality feedback Less use of feedback Less appropriate assessment Less clear goals and standards Less learning from the examination Less deep approach More surface approach Less overall satisfaction

49. Assessment characteristics and student learning response: 4 When the volume of formative-only assessment is high, students’ experience is characterised by: More coverage of the syllabus More and better quality feedback More use of feedback More appropriate assessment More clear goals and standards More learning from the examination More deep approach More overall satisfaction

50. Assessment characteristics and student learning response: 5 When the volume of oral feedback is high, students’ experience is characterised by: More coverage of the syllabus More and better quality feedback More use of feedback More appropriate assessment More clear goals and standards More learning from the examination More deep approach More overall satisfaction