Designing assessment with the ‘Assessment Design Decisions’ framework

1. Designing assessment with the ‘Assessment Design Decisions’ framework Phillip Dawson Office of the Vice-Provost (Learning & Teaching) Monash University

2. The ‘Assessment Decisions’ team • Dr Phillip Dawson, Monash (Co-lead) • A/Prof Margaret Bearman, Monash (Co-lead) • A/Prof Liz Molloy, Monash • Prof David Boud, UTS • A/Prof Gordon Joughin, UQ • A/Prof Sue Bennett, UOW • Dr Matt Hall, Monash

3. Agenda • “Assessment design decisions?” • Your assessment designs • The framework • Applying the framework • Summary and close (and feedback!)

4. 1. “Assessment design decisions?”

6. We know a lot about ideal practice Hattie, 2009

7. Actual practice is different

8. Research suggests … • Changing assessment ‘thinking’ in academics doesn’t necessarily change practice (Offerdahl & Tomanek, 2011) • “…achieving a balance between summative and formative assessment requires complex, contextual thinking” (Price, et al., 2011)

9. Focus on assessment design • Assessment policy and procedures • Decisions in the design and implementation of assessment • Judgements about student work ‘assessment design decisions’

10. 2. Your assessment designs

11. Your designs • Think of an assessment design • It might be new • You might be thinking of revising it • What is the impetus for change? • Pair-share

12. Absorb the complexity, but don’t try to read

13. 3. The framework

14. Framework design • Joined research data with conceptual frameworks, and literature • Drew from own experience as expert practitioners • Oriented towards educators’ agency rather than prescriptive

16. Engaging with the framework • One-page summary • Online ‘guide’ with detailed information, resources and educator experiences • 55-page document version of ‘guide’ • (as well as a ‘Five insights’ document to give to L&T leaders) • http://assessmentdecisions.org

17. 4. Applying the framework

18. Applying the framework • Focus on a specific unit • We will work through the six framework components • Address key questions on one-pager • Use hard copy or online versions

25. 5. Summary and close (and feedback) assessmentdecisions.org phillip.dawson@monash.edu philldawson.com Support for this project/presentation/resource has been provided by the Australian Government Office for Learning and Teaching. The views expressed here do not necessarily reflect the views of the Australian Government Office for Learning and Teaching.

26. References • Carless, D. (2009). Trust, distrust and their impact on assessment reform. Assessment & Evaluation in Higher Education, 34(1), 79-89. doi: 10.1080/02602930801895786 • Hattie, J., The Black Box of Tertiary Assessment: An Impending Revolution, in Tertiary Assessment & Higher Education Student Outcomes: Policy, Practice & Research, L.H. Meyer, et al., Editors. 2009, AkoAotearoa: Wellington, New Zealand. • Offerdahl, E. G., & Tomanek, D. (2011). Changes in instructors' assessment thinking related to experimentation with new strategies. Assessment & Evaluation in Higher Education, 36(7), 781-795. doi: 10.1080/02602938.2010.488794 • Price, M., Carroll, J., O'Donovan, B., & Rust, C. (2011). If I was going there I wouldn't start from here: a critical commentary on current assessment practice. Assessment & Evaluation in Higher Education, 36(4), 479-492. doi: 10.1080/02602930903512883

27. Extra slides after this point

28. How do university teachers make decisions about assessment?

29. Research design • 30 semi-structured interviews • Gritty, coalface, ‘actual’ not ‘ideal’ • Thematic analysis; coding against framework • Meaning-making from coded data

30. What we can say • Improving assessment is more than just a problem of knowledge transmission/translation • Rarely about rationally selecting from options • Assessment decisions are complex; situated; pragmatic

35. “I think a lot of us have good intentions, we just don't have the time” – science lecturer

36. Time

37. “I was redeveloping a unit, it had already had a particular format of assessment. I elected to run with that rather than to go though the processes of trying to alter it … I wasn't gonna [jump through] those other hoops.” – humanities lecturer

38. Committees and paperwork

39. “I don't think an assessment should be painful for the students or painful for the staff that assess it” – science lecturer

40. Beliefs

41. “technology becomes really critical where assessment is concerned. If you set something up and it doesn't work, they don't trust you. Getting them on board again is a killer … students can be very hostile to you making mistakes. They're not very forgiving” – arts lecturer

42. Technology

43. Improving assessment Requested supports Our analysis adds Understanding of freedom to move • Exemplars • Time, money, sessionals • Someone to help • Involvement from senior academics

44. More information http://assessmentdecisions.org A short paper: sclr.li/19 This research is supported by an Australian government Office for Learning and Teaching grant titled “Improving assessment: understanding educational decision-making in practice” (ID12-2254)

45. References • Carless, D. (2009). Trust, distrust and their impact on assessment reform. Assessment & Evaluation in Higher Education, 34(1), 79-89. doi: 10.1080/02602930801895786 • Hattie, J., The Black Box of Tertiary Assessment: An Impending Revolution, in Tertiary Assessment & Higher Education Student Outcomes: Policy, Practice & Research, L.H. Meyer, et al., Editors. 2009, AkoAotearoa: Wellington, New Zealand. • Offerdahl, E. G., & Tomanek, D. (2011). Changes in instructors' assessment thinking related to experimentation with new strategies. Assessment & Evaluation in Higher Education, 36(7), 781-795. doi: 10.1080/02602938.2010.488794 • Price, M., Carroll, J., O'Donovan, B., & Rust, C. (2011). If I was going there I wouldn't start from here: a critical commentary on current assessment practice. Assessment & Evaluation in Higher Education, 36(4), 479-492. doi: 10.1080/02602930903512883

46. Extra slides

47. Research and ideas from the literature • “there's some interesting papers that look at the use of wikis and peer learning and those sorts of things” – science lecturer (early-career)

48. Research and ideas from the literature • “That was really quite confronting to me and reading education literature, and I'd still find it difficult because it's completely different to my discipline … I just find it really, really difficult. Because I’m going, "My god, that's just their feelings."” – science lecturer (late-career)

49. Time andworkload • “I think a lot of us have good intentions, we just don't have the time” – science lecturer • “But the main thing is that, it has to be feasible from a resource point of view as well. And if at a resource point of view, as well, as that educational perspective as well.” – health professions lecturer

50. Being strategic • one of the other things that we learned as well was not to put too much data into the FEC documents, which I remember in the early days, again, we had 1500-word essay on topic X and then afterwards when you go 'that's crazy' we need to change it, we've got to go back to FEC and that load of paperwork and things. So, we do... You know, we are much more general about what we're putting into the FEC documents