Evaluation for Learning

1. Evaluation for Learning Paul Ramsden University of Sydney

2. The problem To evaluate university teaching effectively, we should apply the principles of good practice in assessing student learning. Sounds elementary...

3. “Chapter 2 has demonstrated that even the best of current practices are by and large not good practice...” Lewis Elton and Brenda Johnston ‘Assessment in universities: a critical review’ (http://www.ltsn.ac.uk/genericcentre/docs/Critical%20review%20of%20assessment %20research.rtf)

4. Source: McInnis et al, 2000

5. “There are no practice exercises – every piece of work is assessed, which tends to focus me on attaining marks rather than exploring ideas” “Assessment and marking have not been good. It’s quite subjective, and group work assignments are sometimes unfair in their assessment criteria (for example, everyone gets the same mark even though one person may do less work than another)” “The amount of multiple choice questions in some subjects does not provide opportunities to show how much you have learnt and understood.” “We need more feedback during semester. It is not reasonable to get a mark at the end of the year and have no idea what it is based on. It is also not ideal to front up to an exam not knowing how well we have been going in earlier assessments.”

6. Bad practice remains common • Multiple purposes, same techniques • No formative assessment at all • No match to learning objectives • Feedback too little, too late • Narrow range of methods unreflectively chosen • No student choice • Incorrect use of group assessment • Ignorance of Heisenberg! • Standards assumed to depend on norm referencing

7. Bad practice remains common • Interaction reliability x validity not understood • Unreliable assessments weighted less • Limited application of grade descriptors • Use of MCQs without professional training • Plagiarism opportunities not designed out – focus on technical fixes

8. Underlying all this is... • a focus on the producer’s concerns • a continuing emphasis on teaching (rather than learning) • a remarkable neglect of existing evidence (people seem to prefer ‘dissemination and projects’ despite overwhelming proof that it doesn’t work) • and a tendency to blame the student We should try adapting these conclusions to the evaluation (and management) of university teaching.

10. Sydney 1999-2003 Assignment: Design a system to evaluate teaching which leads to a better student experience and improved learning outcomes. … and make it work.

11. The response 1. Break the problem into manageable parts 2. Design systems that resonate with values and leverage strengths 3. Align evaluation with outcomes 4. Benchmark good practice 5. Test the impact against the evidence

12. The response 1. Break the problem into manageable parts 2. Design systems that resonate with values and leverage strengths 3. Align evaluation with outcomes 4. Benchmark good practice 5. Test the impact against the evidence

13. Break the problem into manageable parts • Make goals, methods, measurement and outcomes cohere ( aka ‘alignment’) • Recognise importance of perceptions/ theories in use • Design the system around the culture (avoid one-size-fits-all solutions) • Learn from others’ mistakes

14. The response 1. Break the problem into manageable parts 2. Use strategies that resonate with values and leverage strengths 3. Align evaluation with outcomes 4. Benchmark good practice 5. Test the impact against the evidence

15. Principles • Adopt a single, but flexible, SAL perspective, derived from research evidence. Use it to inform every policy and process • Plan for coherence between collegial and managerial strategies • Use an evidence-based approach to change and leadership, aligned with academic values

16. Mechanisms to leverage strengths • Academic Board reviews • Rigorous, peer review-driven QA process • Manage teaching proactively • Funding drivers aligned with research measures and national indicators • Plans that work • New role for academic development unit • Strategic projects

17. Management of teaching • Teaching Dividend currently $4.5M • Scholarship Index c. $650,000 • Teaching Improvement Fund $1.3M • Required T&L plans • Annually updated operational plans • Assess progress against targets • Condition of access to performance-based funds • Interrogated in Academic Board reviews

19. Strategic projects • Re-engineered academic development unit • First year experience: learning community • Expansion of training opportunities • including mandatory 21 hour training • Research-led teaching, including PIs • Evaluation and QA working group • Graduate attributes for a research university • Research supervision initiatives

20. The response 1. Break the problem into manageable parts 2. Use strategies that resonate with values and leverage strengths 3. Align evaluation with outcomes 4. Benchmark good practice 5. Test the impact against the evidence

21. Align evaluation with desired outcomes • Rewards and recognition at multiple levels • SI rewards trained staff and scholarly outputs • Array of student-focused evaluation instruments, consistent with SAL theory • New teaching awards • Performance linked to funding at Faculty level • Material support for changes • New promotions policy

22. Evidence-based academic promotions criteria 1. Fundamentals 2. Criteria Performance Research-led Student-focused Scholarship Leadership 3. Evidence

23. Fundamentals Fundamental things have got to be simple… we must look for simplicity in the system first. Ernest Rutherford

24. Fundamentals • Interest and explanation • Respect for students • Appropriate assessment • Clear goals and challenge • Independence: student control • Learning from students (Learning to Teach in Higher Education, Chapter 6)

25. Even more fundamental... • Positive attitude towards students • Ability to communicate well • Lively interest in improving teaching

26. And plainer still... The aim of teaching is simple: it is to make student learning possible.

27. Performance A lecturer should appear easy and collected, undaunted and unconcerned, his thoughts about him and his mind clear for the contemplation and description of his subject … His whole behaviour should evince a respect for his audience Michael Faraday

28. Performance • Planning(e.g. effective subject design, clear objectives) • Process(e.g. presentation technique, WebCT design) • Assessment(e.g. use of variety of appropriate methods) • Outcomes(some evidence of link to learning) • Evaluation(some evidence of use of evaluation to improve)

29. Research-led teaching This atmosphere of excitement, arising from imaginative consideration of knowledge, transforms knowledge. A. N. Whitehead

30. Research-led teaching • Imagination and enthusiasm: a shared journey to understanding rather than delivery of content • Effective design of curricula to engage students in inquiry • Materials make use of primary sources, recent discoveries, progress in field • (“If you can’t explain it to the charlady, you don’t know anything about it”)

31. Student-focused teaching The two secrets of lecturing from which everything else follows: first, to believe that you have something worth telling your audience; second, to imagine yourself as one of that audience. R.V. Jones

32. Student-focused teaching • Use of evaluation evidence to redesign curriculum • Use of assessment data to modify teaching strategy • Focus on relation between students and subject matter • Choice of technique reflects level of student knowledge • (From “Did I make the goals clear?” to “Are the goals clear to the students?”)

33. Scholarship in teaching What is needed is for teachers in higher education to bring to their teaching activities the same critical, doubting and creative attitude which they bring habitually to their research activities. Lewis Elton

34. Scholarship in teaching • Systematic use of best available evidence to select and deploy teaching and assessment strategies • Publication of refereed journal articles on university teaching in discipline • Invitations to address international conferences on university teaching

35. Leadership in teaching She successfully inspired us to transform the course and to re-focus on our students. She melded a diverse group of academics into a team of great teachers. A lecturer

36. Leadership in teaching • Policy development and implementation • Successful re-design and coordination of courses; team leadership in teaching • Mentoring of junior academics as teachers • Application of teaching strategies and curriculum designs in other institutions • Coordination of benchmarking activity with other universities

37. Criteria are hierarchically ordered... Non-negotiable basis: Performance Second level: Research-led Third level: Student-focused Fourth level: Scholarship Fifth level: Leadership … leading to a structure that can be mapped on to promotion at different levels.

38. And the evidence? • Are the basics in place? • Use multiple sources (never rely on student evaluations alone) • Evaluate teaching like research • Use peer review if possible • Use hard data when available (e.g. S of T publications) • Do the different sources tell a similar story? • Do the claims made by the teacher match the evidence?

39. Required Administrative Academic Recognised and encouraged

40. Required • Fundamental values: • research intensive • academic-led • self-regulation • evidence base • international • referencing • focus on student • experience Administrative Academic Recognised and encouraged

41. Required Surveys TPIs and performance funding External QA benchmarks Required training in teaching Teaching & Learning Plans Academic Board reviews (self-evaluation, visit, report) Policies on teaching: evaluation, assessment, ICT QA, promotions Administrative Academic University teaching awards Supervision awards Teaching Improvement Fund Scholarship Index Research-led teaching (policy and indicators) ‘Guidelines for Good Practice’ (teaching, learning with ICT); ITL courses and support groups: quality, graduate attributes, first year experience, research-led teaching Recognised and encouraged

42. The response 1. Identify the problem 2. Use strategies that resonate with values and leverage strengths 3. Align evaluation with outcomes 4. Benchmark good practice 5. Test the impact against the evidence

43. Benchmarks Oxford Student surveys (SCEQ), QA policies Lund QA policies UCL QA policies OU ICT evaluation and QA ANU ICT in T&L for research universities Monash Research-led teaching PIs Queensland Student surveys (SCEQ) Hong Kong Academic development standards

44. The response 1. Identify the problem 2. Use strategies that resonate with values and leverage strengths 3. Align evaluation with outcomes 4. Benchmark good practice 5. Test the impact against the evidence

46. Demand indicator for high quality students (percentage of offers to students with UAIs 95 or greater 1999-2003)

47. Changes in the Sydney first year experience, 1999-2002 1999 2002 % change Teaching staff give helpful feedback 39 48 +9 Teamwork skills developed 48 58 +10 Motivated to do best work 38 48 +10 More confident to tackle new problems 47 54 +7 IT supports my learning 56 64 +8 Problem solving skills developed 52 58 +6 Feel part of a learning community 39 53 +14 Satisfied with dept/faculty admin. 51 68 +17 Overall course satisfaction 66 71 +5 Overall satisfaction (services & admin) 57 65 +8 Percentage agreements, annual survey

50. Oxford vs. Sydney, 3rd year Undergraduates (broad agreement)