Evaluating your project G oing beyond student satisfaction

1. Evaluating your projectGoing beyond student satisfaction Ivan Moore RAEng, Eng SC ivan@engsc.ac.uk

2. Evaluation • Why evaluate? • Minimise risk • Partnership with students • Professional practice • Publication

3. Developing teaching practice Theory

4. Developing teaching practice scholarship Theory

5. Developing teaching practice scholarship Previous experience Theory

6. Developing teaching practice scholarship Previous experience Theory Different contexts

7. Developing teaching practice scholarship Previous experience Theory Different contexts Teaching in practice

8. Developing teaching practice scholarship Previous experience Theory Different contexts Teaching in practice Outcome

9. Developing teaching practice scholarship Previous experience Theory Different contexts Teaching in practice Outcome Further scholarship Reflection, feedback

10. Principles of evaluation • An integral part of our teaching practice • An ongoing process, so that we learn from systematic reflection • Should be participatory • Should enable us to make appropriate modifications along the way • Should enable us to make judgments on specific sessions, but also to draw out wider implications

11. Purposes of evaluation • Mike Prosser • Quality Assurance • student satisfaction • the mean score is important as a measure of quality • Quality Enhancement • Student conceptions/how they experience the course • The deviation is important: more focused view

12. Purposes of evaluation • Mike Prosser • Is the learning environment/teaching approach having any influence on student conceptions/approaches? • A student experience survey is more important than a student satisfaction survey

13. Evaluating your development • Three approaches • Goal-oriented evaluation • Goal-free evaluation • Context-based evaluation • Several methods • Questionnaire • Focus groups • Structured interviews • frequent, large student numbers, superficial • summative, representative sample, rich information

14. Student learning as a journey The outcomes (or goals) The learning environment Starting point

15. Evaluating the learning journey The outcomes The learning environment Expectations and preparedness

16. 3 Evaluating the outcomes • What skills have they developed? • intellectual • professional • inter-personal • What other attributes or attitudes have they developed? • confidence • problem-solving • self-awareness (metacognition) The learning environment The outcomes Expectations and preparedness

17. 2 The learning environment • How are they experiencing the learning environment? • What opportunities is it providing? • What skills/attitudes do they feel they are developing? • What model or theory are you using? • motivation • sense of control • confidence The learning environment The outcomes Expectations and preparedness

18. 1 Expectations and preparedness • Expectations • Standards - challenge • skills and attitudes they will develop • extent of collaboration and independence • role of the tutor • Preparedness • meeting demands • working collaboratively • asking questions • active learning • reading and analysing • finding information The learning environment The outcomes Expectations and preparedness

19. Goal oriented evaluation • Determine your goals • 4 or 5 at the most • For each goal, devise 4 or 5 statements to test achievement or progress • Scored 1-5

20. Deconstructing an example • Find the statements that focus on: 1 Team work 2 Research skills 3 Motivation

21. Deconstructing a goal • Can you identify any theory or model to help you with this? • Motivation • interest in the subject/task • clear goals, tasks and standards • support • feedback • sense of belonging and sharing • success - improvement

22. Designing an evaluation questionnaire • In your group • Quickly agree and write down some learning experience(s) and outcomes you would like for your students [goals] • For each goal • Devise 4 statements that will help to determine if the goal is being achieved • Record the goals and statements on a flip chart

23. Goal – free evaluation (Scriven) • What did the students get out of the experience? • skills • attitudes • enjoyment • self-awareness

24. Goal – free evaluation (Scriven) • The module has helped me develop my ability to work as part of a team • Is goal oriented. The goal is team working • The most useful thing I learned in this module was • Is goal free. • Why? • Illuminative evaluation

25. Goal-free evaluation • In your group • Devise up to 3 ‘questions’ that you might ask students to answer that might provide information on what they are experiencing • Record your questions • Discuss what information you might receive from these questions • Draw up your conclusions for feedback • Share our outcomes

26. Context-oriented evaluation • Where, when and how are the students gaining the experiences and outcomes you wish for them? • (may not be within your module) • Since the beginning of the year/module: What skills, if any, have you developed? What helped you to develop these skills? How have you changed the way you study? What prompted you to make these changes?

27. Context-oriented evaluation • Since the beginning of the year: • What skills, if any, have you developed? • What helped you to develop these skills? • How have you changed the way you study? • What prompted you to make these changes?

28. Other methods • Focus groups • Structured interviews • Continuous feedback • What did you find most difficult/confusing today? • What things did you find helped you learn last week? • What should I: • Start/stop/continue?

29. The three minute paper • What was the most useful or meaningful thing you learned during this session? • What question(s) remain uppermost in your mind as we end this session? • What was the ‘muddiest’ point in this session? • What would you like me to stop doing? • What would you like me to start doing? • What would you like me to continue doing?

30. Other instruments • Kolb’s Learning Style Inventory (LSI) • Honey and Mumford’s LSQ • Sternberg’s thinking styles • Felder’s Index of Learning Styles • Weinstein’s Learning and Study Strategies Inventory (LASSI) • Entwistle’s Approaches to Study Inventory

31. The Course experience questionnaire(P Ramsden) • Designed as a performance indicator • 24 statements relating to 5 aspects • 1 overall satisfaction statement • Research-based • Drawn from statements made by students in interviews • Students with positive responses take a deep approach

32. The five sub-scales • Good teaching • Clear goals and standards • Appropriate assessment • Appropriate workload • Generic skills

33. The five sub-scales • Good teaching 3, 7, 15, 17, 18, 20 • Clear goals and standards 1, 6, 13, 24 • Appropriate assessment 8, 12, 16, 19 • Appropriate workload 4, 14, 21, 23 • Generic skills 2, 5, 9, 10, 11, 22

34. From Ramsden CEQ • The module has helped me develop my ability to work as part of a group • I have usually had a clear idea of where I was going and what was expected of me • I have found this module interesting • The tutor motivated me to do my best work • Learning this way has helped sharpen my analytical skills • There has been more assessment of what I have memorised than of what I have understood • This module has helped me to develop the ability to plan my own work