Inspire Research Ltd 10 th January 20 11

1. JISC LLL WFD AssemblyMaximising Impact – assembling & leveraging the evidence baseDr Rachel Harris Inspire Research Ltd 10thJanuary 2011

2. Curriculum Delivery & Evaluation The programme invited projects to “transform how they deliver and support learning across a curriculum area through the effective use of technology”. 13 projects funded by JISC, and 2 by Becta • What did they want to know? • How did they do it? • What impact did they have? (And, what was the evidence?) Project outputs: http://jiscdesignstudio.pbworks.com/ Project blogs aggregated at: http://www.netvibes.com/circlejisc#Curriculum_Delivery_Projects Inspire Research Ltd

3. Live map Delivery evaluation map Inspire Research Ltd

4. Impact on the bottom line • eBiolabs – no more breakages or how to balance a centrifuge • Cascade – reduced admin time & the evidence to prove it • eReflect – students engage with assessment feedback Inspire Research Ltd

5. eBiolabs • Since introduction of eBiolabs, student lab prep time increased from 20 to 35 minutes (average). • 54% students agreed they “felt well prepared when they entered the lab” compared to 18% prior to introduction of eBiolabs. • Designed to support one 40 credit unit in one School. Now supports ten units across the Faculty of Medical and Veterinary Sciences. • Approx 120 teaching staff impacted by use of eBiolabs at Bristol. Currently, 900 student users. Inspire Research Ltd http://ebiolabs.learnsci.co.uk/

6. Cascade – Oxford University • “We forecast savings equivalent toover 38 weeks of productive admin time a year.”(Savings identified via historical data and time motion studies.) • “These efficiency savings, and ourability to quantify them, have been useful when we have needed to make the case to fullyembed and sustain these services going forward.” Inspire Research Ltd http://cascade.conted.ox.ac.uk/

7. eReflect – Westminster University Inspire Research Ltd https://sites.google.com/a/staff.westminster.ac.uk/mac/Home

8. “The e-Reflect feedback suggested that I speak to the module leader … I wouldn’t normally have thought to do this and it helped. He gave me ways I could improve and suggested a different way of approaching some of the subject matter. It has helped I think because since then my marks have shot up.”Link to video Inspire Research Ltd

9. Evaluation: things to think about • What are your evaluation questions – what do you want to know? • A bit on reporting Inspire Research Ltd

10. Questioning the project • How well was the project implemented? • To what extent were the objectives met? • What were the barriers and enablers that made the difference between successful and disappointing outcomes? • What else was learned? (why or how did outcomes happen?) Adapted from Davidson, E.J. (2009) Improving evaluation questions and answers: Getting actionable answers for real-world decision makers. Presented at AEAconference. Inspire Research Ltd

11. Value or So what? • How valuable are the outcomes to stakeholders (students, lecturers, admin staff, the sector)? • Was the project worth doing, given the resources used? (Consider the balance of resources to outcomes.) • Why would the project, or aspects of it, be worth implementing elsewhere? (How reusable is it?) • How sustainable is the project? (Could it continue or expand with limited external funding?) Adapted from Davidson, E.J. (2009) Improving evaluation questions and answers: Getting actionable answers for real-world decision makers. Presented at AEAconference. Inspire Research Ltd

12. Saving a million How do they know?Average cost of commercial alternatives to MyStudyBar = £115 Estimated saving based on MyStudyBardownloads, multiplied by ave cost = £1M Inspire Research Ltd

13. What is valuable about your project? In ten words outline a • Benefit to particular stakeholders(learners, lecturers, admin staff, senior managers, the sector) Inspire Research Ltd

14. Telling the evaluation story • Executive Summary - with headings linked to your key evaluation questions (or objectives). • Background– contextual issues that influenced the project– what the project evaluated and why, including Key project evaluation questions. • Methodology – the evaluation approach and techniques used (& why). • Findings/Results - sections that address evaluation questions, summarising what was found, followed by the evidence. Inspire Research Ltd

15. But! How policy makers & practitioners access research • Very short time to access project outputs • More likely to trust information received face-to-face (or from peers) • Reluctant to pay for access to academic journals (or ‘pay’ time to look for project findings) • Access new research mostly via the internet Adapted from Beckmann, K., & Mason, C. (2010). An investigation of the knowledge exchange practices of the end-users of sustainability research. Inspire Research Ltd

16. And, it is worth remembering • Institutional processes are really complicated. • ‘Project’-based evidence usually plays a veryminor role (in institutional decision-making). • It is possible to influence policy/your institution, but you need: • a holistic understanding of thecontext you are working in, • additionalskills, • to really want to do it. Adapted from Young, J. (2008). Strategies to Enhance Research Impact. ODI, London. Retrieved from http://www.slideshare.net/RECOUP/strategies-for-impact-and-policy-relevance-v2 Inspire Research Ltd

17. Any questions and thankyou! Dr Rachel A Harristel: 07779 980333email: rachel@inspire-research.co.ukweb: www.inspire-research.co.uk