Evaluating the Quality of the e-Learning Experience in Higher Education

1. Evaluating the Quality of the e-Learning Experience in Higher Education Anne Jelfs and Keir Thorpe, Institute of Educational Technology (IET), The Open University

2. Introduction • Complexity of student experience • Problems facing evaluators • Different research methodologies • Audits promote evaluation • Holistic view – why? how? when? and what? of evaluation

3. Pressures • Develop transferable student abilities • Pressure within the UK and worldwide • Governmental and employer pressure • Requirement – learners to participate in a networked information society • Emphasis on student-centred learning

4. Evaluators • Experienced evaluators needed in production, usability and effectiveness of online delivery • The Internet and ‘blended learning’ with ICT elements now widespread

5. Promoted by government initiatives Cost effectiveness of e-Learning Permits tailored education Can develop in ways which impoverish education Time & production costs of development Students & tutors lacking skills to use Two Sides of e-Learning

6. The Impact of Quality Assurance • Parallel to increased use of ICT is rising interest in quality assurance strategies • Lighter QAA touch & emphasis on stakeholders – students, employers, funders, society • Emphasis on outcomes rather than processes • QAA looking for an institution’s strategy

7. Why Evaluate the Learner’s Experience? • Subject review & institutional audits • Act as: educational ‘connoisseurs’ with approaches and tools to use in order to act as experts informing policy • Require: awareness and understanding of what is experienced • Require: personal judgement • Require: ability to articulate subtleties to a wider audience

8. Is the student experience exceptional? Are there defects in the system or delivery? Fit for purpose? Value for money or adds costs? Transformative/Value adding? Students’ perceptions? Effect on student numbers? Questions to Ask yourself

9. Metrics for e-Learning Evaluation • Efficiency of the teaching and learning process • Readability & usability of materials • Communication • Social relationships • Costs

10. Quality of the e-Learning Experience • Efficiency of the teaching and learning process • Accurate student data • Drop out rates • Student assessment results • Student satisfaction ratings • Training of tutors

11. Quality of the e-Learning Experience • Efficiency of the teaching and learning process To best understand the potential for e-Learning: • Start with 1-2 courses • Is this course needed by students? • Is the assessment electronic? • Collection of computer log data

12. Materials and Communication • 3. Communication • Communication with tutors & other students • Skills in using technology • Access from home/campus/3rd party • Group work 2. Readability & usability of materials • Peer review of materials • Reuse as learning objects • Online exercises • Study skills • e-Library and e-journal access

13. Social Relationships and Costs • 5. Costs • Costs per student of recruitment • Training of tutors • Costs of electronic access • Reuse of learning materials & learning objects 4.Social Relationships • Hours working online • Student motivation • Student support & interaction with other students

14. Robson’s 3 Approaches to Human-Computer Interaction • Learning theory is dominant – how students and their interaction with computers fit together • How students learn when using computers • Using the technology to teach in ways that are novel and unique to the environment

15. How are we going to Evaluate the e-Learning Experience? • Evaluation goals need to be clearly defined • Is this a formative or summative evaluation? • How should the data be gathered? • Should quantitative or qualitative methods be used?

16. How are we going to Evaluate the e-Learning Experience? Methods include: • Surveys • Observations • Semi-structured interviews • Focus groups • Cognitive walk-throughs • Qualitative studies can provide examples of usability difficulties

17. How are we going to Evaluate the e-Learning Experience? Surveying students’ views: • detects feelings of overload • informs necessary course adjustments Survey methods: • paper based with • telephone supplementary interviews • increased use of electronic surveys • focus groups

18. When to Evaluate the e-Learning Experience? • Dependent on whether the evaluation is formative or summative? • When is the student deemed to have completed the course? • Will formative evaluation during the course engage students more? • Harvesting evaluation information from student interaction with the course

19. What does the Evaluation provide? • Audio/video record of the computer screen, the user’s comments and keystrokes Baseline data to: • make adjustments, possibly ‘instant’ feedback • for external audit • shape e-learning strategy • input into associated resources – library and information literacy provision

20. Data Protection • Students and tutors need to be kept informed about how the data they provide is being used • Students and tutors retain the right to remove what they have provided at any time • Evaluators have to ensure secure access to the data • Evaluators must get consent especially if using the data in a different context

21. Conclusion • The Open University is interested in the quality of the student experience • To improve that experience needs understanding of student and staff needs • Here we have aimed to provide a framework in which to conduct evaluations • Current work on modelling courses will provide data on points of concern that arise in blended and e-learning

22. To praise & ask queries, contact Anne: Contact a.e.jelfs@open.ac.uk To complain, contact Keir: k.m.thorpe@open.ac.uk