Comprehensive Evaluation of Computer-Based Learning Resources: Methods and Best Practices

1. Evaluation (of Computer Based Learning Resources) What is evaluation? What to evaluate? How to evaluate? Phil Barker, philb@icbl.hw.ac.uk (Roger Rist, rjr@macs.hw.ac.uk)

2. What is Evaluation?

3. What is Evaluation • An attempt to appraise quality/suitability of a resource • Evaluation is not the same as Assessment (at least not in the UK)

4. Why Evaluate? • Selection - which resource is better? • Monitoring – is our development heading in the right direction? • Justifying - show something is value for money. • Validating - did we make the right decision? • Improving - can we improve if we change something? • Research - add to body of knowledge.

5. Formative and Summative • Formative Evaluation: Evaluation during a project to guide future development. • Summative Evaluation: Evaluation after completion of a project to appraise its success.

6. When the cook tastes the soup, it is formative evaluation; when the dinner guest tastes the soup it is summative evaluation.

7. Integrative Evaluation • The context in which a resource is used affects how useful it is. • Formative / Summative evaluation during the use of a resource for teaching and learning. • Formative / Summative evaluation of the use of a resource for teaching and learning.

8. What to Evaluate

9. What to Evaluate • no simple answer to this, it will depend on “Stake holder” concerns. • “Stake holder” - the people or organisations who are affected by the success or failure of what you are evaluating. • So who are the stake holders?

10. Stake holders • Students • Lecturer / teacher • Institution / department / school management • IT support officers • Other support services (staff development, library) • Funders • ...

11. Stake holder Concerns • Improved learning • Cost • Easier access • Motivation • Ease of use • Effect on other systems / processes

12. An Example: EASEIT-Eng • From the mission statement “...enable an academic tutor to make an informed choice from a range of evaluated computer based materials...”

13. An Example : What to evaluate • navigation, online help, error trapping, use of colour, ease of installation, media used, system requirements ...

14. An Example : What to evaluate • Usability • potential role, clearly stated objectives, assessment and feedback, challenge and motivation ...

15. An Example : What to evaluate • Usability • Pedagogic aspects of package • Reason for wanting to use CAL, How many students, Availability of computers, was CAL perceived as optional extra, relevant to subject of course ...

16. An Example : What to evaluate • Usability • Pedagogic aspects of package • Integration into course

17. How to Evaluate

18. Evaluation Methods: Checklist Uses: • Making sure early versions of software conform to standard guidelines • or • Checking whether a range of materials meet your criteria.

19. Evaluation Methods: Checklist Heuristics : “Rules of thumb” for successful design • Squires (for learning): http://www.media.uwe.ac.uk/masoud/cal-97/papers/squires.htm, • Nielson (for web usability): http://www.useit.com/papers/heuristic/ .

20. Checklist Pros and Cons • Write down two or three pros and cons of using checklists.

21. Checklist Pros and Cons Pros • Can be cheap and easy • Can apply to prototypes • Quick: can cover a lot of factors Cons • Degree of interpretation needed • Not much depth or richness to information gathered

22. Evaluation Methods: Questionnaires Uses: • Obtaining feedback from large numbers of students • Obtaining semi-quantitative ratings from students • e.g. Likert scale questions

23. Likert Questions Circle 1 if you disagree strongly, 2 if you somewhat disagree, through to 5 if you agree strongly: • This software is the best thing since sliced bread 1- 2 - 3 - 4 - 5

24. Questionnaire Pros and Cons • Write down two or three pros and cons of using questionnaires.

25. Questionnaire Pros and Cons Pros • Can get input from many people • Can get quantitative input Cons • Can be difficult to get questionnaires returned • Often only get the simplest form of question completed • Questions may be mis-interpreted

26. Interviews Uses: • Anything that requires more than a two-word answer. • Anything where the questions need explaining. • Explorative work.

27. Type of interviews • Closed: like a spoken questionnaire • Conversational: let the interviewee take the lead • Can be one-to-one or involve a (focus) group

28. Interview Pros and Cons • Write down two or three pros and cons of using Interviews.

29. Interview Pros and Cons Pros • Open ended, can pick up on important issues as they are raised • Can make sure questions are understood • Rich information Cons • Time consuming for informant and evaluator

30. Other approaches • Observation • User diaries / logs • Nominal group technique • Controlled Experiments • Pre-post testing • Concept maps And more, see LTDI Evaluation Cookbook http://www.icbl.hw.ac.uk/ltdi/cookbook/

31. Report Need to detail: • the concerns addressed • the methods used • the data gathered • what was inferred • conclusions / recommendations

32. Reference • LTDI Evaluation Cookbook http://www.icbl.hw.ac.uk/ltdi/cookbook/

33. Summary • What is Evaluation? • Formative & Summative • Integrative evaluation and context • Stake holder concerns • Methods