Evaluating electronic textbooks: a methodology Ruth Wilson and Monica Landoni ECDL 2001 5 th September 2001

1. Evaluating electronic textbooks: a methodology Ruth Wilson and Monica Landoni ECDL 2001 5th September 2001

2. The Visual Book The WEB Book EBONI’s methodology for evaluating electronic textbooks Outline

3. Studied the application of the paper book metaphor to the design and production of electronic books Found that users were able to rely on their experience with paper books to interact with the electronic book. The Visual Book

4. Focused on the impact of appearance on the usability of textbooks on the Web. Comparison of two versions of a textbook chapter: Original version: very plain, one “page” per chapter, basic use of hyperlinks. Revised, “scannable” version: chapter divided between 10 “pages”, heavier use of hyperlinks, extra headings, indented lists, coloured keywords. Revised version was 92% more usable. The WEB Book

6. EBONI aims to: Evaluate the different approaches to the design of electronic textbooks Identify and report on the individual requirements of students and academics in learning and teaching using electronic textbooks Compile and promote a set of best practice guidelines for the publication of electronic textbooks EBONI: aims

7. Peter Kaiser. The Joy of Visual Perception. http://www.yorku.ca/eye/thejoy.htm Eric Chudler. Neuroscience for Kids. http://faculty.washington.edu/chudler/neurok.html C. Robin Timmons and Leonard W. Hamilton. Drugs, Brains and Behaviour. http://www.rci.rutgers.edu/~lwh/drugs/ Experiments:psychology Web books

8. Cliff McKnight, Andrew Dillon and John Richardson. Hypertext in Context. Print Original electronic version Revised electronic version Experiments:Hypertext in Context

9. Rosa Ainley (ed.). New Frontiers of Space… MobiPocket Reader Adobe Acrobat eBook Reader Microsoft Reader Paperback Experiments:commercial formats

10. Britannica Columbia Encarta Experiments:e-encyclopaedias

11. HP Jornada Franklin eBookMan PalmPilot Rocket eBook Softbook Experiments:portable ebooks

12. Selection of material Selection of participants Selection of tasks Scavenger hunt, memory tasks, high cognitive skill tasks Selection of evaluation techniques Questionnaires, observation, think-aloud, interviews Ebook evaluation model

13. Electronic books offer a diverse array of material for evaluation: Hardware devices, e.g. REBs, eBookMan, goReader Ebook reader software, e.g. Microsoft Reader, Adobe Acrobat Ebook Reader “Web books” Selection of material (1)

14. EBONI aims to identify design techniques which maximise information intake by users, in order to provide guidance to creators of digitised educational material. Texts can be compared according to three parameters: Format Content Medium Selection of material (2)

15. Four main roles can be distinguished: Participant Evaluator Task developer Task assessor Selection of actors

16. Tasks are a way of bringing together participants and the test material in a structured manner. Three types are suggested: “Scavenger Hunts”, or retrieval tasks Memory tasks High cognitive skill tasks Selection of tasks

17. Qualitative feedback can be obtained via: Subjective satisfaction questionnaires Behaviour observation Think-alouds Interviews Selection of evaluation techniques

18. Technique complexity Study X Interviews Think-aloud Covert observation WEB Book Questionnaires Task complexity Scavenger hunt Memory tasks High cognitive skill tasks

19. Sophisticated experiments can be “mapped” to simple experiments. Results of all evaluations will be comparable at some level. Results will feed into a set of best practice guidelines for producing electronic textbooks. Guidelines

20. Ruth Wilson: ruth.m.wilson@strath.ac.uk EBONI Web site: http://eboni.cdlr.strath.ac.uk/ To join the EBONI mailing list, follow the instructions at: http://www.jiscmail.ac.uk/lists/open-eboni.html Contact details