Structuring an on-line assessment for HCI

1. School of Technology Structuring an on-line assessment for HCI Sharon Curtis and Mary Zajicek Department of Computing Oxford Brookes University

2. HCIQ • An on-line assessment system for a second year undergraduate level HCI course. • Total assessment consists of 50% HCIQ + 50% group coursework using User Centred Design. • The assessment questions are structured to assess understanding of HCI concepts in a way that allows for automated marking. • Can take place in multiple short sittings in computer labs with students in close proximity , in such a way that cheating is prevented. School of Technology

3. Different aspects of HCIQ • Question structure • Blueprints • Equivalence classes and randomisation • Test from the student’s perspective • Technological details • Pilot test • Student perceptions • Lessons learnt School of Technology

4. School of Technology

5. Question Structure • A situation is described (the stem of the question) along with several small multiple choice questions (MCQs) pertaining to that situation. • Questions can also include diagrams, and are not restricted to a binary True/False format, e.g. an alternative format might be Always/Sometimes/Never. • The description in the stem of the question must give sufficient information so that a student knowing and understanding the assessed concepts cannot be confused about which is the correct answer. • Care is taken to ensure that it is not possible to obtain marks simply by guessing, and as recommended by Bull & McKenna (2004), a system using negative scoring for wrong answers was used together with an option of “Don’t Know” (scoring 0). School of Technology

6. Advantages of HCIQ compared with exam • These questions have potential to show that students understand the concepts, rather than ask students to explain concepts, which usually results in them parroting short descriptions from lecture notes, whether the concepts are understood or not. • Linking several multiple choice questions (MCQs) with a single stem means less time reading and understanding the descriptions of the situations than with several separate MCQs. • Such questions are quick to answer, as students don’t have to spend time thinking of how to phrase their answers and laboriously writing them out. School of Technology

7. Sample blueprint for an equivalence class Question Bank: Class Test December 2004 Equivalence Class: EVAL (for qns on usability evaluation) Blueprint A description is given of a usability evaluation or testing situation. Students are then asked to say, for each of three usability evaluation concepts, whether the concept is applicable to this situation. The three concepts will include • “formative evaluation” or “summative evaluation” • 2 other concepts from the following list of topics taken from the lecture notes: “GOMS analysis”, “cognitive walkthrough”, “heuristic evaluation”, “iterative design”, “performance measurement” “thinking aloud protocol”, “focus groups”  Answers are either Yes (+2 marks), No (-2 marks), or Don’t Know (0 marks). School of Technology

8. Equivalence classes and randomisation • Questions are randomised using a system of equivalence classes of questions to avoid cheating either by looking at the student next to them or discussing questions. • All equivalent questions had the same format and marking scheme, according to a blueprint for each equivalence class of questions, which defines the equivalence. • This ensured that all students are assessed at the same level and that they themselves are confident that they have been assessed fairly. • Questions within an equivalence class are deliberately constructed to have a variety of orderings of the small MCQs, with mixed patterns of correct answers. • Individual test papers are made up with one question chosen randomly from each equivalence class, ordered in a way specified by the test organiser, where there are several equivalents for each question.. School of Technology

9. Test from the student’s perspective • The student: • is given a printed test paper • logs in at a computer, going to the web page with the answer sheet on it • fills out the answer sheet both on the screen and on the paper (in case of power failure and/or for independent verification purposes), along with their student ID number, and a unique keyword identifier written on the paper • hands in his or her paper before leaving the test • The student may be able to see the screen of a nearby computer. However this will not enable them to cheat, as the questions are not written on the screen, but only written on the printed test papers. • Test papers are not easily readable by a nearby student, and are individual (the students know about the randomization), it is not easy for them to see for a particular question whether it matches that of his or her neighbour. • When the student discusses the test with classmates, it will be discovered that they had different questions. School of Technology

10. School of Technology

11. Rationale for our technological approach • Much software to produce question banks (e.g. Respondus, QuestionMark) • Many can cope with randomization... but not the particular generation of non-online papers with matching answer forms without questions • Maybe existing XML languages would at least be a help for a format to store the question bank? • QML not suitable • IMS Question & Test Interoperability Specification looked suitable but rather complicated! • Settled for writing our own - could always translate easily enough into another XML language later School of Technology

12. Pilot Qns Class Test Qns HCIQ Program(Marking) HCIQ Program (Generating) Checking XML Pilot Qn Bank XML Class Test Qn Bank Individual Marking Schemes LaTeX sources for test papers Class Test Marksheet (for whole class) Test papers Web page for submitting test answers Students take test Electronic answer files School of Technology

13. Technological Details • Questions stored in XML • Papers generated in LaTeX (from a template LaTeX file giving the rubric to the paper) • Marking schemes generated along with papers (plain text files) • Keywords file provided the keywords to connect papers with marking schemes • HTML page generated to match the structure of the questions • Program HCIQ to do the checking, generating and marking School of Technology

14. Pilot Test • Some consolidation time was spent talking to students about the pilot test • In addition to the students doing a small version of the test that would be very similar to the real thing, students were timed to see how long they took on the questions relating to part of the course • Students also filled out a questionnaire with their opinions on several aspects of the test • Doubled up as a practical about usability testing • It was very much a usability test for us!! • Very valuable as a dry run School of Technology

15. Run-up to the Real Thing • Students with accessibility problems from the first test contact to make sure they'd be ok for the real class test • Sample papers produced for students, one with answers • Each blueprint of an equivalence class used on the final test had a question following the same blueprint on at least one of the two sample papers • Some consolidation time was spent going over a sample paper School of Technology

16. Class Test Statistics • 139 students took the test • Class test scores: • Average score was 55% • Standard deviation 15% • Comparing group assignment to class test: • On average, test scores were worse than assignment scores by 6% • The average difference between a student's scores for test and assignment was 20% School of Technology

17. Student Perceptions (1) Having the test: • Many positive comments about the test vs an exam • Nobody said that they preferred exams "I think it's a step in the right direction for many computing modules" "Good idea" "It's a good idea for people who are technically-minded. Very easy to use and navigate" "This is a very good way to test this module. This type of test makes the whole test process more interesting and less stressful. You are able to take your time and not much writing involved, which is always a good thing." School of Technology

18. Student Perceptions (2) Negative comments were about aspects of implementation rather than the test itself "Don't see point of filling answers on paper, should be online only" "Good idea, though defs keep filling in both ie paper and computer for a while just in case!" "submit button at the top not needed, could result in people submitting answers before attempting questions" "I really don't like the negative marking system; I know that won't change, but I just wanted to say that" Students provided several constructive comments to improve the final test School of Technology

19. Lessons Learnt (1) • The blueprint book is very valuable if you haven't done much of this sort of thing before • A lot of questions to set, BUT setting questions is a whole lot more fun and pedagogically interesting than marking • Makes you rethink about what you're teaching the students and how you can do it better • A lot of time and attention to detail required; this paid off in the smooth running of the tests • Investment in the questions means we are looking forward to next year being a whole lot less work! School of Technology

20. Lessons Learnt (2) • It is very challenging getting the wording of the questions right • Students who think the answer is unclear will complain that they think the question is unclear • Pilot test extremely helpful to both staff and students • An extra person on hand for invigilation is useful • Too much paranoia about the technology not working is almost enough School of Technology

21. References • Joanna Bull and Colleen McKenna: "Blueprint for computer-assisted assessment" RoutledgeFalmer, 2004. • QML (Questions Markup Language) http://cnx.rice.edu/content/m10140/latest/ • IMS Question & Test Interoperability Specification http://www.imsglobal.org/question/ School of Technology