Methods of User Evaluation for Human-Computer Interaction Design

1. ICS 463, Intro to Human Computer Interaction Design: 8. Evaluation and Data Dan Suthers

2. Evaluation • Formative: early and often; informs design • Summative: near end; have we done well? • We focus on formative

3. When and why? • Early in the life cycle • understanding target application • check understanding of requirements • quick filtering of ideas • Middle • predicting usability • comparing alternate designs • engineering towards a usability target • Later • fine tuning of usability • verifying conformance to a standard

4. Preview of Methods of Evaluation • Collecting users' opinions  attitudes • Observing and monitoring use  how users interact • Experiments  hypothesis testing • Interpretive evaluation  how used in natural settings (ecological validity) • Predictive evaluation  anticipated usability issues

5. Typical Procedure (formal evals) • Identify questions • Plan the evaluation • Pilot the evaluation and revise if needed • Run the sessions and collect the data • Analyze the data • Draw conclusions • Redesign and revise system • Details next week (Hix & Hartson)

6. Dimensions to consider Evaluation planning should consider • Characteristics of users • Types of activities • Environment of use • Nature of artifact So should design … from the outset!

7. Data • Types of Data • Objective versus Subjective • Quantitative versus Qualitative • What are the independent variables? • Dependent variables? • Controlled variables? Now on to details of evaluation methods …

8. Collecting Users’ Opinions Tells us about attitudes Caveat: "First rule of usability: don't listen to users, watch what they do!" Two major methods • Interviews - qualitative analysis • Surveys - quantitative analysis

9. Interviews • Structured: • fixed questions, fixed or conditional sequence • easier to conduct and analyze • may miss opportunistic information • Semi-structured • set of questions “to get to” • Flexible • no set questions or sequence

10. Questionnaires and Surveys • Large numbers, analyzed quantitatively • Design with your analysis in mind • Piloting important • Closed questions versus open questions • Types of closed questions • Checklists: background information • Likertscales: range of agreement or disagreement with a statement • Semantic differentials: place on scale of adjectives • Ranked order: e.g., rank in order of usefulness

11. Observing Users • Observing and monitoring use of artifact • in laboratory • in natural setting  how users interact with system  usability issues

12. Direct Observation • Researcher watches use, takes notes • Hawthorne Effect (users act differently under observation) may contaminate results • Record may be incomplete • Only one chance • Helpful to have shorthand and/or forms which which you are fluent

13. Indirect Observation • Video logging • User(s) body language, gestures • Screen activity • Two uses: • Exploratory evaluation: review tapes carefully and repeatedly to discover issues • Formal studies: know what you are looking for! • Interaction logging (software) • Often use two or more together • Must synchronize all data streams • High volume of data can be overwhelming

14. Verbal Protocols • Audio record of spoken language • Spontaneous utterances • Conversation between multiple users • Think-aloud protocol • Post-event protocols • Dangers of introspection, rationalization • Analyze along with video

15. Video/Verbal Analysis • Diversity of approaches • Task-based • how do users approach the problem • difficulties in using the software • need not be exhaustive: identify interesting cases • Performance-based • frequency and timing of categories of actions, errors, task completion

16. More on Analysis of Video/Verbal • Requires classification scheme, invented or borrowed • May involve inter-rater reliability • Often exhaustive and time intensive! • Tools important • we transcribe conversation to text merged with transaction log • better approach would be direct analysis of digital video

17. Software Instrumentation/Logging • Time-stamped logs • key-presses or higher level actions • record what happened but not replayable • Interaction logging • replayable • Synchronize with video data for rich but overwhelming data • Analysis issues are similar

18. Methods of Evaluation • Collecting users' opinions •  attitudes • Observing and monitoring use of artifact • in laboratory • in natural setting •  how users interact • Experiments • testable hypothesis • comparative with controlled variables • quantitative analysis • Interpretive evaluation • how used in natural settings • qualitative data and analysis • Predictive evaluation • predict usability issues based on model • applied to specifications or prototypes

19. Additional Methods • Experiments • testable hypothesis • comparative with controlled variables • quantitative analysis • we’ll get into this next week • Interpretive evaluation • how used in natural settings • qualitative data and analysis • Predictive evaluation • predict usability issues based on model • applied to specifications or prototypes

20. Assignment 7 (project groups) • The good news: your project has been accepted by The Boss! • The bad news: you have 6 weeks to finish it!

21. Assignment 7 continued • Write a 2 page plan expressed in terms of the Star model • What is the expected product? • At which phase will you start? • What is your estimated timeline? • How will you incorporate evaluation and other user-centered techniques, and use the outcomes to adjust the design? • What resources do you need? • This can be in outline form and will be assessed for its value as an efficient briefing for The Boss