Usability Testing & Think Aloud

Usability Testing & Think Aloud 2/20/14 HCC 729

In-class Homework Critiques What happened after the UARs? Pair up and share your HEs with another group (5 minutes each) Identify good examples, questions

Sharing example HEs • Good examples • Questions / unsure?

UAR Awards • Best written UAR • Funniest usability problem • Interesting edge case

Discussion Inspirations or UI news? Readings For next time: I’ll pick selected examples (to save time)

Recap from last week • HTAs & personas feedback coming soon (Saturday) • Heuristic evaluation process

Phases of Heuristic Evaluation 0) Pre-evaluation training (optional) Give evaluators needed domain knowledge & information on the scenario • Evaluate the interfaceto find usability problems • Record the problem • Aggregate problems • Assign severity rating 5) Find a solution complexity rating

Find which heuristic is violated • Simple & Natural Dialog • Speak User’s Language • Minimize User’s Memory Load • Consistency • Feedback • Clearly Marked Exits • Shortcuts • Good Error Messages • Prevent Errors • Help & Documentation Nielsen’s 10 Heuristics http://www.nngroup.com/articles/ten-usability-heuristics/

Marie Silverstrim

Today: Working with Real Users

When should you user test? Early and often

Categories of user testing • Formative user testing • Usually happens early in the design process • Identify problems for the next round of design • Can be done with non-functional prototypes • Summative user testing • Done later in the development process • Can be part of QA

Assumptions of user testing • Even an expert can’t find all the usability challenges • Expertise can blind us problems • Domain experts, technical experts • Understanding the system image can help us to find bugs • People focus on aesthetics / other issues

Think Aloud Method • Usually used as a summative evaluation technique: • Use it to validate/asses an interface • Could be used as a Formative one: • Use Think Aloud to discover process

Goals for Today • Things you will learn: • How to conduct a Think Aloud • When to conduct a Think Aloud • What data can & should be captured • Preparation: picking users & tasks • Think Alouds are useful to discover how users think about doing a task • Interface evaluation, problem solving, learning, etc. • Follows detailed usability testing

How to conduct a Think Aloud Evaluation

What is the Think Aloud Method? • “[Think-Aloud]...may be the single most valuable usability engineering method” [Nielsen, p. 195] • An empirical technique where you ask users to “think aloud” as they work on a task you believe is interesting or important, using the interface you are interested in improving

Think Aloud Metrics • Think Aloud is about observing • how users think about tasks • where users have problems with a system • What kind of information do you get? • Qualitative Insights (from a few users) • What kind of information do you NOT get? • Performance Measurement (time, accuracy) • This comes from a usability test

Think Aloud Process • Preparation: Design a test plan • Preparation: Choose what to observe • Preparation: Prepare for the user test • Run the user test • Aggregate, organize, and analyze your notes

Preparation: Design a Test Plan • What problem is your system trying to solve? • Who are your target users? • What scenarios are you evaluating? • What are your usability goals? • What is your hypothesis?

Choose Tasks to Observe • Choose what to observe • Don’t pick obvious or impossible tasks • Should be more than just “try out the interface” • Pick tasks that are: representative, critical, frequent • Often better to integrate small tasks as a sequence

Example Task Sheet • Company ABC’s photo website • Task 1: A friend mentions ABC’s website. You Google it & reach its front page. Without leaving this page, think aloud about your thoughts. What would you click next? • Task 2: You’re preparing a talk about Paris & need some photos for illustrations. Use ABC’s website to find a photo from Paris. Remember to think aloud as you work. • Task 3: You’ve found good photos from Paris, and you’d like to bookmark them for future use. How would you bookmark a photo that you are viewing now? Remember to think aloud as you work. • Task 4: Your friends have asked for your Rome photos from last year. Publish one photo from Rome using ABC’s website. The photos are on your desktop in the “Rome” folder. Remember to think aloud as you work.

Notes about the Task Sheet • Give each user multiple tasks to perform • Break complex tasks into small parts • Not all tasks need to start in the beginning • Clearly structured tasks help the user focus on evalution • Make the tasks independent from each other so you can skip tasks and move around • i.e. make the tasks independent • Always remind them to think aloud

#3 Piloting your user test • It is always good to “pilot” your user test before you recruit real users • Practice your think aloud • Learn the script • Test out the tasks, see how long it takes

Run a Think Aloud: Warm-up • Introduce yourself • Describe Think Aloud: • May initially be awkward • It’s important that they talk • You may prompt them • Best to provide examples • Promise confidentiality • Get permission to record

Run a Think Aloud: Tasks • Introduce your system • Don’t rationalize or defend system • Remind them this isn’t your system • Describe the tasks • Use your Task Sheet • User can read these aloud • Ask for questions

Run a Think Aloud: Tasks • Keep the user talking • A few phrase you might use • “Please keep talking” • “Tell me what you are thinking” • “Tell me what you’re trying to do" • “Can you tell me more?” • “What is your opinion?”

Run a Think Aloud: Tasks • You are only an observer • Don’t help do the task • Unless they are really struggling and taking too long • Do respond to & note questions • Take good notes • Record all problems, not just what you were looking for • Helpful: watch, A/V recordings • Must: take notes

Notes about your notes • Keep track of notes for each task (for each user) • Keep track of the actions and the behaviors you saw for each action • Include important quotes • Find the fastest note taking medium that isn’t distracting

User Test Notes • Example: Company ABC’s photo website • User 1: Task 2 (search for photos) • Scrolled through several pages of photos • Found Paris photos but didn’t use search button • User 1: Task 3 (bookmark photos) • Successfully clicked on bookmark button with no delay • Used copy and paste to enter bookmark data • Confused by the fact that fields aren’t necessarily in the same order as they would normally be entered • User 2: Task 2 (search for photos) • “so...it’s not one of the categories...so I’ll search for it” • “I want to search for the phrase Paris, but don’t see a way to do it other than paging” (finds photos by clicking Next button)

Run a Think Aloud: Tasks • When it’s over, it’s helpful to ask overall impressions • What were the best three things about it? • What were the worst three things about it? • Thank the user • Provide any compensation

Organize your notes • Aggregate, organize, and analyze your notes • This is not just about transcribing what you recorded. • You need to analyze both what they say & do. • Establish criteria for “critical incidents” • Record these critical incidents on UAR forms

Critical Incidents • First introduced by Flanagan, 1954 • “To be critical, an incident must occur in a situation where the purpose or intent of the act seems fairly clear to the observer and where its consequences are sufficiently definite to leave little doubt concerning its effects.” • In other words: describes a discrete event • Observable behavior leads to specific consequence • Can be good or bad!

Bad Critical Incidents • Possible criteria: • User articulates goal but doesn’t attain goal within [3 minutes] • User articulates goal, tries several things (or same thing) before giving up • User articulates goal but tries multiple ways before finding a solution • User accomplishes task but in a suboptimal way • User expresses surprise or hesitation • User expresses some negative sentiment, either about interface or about their skills

Good Critical Incidents • Possible criteria: • User expressed some positive affect or says something is really easy • User expresses happy surprise • Some previous analysis has predicted a usability problem, but user has no difficulty with that aspect of the system

Use the data • Make sure your reflections are supported directly by the data • Don’t include your own hunches (there are other opportunities to do this)

Record your evidence using UARs!

Critical Incident UARs 1. Summarize user test notes 2. List critical incidents: both good & bad 3. Complete UARs: refer back to raw data 4. Try to judge why each difficulty occurred

User Test Report • What did Think Aloud tell you? • Does the UI work the way you thought it would? • Do the users take the approaches you expected? • Is something missing from your UI? • Rethink your design • Based on UAR rankings, what fixes will you work on? • Redesign is often more than just fixing bugs

Live demo • Need two volunteers

Pro Tips

Notes • Search “snow boots”! • Jump to bottom • What do widths mean? • Repeat size • Colors • Heel styles – what were they? • Walking through hierarchy • Didn’t see “womens” • Hard to see pictures • Positive – said what was missing • Backtracking • Out of stock

Notes on think aloud • Letting her know when to stop • Hard not to call out • Not clear when to end • Hard to keep track of both

Pro tips from our demo?

Think Aloud Introspection • Think Aloudscan disrupt a user’s thought process • When people are trying to figure something out • When a problem requires “insight” • If you notice your participant looking confused, now might not be exactly the time to push them to “Keep Talking”

Usability Testing & Think Aloud