Unit 3. Think-aloud Usability Testing

1. Unit 3. Think-aloud Usability Testing • 3.1 Basic Issues In Think-Aloud Testing • 3.2 How To Conduct A Think-Aloud Test • 3.3 Think-Aloud Testing Vs Heuristic Evaluation

2. Resource • In the ftp server • Audio and video clips CDForThinkAloud • Transcripts: SSD4_TA_ALL_transcripts_v1.4.2.doc

3. 3.1Basic Issues in Think-Aloud Testing • 3.1.1What Is Think-Aloud Usability Testing? • 3.1.2 Ethics for Empirical Studies

4. 3.1.1What Is Think-Aloud Usability Testing? • an empirical technique for assessing the usability of a prototype of an interface • "may be the single most valuable usability engineering method"-- Nielsen, 1993 • How to do? • ask a user to think-aloud while performing a task on your system • you watch silently and learn • how the user thinks about the task • where the user has problems using your system

5. 3.1.1What Is Think-Aloud Usability Testing? • root in psychology research — two observational techniques • Think-Aloud Protocol Analysis • Understanding Verbalization • Understanding Working Memory • Critical Incident Analysis

6. Think-Aloud Protocol Analysis • Cognitive psychology • interested in understanding how people solve problems • discover and understand the details of • what information people pay attention to • how they represent that information • how they bring prior knowledge to bear • what transformations they make to information in the course of solving some puzzle or performing some task

7. Think-Aloud Protocol Analysis • think-aloud protocol analysis: been in use for 40 years • a method of understanding these details of thought • In cognitive psychology research, this method has two parts • collect think-aloud data (protocols) • analyze the data by building a model of it (usually a computer simulation).

8. 1st parts to this method • Collecting think-aloud data • proven to be extremely useful in understanding the usability of computer systems • cognitive psychologists have studied it quite thoroughly, and UI design can learn a lot from their studies

9. two parts to this method-2 • making a formal model(形式模型) of the data and processes • has not been as useful in most UI design • although there have been some dramatic successes • The analysis step of think-aloud data in UI design uses the critical incident technique

10. Understanding Verbalization • three types of verbalizations of thoughts • Talk-Aloud (Type 1) • Think-Aloud (Type 2) • Mediated processes (Type 3) • these three types can be understood by thinking about what is in Working Memory (WM).

11. Understanding Working Memory

12. Understanding Working Memory • WM stores several types of things • It stores all the results of perception once those things have been understood by the person, • E.g.,The picture of the Lion in last slice • WM also stores all the information that is brought in from long term memory (LTM) to solve a problem • WM also holds all the intermediate states in a problem solution, information that is figured out along the way to the solution

13. Understanding Working Memory • WM stores several types of things • WM holds a lot of clues as to what a person was thinking about as they solved the problem or performed a task • On the other hand, WM does NOT hold the processes that are used on the information • WM holds to generate those intermediate pieces of information. • Those processes are used by thecognitive processor but are not explicitly represented in WM.

14. Theory behind Think-aloud protocols • People can verbalize(描述) the linguistic contents（可用词语表达的内容）of their WM • A lot of information in WM is already in linguistic form (expressed in words) • Type 1 protocol: • to "talk aloud," — get these pieces of information to come "out of their mouth" right after they enter WM

15. Type 1 protocol

16. a simple addition problem: 2+4 =?

17. Cognitive psychologists have shown that: • Cognitive psychologists have shown that • for the most part, asking a person to "talk aloud" as they work on a task does not • change their thinking strategies or • slow them down in their thinking.

18. However, much of the information is not linguistic in nature • with modern GUI computers systems • may include information about space, color, time, or other things that are notnaturally spoken about with words • This is not to say that theycan’tbe expressed with words, people have to learn • a vocabulary • how to translate the perceptual information they are getting into that vocabulary • Learn how to express those new thing

19. An example • wine tasting • if you are not familiar with the skill, you will not be able to translate the sensations on your tongue into words • you will have sensations on your tongue when tasting the wine (non-linguistic information), • you could learn the translation into words given enough time and exposure to the "language."

20. Type 2 protocol

21. Another example • do a jigsaw puzzle picturing a beach • Thinking aloud as you do the jigsaw puzzle would be as shown in the figure of next slice

22. Another example

23. What Cognitive psychologists tell us? • asking people to translate everything they are thinking into words ("think aloud") does not change the way people think about problems • it does slow them down（？） • In fact , some time it will speed them up(according to the book of Usability Engineering ) • This is the type of protocol most often used in usability research • can get a lot of information about the quality of the UI

24. The Type 3 protocol • The Type 3 protocol is when you ask someone to verbalize, but make some sort of demand on them to add more processing to the information. • This type of protocol is not recommended for UI design because psychologists have shown that • this does changethe way people think as they solve problems, • in addition to slowing them down.

25. the Type 3 protocol

26. asking a person doing the jigsaw puzzle to explain how he found each piece.

27. Conclusion of type 3 • In type 3: • the information state reported was one that the person never passed through without the instruction to "explain," • the explanation of the process was not what she actually did.

28. Conclusion of type 3 • For these reasons • Type 3 instructionsshould be avoided • if you notice that a person seems to • be explaining rather than just reporting what they think • you need to put them back on the right track.

29. Critical Incident Analysis • Collecting the data. • by video tape • by action-and-voice software • Analyst role in designing the computer design. • analyze this data to decide how to improve the computer system design

30. Critical Incident Analysis • History • developed during World War II • develop procedures for the selection and classification of aircrew • This method also has parts, involving several different people in different roles • collecting the data and • analyzing it • UI design draws from both parts of this technique for its use.

31. original critical incident technique • observers report critical incidents that they witness in the course of performing a task • E.g., combat veterans reported actions of officers they observed during combat missions. • These observations are usually gathered through … after the task is performed • interviews or • questionnaires.

32. original critical incident technique • Concurrent recording of these observations is recommended in the original papers, but they were acknowledged to be impractical in many of the situations the critical incident technique was used • e.g., during combat missions. • The observations gathered are then categorized and interpreted by analysts and put into a final report that summarizes the findings.

33. the general procedure for the critical incident technique • Someone performs a task in the real world • e.g., combat missions. • An observer (who may or may not be the "someone" performing the task) reports critical incidents after the fact in interviews or questionnaires administered by an analyst. • The analyst categorizes and interprets the observations. • The analyst writes a summarizing report of the data and interpretations.

34. modified procedure of Usability studies in UI design have • A userthinks aloud as he or she performs a task using a prototype of the computer system being evaluated, usually in a laboratory setting, usually videotaped or using screen-and-voice capture software. • An analyst (who is not the user performing the task) looks at the recording of the think-aloud protocol session and reports critical incidents using the UAR format. • The analyst categorizes and interprets the observations. • The analyst writes a summarizing report of the data and interpretations.

35. Differences between the two procedure • users take the role of the "someone performing the task," • the usability analyst takes boththe roles ofthe observerandthe analyst in the original critical incident technique. • Watching the videotape and writing up UARs takes the place of after-the-fact interviews or questionnaires.

36. Definition of the term “critical incident”（关键事件） • The definitions used in the 1954 Psychological Bulletin paper are as follows: • “By incident is meant any observable human activity that is sufficiently complete in itself（本身）to permit inferences and predictions to be made about the person performing the act."

37. Definition of the term “critical incident” • “To be critical, an incident must occur in a situation where the purpose or intent (目的) of the act seems fairly clear to the observer and where its consequences are sufficiently definite（明确）to leave little doubt concerning its effects." • "Such incidents are defined as extreme behavior, either outstandingly effective or ineffective with respect to attaining the general aims of the activity."

38. The three important concepts mapped into UAR format • The three important concepts in these definitions can be mapped into our format for UARs • Map table

39. What a usability study includes? • the procedure for doing a usability study combines • the best of think-aloud protocols • the critical incident technique. • It gets at users’ thoughts • what they pay attention to, • what information they miss, • what prior knowledge they bring to the task, and • what is puzzling or clear to them. • it also provides a tractable way to • record important data in critical incident UARs • summarize the results in a final report.

40. 3.1.2 Ethics For Empirical Studies • Testing the Interface,not the participant • Voluntary Participation • Maintain Anonymity • Informed Consent • Laws

41. You are testing the Interface, Not the Participant • Your attitude • you are testing the interface, not the participant • including • everything you do with the participants in your study • everything you do with the data they generate • come up in every aspect of • the study ethics • procedures • data analysis

42. You are testing the Interface, Not the Participant • Mantra : The user is not like me. • You are the system designer • You can never think like a typical user • You know too much about the system • You cannot prevent that knowledge from helping you use the system

43. get information about how real users will deal with the system • To get information about how real users will deal with your system, • must do empirical testing with people who know as much or as little about computers and your specific system as you expect your real users will know • This leads to the inescapable conclusions that • participants in your study are extremely important

44. the Participant • They are always giving "good data" whatever they say or do, Why? • Since whatever they say or do • comes from a basis of knowledge like that of your eventual actual users • gives you a valid indication of what your actual users will think about your system.

45. You are testing the Interface, Not the Participant • Kinds of people to use the Software • Participant will not pay attention always • What we do? • No matter what the participants do, • always ask yourself what about your system indicate them in that direction, • rather than blaming the participant for not knowing enough or not paying attention or not reading. • You are testing the interface, not the participant.

46. Voluntary Participation • Participation must be voluntary • Do not put any pressure on people to continue in a study once they have started • Or It may get useless data • It is unlikely when sth. objectionable found about participating, if so , let them choose quit or not • it is important that you create an atmosphere in the testing situation • Tell the participants that they are allowed to stop at any time. • Pay bonus to the people

47. Watch carefully and stop at proper time • Be sensitive to stop • to be sensitive to the many ways people express a desire to stop a session • Some people would never say something as direct as "I want to stop now.“ • But they may express a lot of negative emotions • "I'm so stupid I'll never be able to finish this,“ • "This seems like it will go on forever, stupid &%@#$* computer!!!“

48. Watch carefully and stop at proper time • What you must do • gently ask, "Would you like to stop now?" • Do not wait for tears or for the user to throw the equipment across the room! • When they are in highly emotional state, you will not get any additional usable data from them anyway • it is better for both of you if you diplomatically terminate the session • remember to ask yourself later what in your system brought this user to this emotional state

49. Maintain Anonymity • It is your responsibility to maintain the anonymity of your participants, do as follows, • Store data under a code number • Map the number and name • Do not record participants face • Do not show videotapes without participants consent

50. Informed Consent • is fundamental to every kind of experiment using human participants. • Ethically Obligated to tell the particiant • what the experiment is about • what procedures will be used • what compensation they will receive • what they can do if they object to something in the study • because their participation is voluntary, they are free to stop and leave at any time.