Chapter 12 Observing Users

1. Chapter 12 Observing Users Youlan Hu Xiaoqian Hu Nov 15, 2002

2. Outline • What and when to observe • Approaches to observation • How to observe(in lab and in field) • Data collection techniques • Indirect observation • Analyzing, interpreting and presenting data

3. What and When to Observe • Observation is valuable any time during design stages • Help understand users’ need in early design • Find out whether it meets users’ needs during later refinement of a product • Goals and questions determine the paradigm and techniques used • Provide a focus for observation • Should guide all evaluation studies

4. Approaches to Observation • “Quick and dirty” observation • Any where, any time • Find out what’s happening quickly and with little formality • Observation in usability test • Prototype/products • In controlled setting • Observation in field studies • Early on • Real-world test of prototype/products

5. Type of Observation

6. Observers • Outsiders • In lab or field studies • Recording is continuous • Data analysis and observation almost simultaneous • More objective than participant observation • Participant observation

7. Participant Observation • Field studies • Must get co-operation of people observed • Informants are useful • Recording might be interrupted • Data analysis is continuous • Interpretive technique • Questions get refined as understanding grows • Reports usually contain examples

8. In lab – Focus on the details of what individuals do – Easier to control – Easier to replicate – more objective In field – Focus on context in which individuals perform their work – More realistic – Harder to control other factors In Lab Versus In Field?

9. Direct observation In same room Can be intrusive Only see it one time Notes usually incomplete May use 1-way mirror Help get informal and direct information Indirect observation More distance Reduces intrusiveness, but doesn’t eliminate it Gives archival record Data has to be synchronized and managed Need time to analyze data How to Observe

10. Direct Observation – in controlled environments • Issues before observation • Decide what laboratory to do the tests • Set up appropriate equipment • Test equipment to make sure it works • Provide an informed consent form for users to read and sign at the beginning of the study

11. Simple Observation Method • User is given the task, and the evaluator just watches the user • Problem:does not give insight into the user’s decision and attitude

12. Think-aloud Method • Subjects are asked to say what they are thinking/doing • Gives insight into what the user is thinking • Very widely used in industry • Potential problems: • Can be awkward for subject • Think aloud may modify the way users perform their task.

13. Constructive Interaction Method • Join pairs of participants to work together • Perhaps have one person be semi-expert and one be novice • Remove awkwardness of individual think-aloud • Provides insight into thinking process of both beginner and intermediate.

14. How to observe – in the field • Need to consider in advance: • State initial study goal and question clearly • Select a framework to guide activity in the field • Decide how to record events • Be prepared to go through notes and other records as soon as possible after each evaluation session • Highlight and separate personal opinion from what really happens

15. How to observe – in the field(cont’d) • Need to consider in advance: • Refine goals and questions while observing • How to gain acceptance and trust of those subjects • How to handle sensitive issues • Consider working as a team • Whether and how to involve informants • How to analyze the data • Plan to look at situation from different perspectives

16. Frameworks to Guide Field Observation • Why use frameworks • Organize/structure and focus observation • Simple framework to capture context • The person. Who? • The place. Where? • The thing. What? • The Goetz and Lecompte(1984) framework • Who is present? • What is happening? • When does the activity occur? • Where is it happening? • Why is it happening? • How is the activity organized?

17. The Robinson Framework • Space. What is the physical space like? • Actors. Who is involved? • Activities . What are they doing and why? • Objects. What objects are present? • Acts. What are specific individuals doing? • Events . What kind of event is it? • Goals. What are they trying to accomplish? • Feelings. What is the mood of the group and individuals?

18. Dilemmas • When should I stop observing • When see similar patterns of behavior being repeated • When finish listening to all the main stakeholder groups and understand their perspectives • How can I adapt ethnography to fit the development process • Preparation • Field study • Analysis • Reporting

19. Indirect Observation • When direct observation is not possible • Tracking Users’ Activities without intruding • Diaries • Interaction Logging

20. Diaries • Track what users did, when they did it , and what they thought about their interactions with the technology • Useful when users are scattered and unreachable • internet and web evaluations • Templates can also be created online to standardize entry format

21. Advantages Inexpensive No special equipment or expertise needed Suitable for long-term studies Disadvantages Rely on participants being reliable and remembering to complete the diaries Often incentives are needed and the process has to be straightforward and quick Diaries

22. Interaction Logging • For example, you want to track how long people stayed at a site, which areas they visited, where they came from, and where they went next • Web page (visitors) counter • Server logs • can also raises ethical concerns that need careful consideration

23. Advantages Unobtrusive Large volumes of data can be logged automatically Disadvantages Ethical concerns Powerful tools are needed to analyze the logs (WebLog) Interaction Logging

24. Data Collection • Notes plus still camera • Audio plus still camera • Video

25. Advantages Unobtrusive Less technical and flexible inexpensive Easy to transcribe Disadvantages Writing speed limited Difficult to write and observe at the same time Rely on the note-takers Notes + Camera

26. Advantages less obtrusive than video Less expensive Flexible Good for recording “thinking aloud” activities Disadvantages Lack of visual record (user actions missing) Difficult to transcribe the data Have to change tapes and position microphone Audio + Camera

27. Advantages Capture both visual and audio High reliability Permanent original record can be revisited Disadvantages More expensive Can be intrusive Easy to miss other things going on outside of the camera view need more than one camera (subject + screen) Need positioning and focusing camera lens Analysis of video data can be time-consuming Video

28. Dilemma – Observing Without Being Seen • Shall we tell users they are being observed? • Tell: users may react or change their behaviors • Not tell: raising ethical issues • Depends on the context • How much personal information to be collected • How the information will be used

29. Analyzing, Interpreting, and Presenting the Data • Qualitative analysis to tell “the story” • interpret what was observed • Qualitative analysis for categorization • content analysis • Quantitative data analysis

30. Qualitative Analysis to Tell a Story • Review the data – identify key themes, make collections • Record the themes – with examples • Record the date and time – for each session • Check your understanding – with people you observed • Iterate this process • Report your findings – to development team

31. Qualitative Analysis for Categorization • Looking for incidents or patterns • Analyzing data into categories • Analyzing discourse

32. Looking For Incidents Or Patterns • Look for critical incidents, such as times when users were obviously stuck • comments, silence, or looks of puzzlement • Theory may also be used to guide the study • help to focus on relevant incidents • Use Tools to record, manipulate and search the data • NIDIST • Observer video-pro tool

33. Analyzing Data Into Categories • Content analysis – fine grain way of analyzing video data • determine a meaningful set of mutually exclusive categories • decide on the appropriate granularity • train a second person, both will analyze • labor-intensive and time consuming

34. Analyzing Discourse • Focus on the dialog – meaning of what is said, rather than the content • Strongly interpretive, pay great attention to context • E.g. Analyzing discourse on the internet (Chat rooms, bulletin boards) has started to influence designers’ understanding about users’ need in these environments • how to start a conversation • how turn-taking is structured

35. Quantitative Data Analysis • Data from video and interaction logs are annotated • Typically this data is analyzed and treated statistically • Usually presented as values, tables, charts and graphs

36. Summary of data analysis • Flag events in real time and examine them in more detail later • Fine-grained analyses can be very time-consuming • Identifying key events is an effective approach

37. Feeding the Findings Back Into design • Written reports (overview + details) • Verbal presentation • Companied by anecdotes, quotations, pictures and video clips • Both qualitative and quantitative analysis useful • quantitative analysis is used less often and depends on goal