Observing users. Presenter : JungMin shin & JaeYoung Lee. Contents. Goals, questions, paradigms How to observe Data Collection Indirect Observation: tracking user’s activities Analyzing, interpreting, and presenting data Field studies. Goals, questions, paradigms.
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Presenter : JungMin shin & JaeYoung Lee
Goal and question provide a focus for observation,
Having goal helps to guide the observation because
there is always so much going on.
On developing product
starting time : helps understand user’s needs.
ending time : examine to satisfy user’sneeds.
Onlookers, participant observers, ethnographers
Table 12.1 Type of observation
1. “Quick and dirty” observation
Watching and talking to users in a casual way to
get immediate feedback.
2. Observation in usability testing
Video and interacting logs capture everything.
Can watch through a one-way mirror or via TV
Observational data is used to see and analyze for
3. Observation in field studies
Observers may be anywhere along the outsider –
Colin Robson(1993) – participating level
more marginal participants
observers who also participate
people who observefrom the outside and do not participate.
The goal is to cause as little disruption as possible.
3. Observation in field studies (Continue)
A study of the time spent by boy and girl using technology in the classroom.
-> observe and note at the back of room.
A study of understanding how the computer integrates with other artifacts and social interacting.
-> might take of insider perspective.
3. Observation in field studies (Continue)
participant observation evaluator
participate with users in order to learn what they do and how and why they do it.
fully participant observer
a member of group.
being present to share experience and
learning the social conventions of the
Determining goals , exploring questions Choosing
techniques are necessary steps in the DECIDE
Collect and then Make sense of thedata while
watching users in controlled environment.
Practical issues in controlled environment
- where users will be located
- equipment test
- An informed consent form
the observer doesn’t know what users are thinking, and can only guess from what she sees.
Think-aloud technique by Erikson and Simon
Require people to say out loud everything that they are thinking and trying to do.
ex) I’m typing in http://ics.uci.edu (type)
Now I press the enter key, right? (presses enter key)
But if users meet some errors, they could be silent again...
Sol) Evaluator reminds him think out loud, but that would be intrusive.
-> two people work together so that they talk to each other.
Events in the field can be complex and rapidly changing.
help observers to keep their goals and questions in sight.
Space, Actors, Activities, Objects, Acts, Events, Goals, Feelings
1. State the initial study goal and question
2. Select the framework
3. Decide how to record events
4. Be prepared to go through your notes and other records
5. Make and review your notes
6. Be prepared refocus your study
7. Think about how you will get the acceptance and trust of
those you observe
8. Think about how to handle sensitive issues (privacy)
9. Consider working as a team
10. Consider checking your notes with an informant or members
of the group
11. Plan to look at the situation from different perspectives
Checklist for doing ethnography
When have I observed enough?
schedules often dictate when your study end.
stop when you stop learning new things
when you start to see similar pattern of behavior being repeated.
when you have listened to all the main stake holder group and understood their perspectives.
How can I adapt ethnography to fit the development
Ann Rosa Procedure
Preparing (Understand/Familiarize/Set Goal /Gain
Field study (Establish/Observe and interview/
Follow any lead/ Record)
Analysis (Compile/Quantify/Reduce and interpret/
Refine the goals)
being difficult and tiring to write and observe at the same time
bored, the limitation of writing speed
(sol) working with another person
(disadvantage) have to be transcribed
Useful alternative to note taking and is less intrusive than
Drawback: Lack of visual record, transcribing the data
So, evaluators use the recording to remind them about
important details and as a source of anecdotes for reports.
Capturing both Audio and Video Data
Attention becomes focused on what is seen through the lens.
Time-consuming for analyzing video data.
ex) 1 hour video recording : over 100 hours of
When ? Impossible to observe directly
A record of what users did, when did it, what thought about their
interactions with the Technology.
When User scattered and unreachable person.
Inexpensive - no equipment or expertise.
Suitable for long term studies.
Template and Enable the data to go straight into database.
Ex) open-ended online questionnaires
Rely on participant (reliable/remembering).
Participant remember better or worse that they really were.
key presses, mouse or other device movement are recorded usually
synchronized with A/V log.
Has time-stamped (to calculate how long)
Explicit counter / Recording number of visitors
EX) In study of interact art museum (Researchers at USC)
Server log (7 months/ analysis tool : webtrends)
when/what /how long/what browser/what country…
Unobtrusive (but this also raises ethical concerns)
Large volumes of data.
The first things to do is to “eyeball”.
Proceed to analyze it according to the goals and
Power of analyzing descriptive data -> being able
to tell a convincing story
At the end of each observation period
Review their data
Discuss what they observed
Construct a story from their data
To summarize about Interview with Sara Bly.
Review the data.
Record the themes.
Record the date and time of each data analysis
As themes emerge.
Iterate this process (until your story faithfully
Report your findings to the development team.
- ethnographers reconstruct knowledge to produce detailed description known as rich or thick description.
Main activities (Fetterman-1998)
Looking for incident or patterns
Common strategy is to look for critical incidents
ex) Jurgen Koenemann-Belliveau el al(1994)
used this form of critical incident analysis to examine
breakdown or problems in achieving a programming task
and also to identify possible threats of incidents.
Using the theory helped the evaluators to focus in relevant
analyzing data into categories
contents analysis provides another fine grain way of analyzing video data.
determined by the evaluation question.
must also be reliable so that the analysis can be replicated.
training second person and then analyze the same data sample.
if there are large discrepancy between each other. -> what?
Inter-research reliability rating
-> the percentage of agreement between the two researchers.
Strongly interpretive, pays great attention to context.
Language is viewed as a constructive tool and discourse
analysis provides a way of focusing upon how people use
language to construct versions of their worlds (Fiske,1994)
Small changes in wording can change meaning, as the
following excerpts indicate (Coyle, 1995)
(According to Coyle,) discourse analysis is what you do when
you are saying that you are doing discourse analysis…
In case of errors or unusual behavior ->
marks the video and records a brief remark.
To use this annotated recording
1. Calculate performance time
2. further analyzed using simple statistics such as means, standard deviations, T-tests, etc.
The results can be reported to the design team.
Written reports with a overview and detailed
Quantitative data -> its value depends on the type
of study and its goals.
Often both quantitative data analysis are useful
because of they provide alternative perspectives.Analyzing, interpreting, and presenting data