Observing users
Download
1 / 73

Observing users - PowerPoint PPT Presentation


  • 276 Views
  • Uploaded on

Observing users. Presenter : JungMin shin & JaeYoung Lee. Contents. Goals, questions, paradigms How to observe Data Collection Indirect Observation: tracking user’s activities Analyzing, interpreting, and presenting data Field studies. Goals, questions, paradigms.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Observing users' - ciara


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Observing users l.jpg

Observing users

Presenter : JungMin shin & JaeYoung Lee


Contents l.jpg
Contents

  • Goals, questions, paradigms

  • How to observe

  • Data Collection

  • Indirect Observation: tracking user’s activities

  • Analyzing, interpreting, and presenting data

  • Field studies


Goals questions paradigms l.jpg
Goals, questions, paradigms

  • What and when to observe

  • Approached to observation


Goals questions paradigms4 l.jpg
Goals, questions, paradigms

Goal and question provide a focus for observation,

DECIDE framework.

Having goal helps to guide the observation because

there is always so much going on.

  • What and when to observe

    On developing product

    starting time : helps understand user’s needs.

    ending time : examine to satisfy user’sneeds.


Goals questions paradigms5 l.jpg
Goals, questions, paradigms

  • Role of evaluator

    Onlookers, participant observers, ethnographers

    Table 12.1 Type of observation


Goals questions paradigms6 l.jpg
Goals, questions, paradigms

  • Approaches to observation

    1. “Quick and dirty” observation

    Watching and talking to users in a casual way to

    get immediate feedback.

    2. Observation in usability testing

    Video and interacting logs capture everything.

    Can watch through a one-way mirror or via TV

    screen.

    Observational data is used to see and analyze for

    user’s action.


Goals questions paradigms7 l.jpg
Goals, questions, paradigms

  • Observation in usability testing System


Goals questions paradigms8 l.jpg
Goals, questions, paradigms

3. Observation in field studies

Observers may be anywhere along the outsider –

insider spectrum.

Colin Robson(1993) – participating level

complete participants

more marginal participants

observers who also participate

people who observefrom the outside and do not participate.

The goal is to cause as little disruption as possible.


Goals questions paradigms9 l.jpg
Goals, questions, paradigms

3. Observation in field studies (Continue)

ex)

A study of the time spent by boy and girl using technology in the classroom.

-> observe and note at the back of room.

A study of understanding how the computer integrates with other artifacts and social interacting.

-> might take of insider perspective.


Goals questions paradigms10 l.jpg
Goals, questions, paradigms

3. Observation in field studies (Continue)

participant observation evaluator

participate with users in order to learn what they do and how and why they do it.

fully participant observer

a member of group.

being present to share experience and

learning the social conventions of the

group.


Goals questions paradigms11 l.jpg
Goals, questions, paradigms

Determining goals , exploring questions Choosing

techniques are necessary steps in the DECIDE

framework.


How to observe l.jpg
How to observe

  • In controlled environments

  • In the field environments

  • Participant observation and ethnography


How to observe13 l.jpg
How to observe

  • In controlled environments

    Collect and then Make sense of thedata while

    watching users in controlled environment.

    Practical issues in controlled environment

    - where users will be located

    - equipment test

    - An informed consent form


How to observe14 l.jpg
How to observe

  • In controlled environments

    the observer doesn’t know what users are thinking, and can only guess from what she sees.

    Think-aloud technique by Erikson and Simon

    Require people to say out loud everything that they are thinking and trying to do.

    ex) I’m typing in http://ics.uci.edu (type)

    Now I press the enter key, right? (presses enter key)

    But if users meet some errors, they could be silent again...

    Sol) Evaluator reminds him think out loud, but that would be intrusive.

    -> two people work together so that they talk to each other.


How to observe15 l.jpg
How to observe

  • In the field environments

    Events in the field can be complex and rapidly changing.

    Framework

    help observers to keep their goals and questions in sight.

    Collin Robson(1993)

    Space, Actors, Activities, Objects, Acts, Events, Goals, Feelings


How to observe16 l.jpg
How to observe

  • In the field environments

    • Checklist of things to plan before going into field

      1. State the initial study goal and question

      2. Select the framework

      3. Decide how to record events

      4. Be prepared to go through your notes and other records

      5. Make and review your notes

      6. Be prepared refocus your study

      7. Think about how you will get the acceptance and trust of

      those you observe

      8. Think about how to handle sensitive issues (privacy)

      9. Consider working as a team

      10. Consider checking your notes with an informant or members

      of the group

      11. Plan to look at the situation from different perspectives


How to observe17 l.jpg
How to observe

  • Participant observation and ethnography

    Checklist for doing ethnography

    • Identifying a problems or goal and then ask good questions to be answered by the study

    • The most important part of fieldwork just being there to observe, ask questions, and record.

    • Collect a variety of data (Interview, retrospectives interview)

    • Be prepared to move backwards and forwards between the broad picture and specific questions

    • Analyze the data using a holistic approach -> contextualized, usually iterative


How to observe18 l.jpg
How to observe

  • Participant observation and ethnography

    <Dilemma>

    When have I observed enough?

    schedules often dictate when your study end.

    stop when you stop learning new things

    when you start to see similar pattern of behavior being repeated.

    when you have listened to all the main stake holder group and understood their perspectives.


How to observe19 l.jpg
How to observe

  • Participant observation and ethnography

    How can I adapt ethnography to fit the development

    process?

    Ann Rosa Procedure

    Preparing (Understand/Familiarize/Set Goal /Gain

    permission)

    Field study (Establish/Observe and interview/

    Follow any lead/ Record)

    Analysis (Compile/Quantify/Reduce and interpret/

    Refine the goals)

    Reporting (Consider/Prepare)


Data collection l.jpg
Data collection

  • Notes plus still camera

  • Audio recording plus still camera

  • Video


Data collection21 l.jpg
Data collection

  • Notes plus still camera

    being difficult and tiring to write and observe at the same time

    bored, the limitation of writing speed

    (sol) working with another person

    (disadvantage) have to be transcribed

  • Audio recording plus still camera

    Useful alternative to note taking and is less intrusive than

    Video.

    Drawback: Lack of visual record, transcribing the data

    So, evaluators use the recording to remind them about

    important details and as a source of anecdotes for reports.


Data collection22 l.jpg
Data collection

  • Video

    Advantage

    Capturing both Audio and Video Data

    Disadvantage

    Intrusive.

    Attention becomes focused on what is seen through the lens.

    Time-consuming for analyzing video data.

    ex) 1 hour video recording : over 100 hours of

    analysis time


Indirect observation tracking user s activities l.jpg
Indirect Observation: tracking user’s activities

  • Diaries

  • Interaction logs


Indirect observation tracking user s activities24 l.jpg
Indirect Observation: tracking user’s activities

When ? Impossible to observe directly

  • Diaries

    A record of what users did, when did it, what thought about their

    interactions with the Technology.

    (Advantage)

    When User scattered and unreachable person.

    Inexpensive - no equipment or expertise.

    Suitable for long term studies.

    Template and Enable the data to go straight into database.

    Ex) open-ended online questionnaires

    (Disadvantage)

    Rely on participant (reliable/remembering).

    Participant remember better or worse that they really were.


Indirect observation tracking user s activities25 l.jpg
Indirect Observation: tracking user’s activities

  • Interaction logs

    key presses, mouse or other device movement are recorded usually

    synchronized with A/V log.

    Has time-stamped (to calculate how long)

    Explicit counter / Recording number of visitors

    EX) In study of interact art museum (Researchers at USC)

    Server log (7 months/ analysis tool : webtrends)

    when/what /how long/what browser/what country…

    (Advantage)

    Unobtrusive (but this also raises ethical concerns)

    Large volumes of data.


Analyzing interpreting and presenting data l.jpg
Analyzing, interpreting, and presenting data

The first things to do is to “eyeball”.

Proceed to analyze it according to the goals and

questions.

  • Qualitative analysis to tell a story

  • Qualitative analysis for categorization

  • Quantitative data analysis

  • Feeding the findings back into design


Analyzing interpreting and presenting data27 l.jpg
Analyzing, interpreting, and presenting data

  • Qualitative analysis to tell a story

    Power of analyzing descriptive data -> being able

    to tell a convincing story

    • Interview with Sara Bly (User-Centered Design Consultant)

      At the end of each observation period

      Review their data

      Discuss what they observed

      Construct a story from their data


Analyzing interpreting and presenting data28 l.jpg
Analyzing, interpreting, and presenting data

  • Qualitative analysis to tell a story

    To summarize about Interview with Sara Bly.

    Review the data.

    Record the themes.

    Record the date and time of each data analysis

    Session.

    As themes emerge.

    Iterate this process (until your story faithfully

    represents).

    Report your findings to the development team.


Analyzing interpreting and presenting data29 l.jpg
Analyzing, interpreting, and presenting data

  • Qualitative analysis to tell a story

    • Analyzing and reporting ethnographic data

      - ethnographers reconstruct knowledge to produce detailed description known as rich or thick description.

      Main activities (Fetterman-1998)

      • Look for key events

      • Look for patterns of behavior in various situations and among different players

      • Compare sources

      • Report your findings in convincing and honest way

  • S/W tools : NUDIST and Ethnograph


Analyzing interpreting and presenting data30 l.jpg
Analyzing, interpreting, and presenting data

  • Qualitative analysis for categorization

    Looking for incident or patterns

    Common strategy is to look for critical incidents

    ex) Jurgen Koenemann-Belliveau el al(1994)

    used this form of critical incident analysis to examine

    breakdown or problems in achieving a programming task

    and also to identify possible threats of incidents.

    Using the theory helped the evaluators to focus in relevant

    Incidents.


Analyzing interpreting and presenting data31 l.jpg
Analyzing, interpreting, and presenting data

  • Qualitative analysis for categorization

    analyzing data into categories

    contents analysis provides another fine grain way of analyzing video data.

    determined by the evaluation question.

    must also be reliable so that the analysis can be replicated.

    EX)

    training second person and then analyze the same data sample.

    if there are large discrepancy between each other. -> what?

    Inter-research reliability rating

    -> the percentage of agreement between the two researchers.


Analyzing interpreting and presenting data32 l.jpg
Analyzing, interpreting, and presenting data

  • Analyzing discourse

    Dialog

    Strongly interpretive, pays great attention to context.

    Language is viewed as a constructive tool and discourse

    analysis provides a way of focusing upon how people use

    language to construct versions of their worlds (Fiske,1994)

    Small changes in wording can change meaning, as the

    following excerpts indicate (Coyle, 1995)

    (According to Coyle,) discourse analysis is what you do when

    you are saying that you are doing discourse analysis…


Analyzing interpreting and presenting data33 l.jpg
Analyzing, interpreting, andpresenting data

  • Quantitative data analysis

    In case of errors or unusual behavior ->

    marks the video and records a brief remark.

    To use this annotated recording

    1. Calculate performance time

    2. further analyzed using simple statistics such as means, standard deviations, T-tests, etc.


Analyzing interpreting and presenting data34 l.jpg

Feeding the findings back into design

The results can be reported to the design team.

Written reports with a overview and detailed

content list.

Quantitative data -> its value depends on the type

of study and its goals.

Often both quantitative data analysis are useful

because of they provide alternative perspectives.

Analyzing, interpreting, and presenting data



Field study36 l.jpg
Field Study

  • A broad range of data gathering techniques at the user’s location

  • There is no one best way to conduct a field study. It depends on the goals of your study and your access to users.


The goal of a field study l.jpg
The Goal of a Field Study

  • Field studies are excellent for collecting rich, detailed data and obtaining a holistic view of the process or domain.

  • Field studies are not a good technique if you want to collect quantitative data from large samples.

  • The goal of a field study is to make the implicit aspects and processes of the user environment explicit.


Field study methods l.jpg
Field Study Methods

  • Techniques are divided into three categories:

    • Observation only

    • Interacting with the user

    • Method supplements

  • The Goal of the Field Study Methods

    • To observe users and collect information about their tasks and the context in which they are done.


Field study methods39 l.jpg
Field Study Methods

  • Observation Only

    • Pure Observation

    • Deep Hanging-Out

  • Interacting with the user

    • Contextual Inquiry

    • Process analysis

    • Condensed ethnographic interview

    • Discount User Observation (DUO)

  • Method supplements

    • Artifact walkthroughs

    • Incident diaries

    • Observing while you are not present


Field study methods40 l.jpg
Field Study Methods

  • Observation Only

    • Pure Observation

    • Deep Hanging-Out


Observation only pure observation l.jpg
Observation only - Pure Observation

  • Synopsis

    • When you are unable or don’t wish to interact with the user, you simply observe from a distance.

  • Advantages

    • Flexible

    • Low resources

  • Level of effort

    • Minimal, Place your self in a good vantage point and observe as many user/sites/tasks as you feel appropriate

    • You continue to conduct observations until you feel you have a good understanding of the domain of areas of focus.


Observation only pure observation42 l.jpg
Observation only - Pure Observation

  • Valuable in situations where you cannot interact with the end users.

  • Users may or may not know they are being studied.

  • Do not interact with the participant

  • Do not distribute surveys, interview the user, or ask for artifacts from the user.

  • It is essential to have a good sampling plan. The sampling plan should include specific information such as days/times you anticipate key events (e.g., the day before Thanksgiving, or bad weather at an airport), as well as “normal” days.


Field study methods43 l.jpg
Field Study Methods

  • Observation Only

    • Pure Observation

    • Deep Hanging-Out


Observation only deep hanging out l.jpg
Observation only – Deep Hanging out

  • Synopsis

    • This method is similar to pure observation but provides more structure by suggesting focus areas and things to observe.

  • Advantages

    • It has more structured than pure observation so you can do a more detailed level of data analysis and compare data collected across multiple sites.

  • Level of effort

    • Because there is more structure, it takes more effort than pure observation.

    • You are “on” at all times, which can be tiring.

    • It is also valuable to become a user yourself (if possible) and collect artifacts along the way.


Observation only deep hanging out45 l.jpg
Observation only – Deep Hanging out

  • A more structured form of pure observation.

  • Involves significant amounts of observation along with involving yourself in the process

  • Method of Deep Hanging-Out includes

    • Structured observation

    • Collection of artifacts

    • Becoming a user yourself

  • However, you do notinterview participants, distribute surveys, or present design ideas for feed back.


Field study methods46 l.jpg
Field Study Methods

  • Observation Only

    • Pure Observation

    • Deep Hanging-Out

  • Interacting with the user

    • Contextual Inquiry

    • Process analysis

    • Condensed ethnographic interview

    • Discount User Observation (DUO)

  • Method supplements

    • Artifact walkthroughs

    • Incident diaries

    • Observing while you are not present


Field study methods47 l.jpg
Field Study Methods

  • Interacting with the user

    • Contextual Inquiry

    • Process analysis

    • Condensed ethnographic interview

    • Discount User Observation (DUO)


Interacting with the user contextual inquiry l.jpg
Interacting with the user - Contextual Inquiry

  • Synopsis

    • Interview, apprentice with, and interpret the resulting data with users

  • Advantages

    • Contextual Inquiry is more focused and context dependent than the other methods

    • At the end, you walk away with actionable items

  • Level of effort

    • The effort level is higher than for observation-only techniques. You must develop an observation guide, observe users, apprentice with them, and discuss your observations with them


Interacting with the user contextual inquiry49 l.jpg
Interacting with the user - Contextual Inquiry

  • Four main parts to Contextual Inquiry:

    • Context

    • Partnership

    • Interpretation

    • Focus


Interacting with the user contextual inquiry50 l.jpg
Interacting with the user - Contextual Inquiry

  • Four main parts to Contextual Inquiry:

    • Context : You must go to the user’s environment in order to understand the context of his/her actions. Contextual Inquiry assumes that observation alone or out-of-context interviews are insufficient


Interacting with the user contextual inquiry51 l.jpg
Interacting with the user - Contextual Inquiry

  • Four main parts to Contextual Inquiry:

    • Partnership : To better understand the user, tasks, and environment, you should develop a master-prentice relationship with the participant. Immerse your self in the participant’s work and do as he/she does. Obviously, this is not possible with many jobs (e.g., surgeon, fighter pilot)


Interacting with the user contextual inquiry52 l.jpg
Interacting with the user - Contextual Inquiry

  • Four main parts to Contextual Inquiry:

    • Interpretation : Observations must be interpreted with the participant in order to be used later. Verify that your assumptions and conclusions are correct.


Interacting with the user contextual inquiry53 l.jpg
Interacting with the user - Contextual Inquiry

  • Four main parts to Contextual Inquiry:

    • Focus : Develop an observation guide to keep you focused on the subject of interest/inquiry.


Field study methods54 l.jpg
Field Study Methods

  • Interacting with the user

    • Contextual Inquiry

    • Process analysis

    • Condensed ethnographic interview

    • Discount User Observation (DUO)


Interacting with the user process analysis l.jpg
Interacting with the user – Process Analysis

  • Synopsis

    • Capture the task sequence for a process that may span several days

  • Advantages

    • Because it is more focused than Contextual Inquiry, it is also much faster

  • Level of effort

    • You need to stay focused on the process at hand to help users walk you through the process of interest. It may span just several days.


Interacting with the user process analysis56 l.jpg
Interacting with the user – Process Analysis

  • A process analysis is similar to Contextual Inquiry. However, you enter with a series of questions and you do not necessarily apprentice with the user

  • Questions

    • When does the first task in the process happen?

    • What triggers it?

    • Who does it?

    • What information does the person have when the task begins?

    • What are the major steps in the task?

    • What information comes out of it?

    • Who is the next person in the chain of the process?


Interacting with the user process analysis57 l.jpg
Interacting with the user – Process Analysis

  • Example - Process map for a travel agent’s typical day


Field study methods58 l.jpg
Field Study Methods

  • Interacting with the user

    • Contextual Inquiry

    • Process analysis

    • Condensed ethnographic interview

    • Discount User Observation (DUO)


Interacting with the user condensed ethnographic interview l.jpg
Interacting with the user – Condensed Ethnographic Interview

  • Synopsis

    • Use the results of semi-structured interviews to guide observations

  • Advantages

    • This technique is considerably shorter than some of the other techniques, because the interviews scope what you observe;

    • It also limitsthe data you are able to collect

  • Level of effort

    • It requires a medium level of effort to conduct interviews and then observe users and collect artifacts


Interacting with the user condensed ethnographic interview60 l.jpg
Interacting with the user – Condensed Ethnographic Interview

  • Employs the standardization and focus of a semi-structured interview along with the context of observations and artifacts

  • Characterized as “top-down”, because the interviews form a general framework from which to interpret specific observations.


Interacting with the user discount user observation duo l.jpg
Interacting with the user – Discount User Observation (DUO)

  • Synopsis

    • One facilitator interviews the user and takes notes while another facilitator takes pictures of everything

  • Advantages

    • You can quickly reconstructthe session through time stamped notes and digital pictures to verify your observations with users

  • Level of effort

    • It takes two investigators to focus on their specific tasks of either interviewing or taking pictures and notes

    • It will take additional time/effort to combine notes with digital photos and put them in a multimedia presentation tools to review with users.

    • It takes additional effort to validate observations with user afterward.


Field study methods62 l.jpg
Field Study Methods

  • Observation Only

    • Pure Observation

    • Deep Hanging-Out

  • Interacting with the user

    • Contextual Inquiry

    • Process analysis

    • Condensed ethnographic interview

    • Discount User Observation (DUO)

  • Method supplements

    • Artifact walkthroughs

    • Incident diaries

    • Observing while you are not present


Field study methods63 l.jpg
Field Study Methods

  • Method supplements

    • Artifact walkthroughs

    • Incident diaries

    • Observing while you are not present


Method supplements artifact walkthroughs l.jpg
Method supplements – Artifact walkthroughs

  • Synopsis

    • Collect all the artifacts used by participants and determine what triggers their use, when they are used, and for what

  • Advantages

    • Quick and easy to conduct

  • Level of effort

    • Low level of effort to review artifacts with participants and make/collect copies of them


Method supplements artifact walkthroughs65 l.jpg
Method supplements – Artifact walkthroughs

  • Quick and easy to conduct but provide indispensable data

  • “Artifacts” are objects or items that users use to complete their tasks or that result from their tasks

  • You have to sign the company’s confidential disclosure agreement promising that you will keep all data collected confidential


Field study methods66 l.jpg
Field Study Methods

  • Method supplements

    • Artifact walkthroughs

    • Incident diaries

    • Observing while you are not present


Method supplements incident diaries l.jpg
Method supplements – Incident diaries

  • Synopsis

    • Worksheets the user takes home or to work to collect ongoing data

  • Advantages

    • No observation is required

    • Able to understand more issues than what can be observed in the lab or during a single visit

  • Level of effort

    • Low effort levelto create and distribute diaries

    • Moderate effortto analyze data across multiple diaries. There will be a time lag between when you distribute the diaries and when you receive the data

    • You are depending on the participants to follow-thorough.


Method supplements incident diaries68 l.jpg
Method supplements – Incident diaries

  • Incident diaries allow you to collect ongoing data from users

  • Participants are provided with a notebook containing worksheets to be completed

  • They are given to users to keep track of issues they encounter while using a product

  • The worksheets may ask users

    • Describe a problem or issue they encountered

    • How they solved it

    • How troublesome it was


Field study methods69 l.jpg
Field Study Methods

  • Method supplements

    • Artifact walkthroughs

    • Incident diaries

    • Observing while you are not present


Method supplements observing while you are not present l.jpg
Method supplements – Observing while you are not present

  • Synopsis

    • Recording users in action when space, time, or restrictions prevent you from being there in person

  • Advantages

    • If you have multiple video cameras, you can view several users simultaneously

  • Level of effort

    • Low effort to set up cameras and record

    • Moderate effort to meet with user again and review tapes

    • Moderate effort to categorize and index behavior


Method supplements observing while you are not present71 l.jpg
Method supplements – Observing while you are not present

  • You can observe users even when you are not present by setting up a video camera and then leaving

  • Excellent way to understand detailed steps




ad