pervasive computing user studies l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Pervasive Computing User Studies PowerPoint Presentation
Download Presentation
Pervasive Computing User Studies

Loading in 2 Seconds...

play fullscreen
1 / 51

Pervasive Computing User Studies - PowerPoint PPT Presentation


  • 128 Views
  • Uploaded on

Pervasive Computing User Studies . A.J. Bernheim Brush. Who am I?. Ph.D. in Computer Science Researcher at Microsoft Research Technology for families, workgroups (HCI/CSCW/ Ubicomp ) “Love” studies. Why do a user study?. Bad Reasons You don’t have anything else to do…

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Pervasive Computing User Studies' - christmas


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
who am i
Who am I?
  • Ph.D. in Computer Science
  • Researcher at Microsoft Research
  • Technology for families, workgroups (HCI/CSCW/Ubicomp)
  • “Love” studies
why do a user study
Why do a user study?
  • Bad Reasons
    • You don’t have anything else to do…
    • You think it’s a requirement to get your paper accepted
    • It might be fun to see how people use your stuff
  • Good Reasons
    • You’re designing stuff for people to use. Wouldn’t it be nice to know how they might use it?
    • There is a new domain or behavior you want to observe
the reality
The Reality….
  • User Studies are a lot of work
    • Really, more work than you ever expected
    • No, really I’m not kidding
  • Understanding your GOALis critical
    • Studying current behavior: What are people doing now?
    • Proof of concept: Does my novel technology work for people?
    • Experience using a prototype: How does using my prototype change people’s behavior or allow them to do new things

WARNING: INTERACTIVE EVENT AT THE ENDSTART THINKING ABOUT ONE OF YOUR OWN PROJECTS

outline
Outline
  • Introduction
  • Types of Studies
  • Study Design
  • Example
  • Ten Mistakes to Avoid
  • Your Turn
many types of studies
Many types of studies

Formative/Current Behavior

Prototyping/Proof of Concept

Summative/

Usage Experience

Ethnography

Survey

Interviews

Focus Groups

Logging

“Discount Usability”

Heuristic Evaluations

Lab studies

Lab studies

Field Studies

Logging

Types

Learn about a domain

Design inspiration

Does it work?

Can people use it?

How does it compare to other designs/prototypes?

Iterative Testing

Fix & understand your prototype

Goal

many types of studies7
Many types of studies

Formative/Current Behavior

Prototyping/Proof of Concept

Summative/

Usage Experience

Ethnography

Survey

Interviews

Focus Groups

Logging

“Discount Usability”

Heuristic Evaluations

Lab studies

Lab studies

Field Studies

Logging

Types

Learn about a domain

Design inspiration

Does it work?

Can people use it?

How does it compare to other designs/prototypes?

Iterative Testing

Fix & understand your prototype

Goal

surveys
Surveys
  • Easy to get large number of people
  • Design guidance
  • Evaluation of deployed system
  • Surprisingly hard to do well….
    • Phrasing of questions
    • Biased responses
  • Pilot your survey!
discount usability jakob nielsen
“Discount Usability” (Jakob Nielsen)

If you are building prototype very useful to get feedback from users early and often

  • Low-cost, Quick, Iterative, Small N, Identify big problems
  • Lo-fi prototypes
    • Paper version can be very helpful
    • People feel ok telling you to change stuff
    • No feedback on responsiveness etc.
  • Heuristic evaluation
    • Experts review the interface based on list of heuristics
  • Cognitive walk-through
    • Determine tasks, review and ask questions for each task

WARNING: Not typically a research contribution

lab studies
Lab Studies
  • Bring participants into a lab
    • Minimize variability
  • Hypothesis testing
  • Independent variables between conditions
    • Interface A vs. Interface B
    • Control condition?
  • Measure dependent variable
    • Speed of use, … (Quantitative)
    • Preference, …. (Qualitative)

http://www2.sta.uwi.edu/usability/facilities.htm

between vs within subjects designs
Between vs. Within Subjects Designs
  • Between subjects
    • Participants each in different condition
      • E.g., everyone randomly assigned to a group
      • ANOVA, t-test
    • Within subjects
    • Each participants experiences all conditions
      • E.g., expertise, practice or individual differences accounted for
      • RM ANOVA, paired t-test
    • Have to worry about repetition and carryover effects
      • Counterbalancing, Latin square, etc.

I PREFER DUE TO PARTICIPANT VARIABILITY

statistics
Statistics
  • Descriptive Statistics (count, mean, …)
    • Nominal – categories: frequency only
    • Ordinal – Ranked preference: frequency, median
    • Interval – numbers: frequency, sum, median, mean..
      • Watch out for outliers
  • Inferential statistics
    • Are results statistically significant between groups?
    • T-test, ANOVA, paired t-test, etc.
    • Significance values
      • If p < 0.01, there is a 99% chance that the data collected represents a real difference in the population rather than a sampling error
f ield study
Field study
  • In-situ (“not on your turf”)
  • Trading “control” for realism
    • Think carefully if this is important
  • All types
    • current behavior,
    • proof-of-concept
    • prototype

WARNING: Often good idea to do lab study before field study

outline14
Outline
  • Introduction
  • Types of Studies
  • Study Design
  • Example
  • Ten Mistakes to Avoid
  • Your Turn
how do i choose
How do I choose?
  • You might not…
  • What is your goal/research contribution?

LINC: An Inkable Digital Calendar

research question goal
Research Question/Goal

Your Research Question is critical

Bad: How will families use SPARCS?

Better: Do sharing suggestions promote sharing?

There are very few right decisions, instead decisions you need to justify

study design
Study Design
  • What type of study?
  • What will your participants do during the study?
    • Give them hardware? Give them tasks?
  • Why type of participants you should recruit?
  • What data will you collect?
  • How long will the study be?
  • Where can you skimp during the study….
  • What absolutely has to work (if it’s a prototype)
human subjects
Human Subjects
  • Ethical treatment of people in the study
    • Respect—remember that they are doing YOU a favor
    • Participants can stop at any time
  • Consent Forms
  • Privacy Statements
  • Compensation
  • Your organization should have some review process
  • THIS IS IMPORTANT!
    • What will they will be doing during the study
    • How will you report on what they did
  • http://www.hhs.gov/ohrp/irb/irb_chapter3.htm
pilot
PILOT!!!!
  • Allow time to a pilot study,
  • Run through the entire methodology with volunteer participants.
    • Uncover system problems
    • Uncover experimental design problems
    • Uncover problems with materials

IMPORTANT IMPORTANTIMPORTANT

participant profile
Participant Profile
  • What types people do you want to participate?
    • All the same?
participant profile21
Participant Profile
  • What types people do you want to participate?
    • All the same?
    • All different?
participant profile22
Participant Profile
  • What types people do you want to participate?
    • All the same?
    • All different?
    • People with extreme characteristics vs. “normal” people
participant profile23
Participant Profile
  • What types people do you want to participate?
    • All the same?
    • All different?
    • People with extreme characteristics vs. “normal” people
  • Consider
    • Age
    • Gender
    • Technology experience
    • ….
how many participants
How many participants?
  • This can be a difficult question
  • Between or within subjects
  • What claims you are making
  • What is feasible
  • Some will drop out!
length
Length
  • How long should the study be?
  • Another difficult question….
    • Novelty
    • What are you asking participant to do?
    • Long term use vs. feasibility
data collection
Data Collection
  • How will you understand if you answered your research question?
  • Quantitative Data (logs, timing, # errors, …)
  • Qualitative Data (interviews, surveys, …)
  • TRIANGULATE between multiple sources
logging
Logging
  • You must have a plan going in about how you will use the log data
    • Risk of forgetting to log something important
    • Logging too much can create an analysis nightmare
  • Make a list of questions you expect to answer with log data
    • How many times did they upload a photo?
    • How many days did they use your prototype?
qualitative data
Qualitative Data
  • Surveys
    • Pre-survey
    • Post condition
    • Post-survey
  • Experience Sampling Methodology
    • Small set of question
    • Event triggered, random….
  • Diaries
  • Interviews (semi-structured, structured)
  • Observation
analyzing qualitative data
Analyzing Qualitative Data
  • Affinity Diagramming
  • Coding of Comments
  • Inter-rater reliability
what if it doesn t work
What if it doesn’t work?
  • There are many ways a study can fail
    • Technical problems
    • They don’t like it
  • Nobody but you cares about usability problems
  • Brace yourself for this
    • Figure out what has to work and skimp other places
    • Comparison between prototypes
    • Pilot studies
outline32
Outline
  • Introduction
  • Types of Studies
  • Study Design
  • Example
  • Ten Mistakes to Avoid
  • Your Turn
example
Example

CareNetConsolvo, Roessler, SeltonIntel Research Seattle

carenet methodology
CareNet Methodology
  • 4 “care networks” (3F, 1M)
  • 13 people (4 elders and 9 care givers)
  • 2-3 people had CareNet displays
  • 3 weeks
  • Compensation
    • With CareNet - $150
    • Elders $75- $300 (depend on data provided)
  • Sept. – Dec. 2003
carenet data collection
CareNet Data Collection
  • Interviews, before and after (semi-structured, recorded)
  • Questionnaires: half-way, end
  • Photos
  • 3-6 times per day by phone to collect data for display
  • Glowing display
  • People will take your technology apart, even when you tell them not too.
lots of other examples
Lots of other examples…

ButterflyNetYeh, Liao, Klemmer, Guimbretière, Lee, Kakaradov, Stamberger, PaepckeStanford, Maryland

UbiFitConsolvo, McDonald, Toscos, Chen, Froehlich, Harrison, Klasnja, LaMarca, LeGrand, Libby, Smith, LandayIntel Research Seattle

SuperBreakMorris, Brush, Meyers Microsoft Research

Wearable Jersey DisplayPage and MoereUniversity of Sydney

outline37
Outline
  • Introduction
  • Types of Studies
  • Study Design
  • Example
  • Ten Mistakes to Avoid
  • Your Turn
10 not enough people involved
#10 Not enough people involved
  • It takes a village
  • Huge time commitment, 24 hour support for field studies
  • Do
    • Include multiple people
9 not being prepared
#9 Not being prepared
  • Don’t want to realize at the end of the study that you forgot to do something important
  • Do
    • Study design document
      • Research Question
      • Participant Profile
      • Methodology (within/between etc)
      • Timeline
      • Data collection
    • Pilot studies

WARNING: If you do not plan, you plan to fail!

8 not enough time for logistics
#8 Not enough time for logistics
  • Everything takes time…. (more than you think)
  • Recruiting
  • Installation (e.g. 16 people X 2 hours = )
  • Support
  • Do
    • Allow plenty of time for the study
    • Have enough people
7 seeing what you want to see
#7 Seeing what you want to see
  • We all want our prototypes to be popular
  • Do:
    • Think carefully about how you discuss the technology with participants
    • Avoid leading questions
    • Stay close to the data and find multiple support for conclusions
    • Use neutral language, “Tell me more,” “ummm”
    • Don’t get defensive
6 being judgmental
#6 Being judgmental
  • Users are always “right”
  • They may say things that are offensive, objectionable, etc.
  • Do:
    • Leave your opinions at home
    • Collect feedback
    • Use neutral language
5 not monitoring usage
#5 Not monitoring usage
  • Don’t want to find out at the end of the study that people were not using the prototype
  • Do:
    • “phone home” messages
    • Server logs
  • Have a plan about when and how you might intervene
4 not collecting a variety of data
#4 Not collecting a variety of data
  • Hard to understand logs without interviews
  • Hard to know if what someone says in an interview matches their use without logs
  • Do
    • Interviews
    • Surveys
    • Logs
  • Support for your findings in multiple ways gives you (and reviewers) more confidence
3 prototype is not robust enough
#3 Prototype is not robust enough
  • Technical challenges are a distraction
  • You are making a “product”
  • Do
    • Usability studies
    • Pilots with friendly folks
2 making inappropriate claims
#2 Making inappropriate claims
  • One of the biggest and most common mistakes
  • Your study is one data point
  • Do:
    • Being careful with your language
    • Include a “limitations” section
    • Tell a clear story (don’t have to tell every finding)
1 not having a clear research question
#1 Not having a clear research question
  • Hard to explain the choices you made
  • Hard to explain your findings
  • May end up focusing on usability problems
  • Do
    • Have a clear research question
your turn
Your Turn….
  • Get into groups of 2-3
  • Choose an example prototype
  • Determine a research question
  • Come up with a study design
    • Participants
    • Length
    • Data to collect
    • Timeline for study

acknowledgments
Acknowledgments

Ed Cutrell

MSR India

Sunny Consolvo

Intel Research Seattle

Beverly Harrison

Intel Research Seattle

user studies are fun
User Studies Are Fun!
  • Always learn something
  • Often surprising!
  • Inspire you

References:

  • Brush, Ubiquitous Computing Field Studiesin Ubiquitous Computing Fundamentals.
  • Green & SalkindUsing SPSS for the Windows and Macintosh: Analyzing and Understanding Data (nth Ed.)
  • APA Publication Manual 
  • HCI textbooks
    • Preece, J., Rogers, Y., and Sharp, H. Interaction design: beyond human-computer interaction. 2002. John Wiley & Sons, Inc

A.J. Brush, ajbrush@microsoft.com