How to run any kind of Evaluation. 3 /6/14 HCC 729, Human Centered Design Amy Hurst. Getting started. Share inspirations, reading reflections http://hcc729s2014.wordpress.com/student-blogs / Homework check in (paper prototypes). Paper prototypes. Activity (10 minutes)
3/6/14HCC 729, Human Centered Design
“We’re going live in two weeks; do you have time to look over our site?”
Nielson’s Categories for Usability Methods, Chapter 7 Usability Engineering
I was at the post office one day, and a student came up to the woman behind the counter and asked “who has the hardest job in the world?” She answered the president of America. He wrote this down, turned to me and asked me “who has the hardest job in world?”
What kind of results do you expect this student will get?
What would you change about how this student is administering this survey?Who has the hardest job in the world?
Always think about how you are biasing (distorting, impacting, controlling) your results
Your goal is to gather data that is reliable, and repeatable.Avoid bias in your evaluations!
3 simple Factors you can control:
Location of evaluation
Your Behavior and Actions
“My Roommate thought the buttons were too small” impacting, controlling) your results
“My mom really liked my color choices”
“My Girlfriend found the following Typos”Who should you recruit for your study?
Undercoverage. Undercoverage occurs when some members of the population are inadequately represented in the sample.
Who are the ideal versus acceptable participants? impacting, controlling) your results
Aim for the actual users of the system If unavailable, aim for the closest approximation
Things to consider:
Often possess particular domain knowledge
Have a history with existing systems, methods, etc.
Where did you find these people?
Others?Select appropriate participants
Laboratory vs. Real World studies
Remember your environmental Analysis?
But, how realistic is your user study setting? impacting, controlling) your results
What changes if your user is… impacting, controlling) your results
Waiting for the train at a crowded MARC station
Sitting on the grass in the park on a sunny day
Curious during a movie
In an office that is quiet and dull
Working at home
Working in a coffee shopExploring the role of environment
Conduct an “Environmental Analysis” and control the evaluation environment.
Understand where your interface will be used
This is usually best done through interviews or observations of real world use
A few things to consider…
Be as faithful to real situations as possible (get creative)
Consider more complicated aspects of the environment: include distractions and stress if appropriate (noise/heat)
Consider how the environment will effect machine performance (internet lag time, sensors not working, etc)
Does this really matter?
Ex: speech recognizer achieved 98.7% word accuracy in your user study; but the real world deployment of your system will be on an airport tarmac…Why does this matter?
What is a consent form?
Why do I care?
“Always wear blue in court”
“Wearing Green makes people think of money”
Your gender, age, ethnicity
Being in a position of “power”
In order to avoid bias, you want to control for as many variables as possible.
Make the experience the same for each user
What are some of the factors of the experimenter that could impact results?
Attitude (are they grumpy, or not paying attention?)
What is said to the participantYes, your behavior matters!
Make sure they get the same instructions
Make sure you ask all questions the same way
Helps control evaluation duration
Makes it easier for you to repeat the study
Write a task script of everything that will happen in the study.
Treat this like a script for a play
I literally include that happens from “hello” to “goodbye”Ensure consistency: use a task script
For the final report