Conducting User Studies: Pilot Testing & Study Setup

1. Running a User Study Alfred Kobsa University of California, Irvine

2. Pilot Test • A pilot test is extremely valuable for detecting problems in the planned user test. • These problems may be harmless, but they may also obstruct the user test or render its results unusable. • 2-5 pilot subjects are sufficient. • Some of them should be people who are very much willing to criticize the experiment (e.g., good friends, colleagues). • At least one of them should be a real test subject. • Pilot test should be run two days before the actual study at the latest (and there are many good reasons to do them far earlier). • Pilot test should include all procedures.

3. Problems that are frequentlyuncovered in pilot tests • Subjects take considerably longer/shorter than expected. • Task instructions are not well understood by subjects • Subjects cannot carry out a task since some necessary functionality has not been explained to them • Subjects use procedures for solving a task that are different from the one that the experimenters wanted to test • Subjects overwrite each others’data files • Experimenters overwrite data of previous experiments • Software crashes • Experimenters forget to do certain things • Some materials are missing • Users have troubles finding parking, the building entrance, the usability lab, etc.

4. Welcome, Briefing,Instruction and Training Welcome • Make participants feel comfortable / reduce anxiety • Bridge time until everyone has arrived • Let them show their ID (“for security/tax purposes”), and announce this beforehand • Ask them to switch off their cell phones. Briefing • Inform about the purpose of the experiment (as far as this is possible) • Emphasize that it will help develop a better product • Encourage criticism and praise • Emphasize that the product is being tested and not they themselves • Make them believe that the people they will see are not the developers • Show them the lab (or movies or pictures of it) • Have them sign legal forms • Let them fill in a pre-questionnaire (e.g., demographics, pre-test) Instruction and Training • Instruction through video, Powerpoint, or instructor who follows a written script • Subsequent training tasks allow users to practice what they learned and to understand it better. • Guided training tasks: task description contains (partial) instructions on how to carry out a task; subjects are asked to follow them

5. Conducting the Test Setup • (Bring subjects to test computers) • Verify initial settings and materials, start recording devices. • [Instruct them to “think aloud” (this makes task completion times unreliable)] • Ask them to begin (and to let you know when they are done) • Enter users in logbook/timesheet (e.g., who used which computer) During the test • Watch participants and take notes, or monitor them from a distance • Record unusual occurrences in logbook/timesheet • Answer questions generically; avoid biasing participants • Inform users about timeouts • If applicable, remind them to think aloud if they become silent • If applicable: make sure that users don’t influence each other After the test • Post-questionnaire and/or post-interview (e.g., opinions on software and on test procedures; post-test) • Payment, call escort • Prepare test station for new test subjects

6. At any time Be organized! Follow checklists!

7. Unexpected events A participant… • does not arrive on time • says s/he forgot his/her ID • refuses to sign the informed consent or non-disclosure form • refuses to be videotaped • does not want to switch off his/her cell phone • is called away in the middle of the test • has their cell phone on and it rings continuously • does not have the required qualifications • exceeds the time limit for a task without finishing it • struggles mightily with a task • becomes confrontational w/ other subjects or the experimenters • takes over the group Software freezes, computer breaks down, etc. Fire alarm goes off Outside interference (construction noise, vacuums, …)