1 / 7

Running a User Study

Running a User Study. Alfred Kobsa University of California, Irvine. Pilot Test. A pilot test is extremely valuable for detecting problems in the planned user test. These problems may be harmless, but they may also obstruct the user test or render its results unusable.

dboyle
Download Presentation

Running a User Study

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Running a User Study Alfred Kobsa University of California, Irvine

  2. Pilot Test • A pilot test is extremely valuable for detecting problems in the planned user test. • These problems may be harmless, but they may also obstruct the user test or render its results unusable. • 2-5 pilot subjects are sufficient. • Some of them should be people who are very much willing to criticize the experiment (e.g., good friends, colleagues). • At least one of them should be a real test subject. • Pilot test should be run two days before the actual study at the latest (and there are many good reasons to do them far earlier). • Pilot test should include all procedures.

  3. Problems that are frequentlyuncovered in pilot tests • Subjects take considerably longer/shorter than expected. • Task instructions are not well understood by subjects • Subjects cannot carry out a task since some necessary functionality has not been explained to them • Subjects use procedures for solving a task that are different from the one that the experimenters wanted to test • Subjects overwrite each others’data files • Experimenters overwrite data of previous experiments • Software crashes • Experimenters forget to do certain things • Some materials are missing • Users have troubles finding parking, the building entrance, the usability lab, etc.

  4. Welcome, Briefing,Instruction and Training Welcome • Make participants feel comfortable / reduce anxiety • Bridge time until everyone has arrived • Let them show their ID (“for security/tax purposes”), and announce this beforehand • Ask them to switch off their cell phones. Briefing • Inform about the purpose of the experiment (as far as this is possible) • Emphasize that it will help develop a better product • Encourage criticism and praise • Emphasize that the product is being tested and not they themselves • Make them believe that the people they will see are not the developers • Show them the lab (or movies or pictures of it) • Have them sign legal forms • Let them fill in a pre-questionnaire (e.g., demographics, pre-test) Instruction and Training • Instruction through video, Powerpoint, or instructor who follows a written script • Subsequent training tasks allow users to practice what they learned and to understand it better. • Guided training tasks: task description contains (partial) instructions on how to carry out a task; subjects are asked to follow them

  5. Conducting the Test Setup • (Bring subjects to test computers) • Verify initial settings and materials, start recording devices. • [Instruct them to “think aloud” (this makes task completion times unreliable)] • Ask them to begin (and to let you know when they are done) • Enter users in logbook/timesheet (e.g., who used which computer) During the test • Watch participants and take notes, or monitor them from a distance • Record unusual occurrences in logbook/timesheet • Answer questions generically; avoid biasing participants • Inform users about timeouts • If applicable, remind them to think aloud if they become silent • If applicable: make sure that users don’t influence each other After the test • Post-questionnaire and/or post-interview (e.g., opinions on software and on test procedures; post-test) • Payment, call escort • Prepare test station for new test subjects

  6. At any time Be organized! Follow checklists!

  7. Unexpected events A participant… • does not arrive on time • says s/he forgot his/her ID • refuses to sign the informed consent or non-disclosure form • refuses to be videotaped • does not want to switch off his/her cell phone • is called away in the middle of the test • has their cell phone on and it rings continuously • does not have the required qualifications • exceeds the time limit for a task without finishing it • struggles mightily with a task • becomes confrontational w/ other subjects or the experimenters • takes over the group Software freezes, computer breaks down, etc. Fire alarm goes off Outside interference (construction noise, vacuums, …)

More Related