Running a user study l.jpg
Advertisement
This presentation is the property of its rightful owner.
1 / 7

Running a User Study PowerPoint PPT Presentation

Running a User Study Alfred Kobsa University of California, Irvine Pilot Test A pilot test is extremely valuable for detecting problems in the planned user test. These problems may be harmless, but they may also render test results unusable. 2-5 pilot subjects are sufficient.

Related searches for Running a User Study

Download Presentation

Running a User Study

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Running a user study l.jpg

Running a User Study

Alfred Kobsa

University of California, Irvine


Pilot test l.jpg

Pilot Test

  • A pilot test is extremely valuable for detecting problems in the planned user test.

  • These problems may be harmless, but they may also render test results unusable.

  • 2-5 pilot subjects are sufficient.

  • Some of them should be people who are very much willing to criticize the experiment (e.g., good friends, colleagues).

  • At least one of them should be a real test subject.

  • Pilot test should be run two days before the actual study at the latest (and there are many good reasons to do them far earlier).

  • Pilot test should include all procedures.


Problems that are frequently uncovered in pilot tests l.jpg

Problems that are frequentlyuncovered in pilot tests

  • Subjects take considerably longer/shorter than expected.

  • Task instructions are not well understood by subjects

  • Subjects cannot carry out a task since some necessary functionality has not been explained to them

  • Subjects use procedures for solving a task that are different from the one that the experimenters wanted to test

  • Subjects overwrite each others’ data files

  • Experimenters overwrite data of previous experiments

  • Software crashes

  • Experimenters forget to do certain things

  • Some materials are missing

  • Users have troubles finding parking, the building entrance, the usability lab, etc.


Welcome briefing instruction and training l.jpg

Welcome, Briefing,Instruction and Training

  • Welcome

  • Make participants feel comfortable / reduce anxiety

  • Bridge time until everyone has arrived

  • Let them show their ID (“for security/tax purposes”), and announce this beforehand

  • Ask them to switch their cell phones off.

  • Briefing

  • Inform about purpose of the experiment (as far as this is possible)

  • Emphasize that it will help develop a better product

  • Encourage criticism and praise

  • Emphasize that the product is being tested and not the user

  • Emphasize that the people they will see are not the developers

  • Show them the lab (or movies or pictures of it)

  • Have them sign legal forms

  • Let them fill in a pre-questionnaire (e.g., demographics, pre-test)

  • Instruction and Training

  • Instruction through video, or through instructor who follows a written script

  • Subsequent training tasks allow users to practice what they learned and to understand it better.

  • Guided training tasks: task description contains (partial) instructions on how to carry out a task, which subjects are asked to follow


Conducting the test l.jpg

Conducting the Test

  • Setup

  • (Bring subjects to test computers)

  • Verify initial settings and materials, start recording devices.

  • Ask them to begin (and to let you know when they are done)

  • Enter users in logbook/timesheet (e.g., who used which computer)

  • During the test

  • Inform users about timeouts

  • Answer questions generically; avoid biasing participants

  • If applicable, remind them to think aloud if they become silent

  • Watch test users and take notes, or monitor users from a distance

  • Record unusual occurrences in logbook/timesheet

  • If applicable: remind users occasionally to think aloud

  • If applicable: make sure users don’t influence each other

  • After the test

  • Post-questionnaire (e.g., opinions on software, test procedures; post-test)

  • Payment, escort

  • Prepare test station for new test subjects


At any time l.jpg

At any time

Be organized!

Follow checklists!


Unexpected events l.jpg

Unexpected events

  • A participant does not arrive in time

  • A participant forgot(?) his or her ID

  • A participant refuses to sign the informed consent form or the non-disclosure agreement

  • A participant refuses to be videotaped

  • A participant does not want to switch off his/her cell phone

  • A participant is called away in the middle of the test

  • A participant’s cell phone rings continuously

  • A participant does not have the required qualifications

  • A participant becomes confrontational with other participants or with the experimenters

  • A participant takes over the group

  • Software freezes, computer breaks down, etc.

  • Outside interference (construction noise, vacuums, …)


  • Login