running a user study
Download
Skip this Video
Download Presentation
Running a User Study

Loading in 2 Seconds...

play fullscreen
1 / 7

Running a User Study - PowerPoint PPT Presentation


  • 303 Views
  • Uploaded on

Running a User Study Alfred Kobsa University of California, Irvine Pilot Test A pilot test is extremely valuable for detecting problems in the planned user test. These problems may be harmless, but they may also render test results unusable. 2-5 pilot subjects are sufficient.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Running a User Study' - Ava


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
running a user study

Running a User Study

Alfred Kobsa

University of California, Irvine

pilot test
Pilot Test
  • A pilot test is extremely valuable for detecting problems in the planned user test.
  • These problems may be harmless, but they may also render test results unusable.
  • 2-5 pilot subjects are sufficient.
  • Some of them should be people who are very much willing to criticize the experiment (e.g., good friends, colleagues).
  • At least one of them should be a real test subject.
  • Pilot test should be run two days before the actual study at the latest (and there are many good reasons to do them far earlier).
  • Pilot test should include all procedures.
problems that are frequently uncovered in pilot tests
Problems that are frequentlyuncovered in pilot tests
  • Subjects take considerably longer/shorter than expected.
  • Task instructions are not well understood by subjects
  • Subjects cannot carry out a task since some necessary functionality has not been explained to them
  • Subjects use procedures for solving a task that are different from the one that the experimenters wanted to test
  • Subjects overwrite each others’ data files
  • Experimenters overwrite data of previous experiments
  • Software crashes
  • Experimenters forget to do certain things
  • Some materials are missing
  • Users have troubles finding parking, the building entrance, the usability lab, etc.
welcome briefing instruction and training
Welcome, Briefing,Instruction and Training
  • Welcome
  • Make participants feel comfortable / reduce anxiety
  • Bridge time until everyone has arrived
  • Let them show their ID (“for security/tax purposes”), and announce this beforehand
  • Ask them to switch their cell phones off.
  • Briefing
  • Inform about purpose of the experiment (as far as this is possible)
  • Emphasize that it will help develop a better product
  • Encourage criticism and praise
  • Emphasize that the product is being tested and not the user
  • Emphasize that the people they will see are not the developers
  • Show them the lab (or movies or pictures of it)
  • Have them sign legal forms
  • Let them fill in a pre-questionnaire (e.g., demographics, pre-test)
  • Instruction and Training
  • Instruction through video, or through instructor who follows a written script
  • Subsequent training tasks allow users to practice what they learned and to understand it better.
  • Guided training tasks: task description contains (partial) instructions on how to carry out a task, which subjects are asked to follow
conducting the test
Conducting the Test
  • Setup
  • (Bring subjects to test computers)
  • Verify initial settings and materials, start recording devices.
  • Ask them to begin (and to let you know when they are done)
  • Enter users in logbook/timesheet (e.g., who used which computer)
  • During the test
  • Inform users about timeouts
  • Answer questions generically; avoid biasing participants
  • If applicable, remind them to think aloud if they become silent
  • Watch test users and take notes, or monitor users from a distance
  • Record unusual occurrences in logbook/timesheet
  • If applicable: remind users occasionally to think aloud
  • If applicable: make sure users don’t influence each other
  • After the test
  • Post-questionnaire (e.g., opinions on software, test procedures; post-test)
  • Payment, escort
  • Prepare test station for new test subjects
at any time
At any time

Be organized!

Follow checklists!

unexpected events
Unexpected events
  • A participant does not arrive in time
  • A participant forgot(?) his or her ID
  • A participant refuses to sign the informed consent form or the non-disclosure agreement
  • A participant refuses to be videotaped
  • A participant does not want to switch off his/her cell phone
  • A participant is called away in the middle of the test
  • A participant’s cell phone rings continuously
  • A participant does not have the required qualifications
  • A participant becomes confrontational with other participants or with the experimenters
  • A participant takes over the group
  • Software freezes, computer breaks down, etc.
  • Outside interference (construction noise, vacuums, …)
ad