Preparing and running user experiments l.jpg
Advertisement
This presentation is the property of its rightful owner.
1 / 46

Preparing and Running User Experiments PowerPoint PPT Presentation

Preparing and Running User Experiments By Mei Li, Pearl Ho, and Deepika Gandhi How to Prepare and Run Usability Testing How to gather participants and prepare a proposal. Preparing the environment, test materials, and test team. How to measure usability and deal with participants.

Download Presentation

Preparing and Running User Experiments

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Preparing and running user experiments l.jpg

Preparing and Running User Experiments

By Mei Li, Pearl Ho, and Deepika Gandhi


How to prepare and run usability testing l.jpg

How to Prepare and Run Usability Testing

  • How to gather participants and prepare a proposal.

  • Preparing the environment, test materials, and test team.

  • How to measure usability and deal with participants.


Preparing materials environment and test team for usability tests l.jpg

Preparing Materials, Environment, and Test Team for Usability Tests

Presenter: Pearl Ho

From Dumas and Redding

Chapters 11, 12, 14, 15, 16


Overview l.jpg

Overview

  • Selecting and organizing tasks to test

  • Creating task scenarios

  • Preparing test materials

  • Preparing the test environment

  • Preparing the test team


I selecting and organizing tasks to test l.jpg

I. Selecting and Organizing Tasks to Test

  • How do you select which tasks to test?

  • What resources do you need for each task?


Selecting tasks l.jpg

Selecting Tasks

  • Tasks that probe potential usability problems

  • Use tasks suggested from concerns and experience

  • Tasks that users will do with the product


Selecting tasks7 l.jpg

Selecting Tasks

  • Tasks that probe potential usability problems

    • The more problems you find, the more successful your test will be

  • Tasks suggested from concerns and experience

  • Tasks that users will do with the product


Selecting tasks8 l.jpg

Selecting Tasks

  • Tasks that probe potential usability problems

  • Tasks suggested from concerns and experience

    • People who develop the product will have some sense of what will be difficult to use based on which parts were difficult to design or which parts where they disagreed about the best approach

  • Tasks that users will do with the product


Selecting tasks9 l.jpg

Selecting Tasks

  • Tasks that probe potential usability problems

  • Tasks suggested from concerns and experience

  • Tasks that users will do with the product

    • There are other tasks that users can do with a product over and above those that relate to usability problems, concerns, or goals. For example:

      • New or modified tasks

      • Tasks that are critical to the operation of the product

      • Frequently done tasks

      • Things done under pressure


Determining the resources you need for each task l.jpg

Determining the Resources You Need for Each Task

  • Estimating task times

  • Listing the resources you need for tasks


Determining the resources you need for each task11 l.jpg

Determining the Resources You Need for Each Task

  • Estimating task times

    • Time it will take to run the task during the test.

    • Time that users will feel is acceptable for completing a task.

    • Consider characteristics of participants who will be in the test

      • Time for experienced person. This forms baseline

      • Problems for typical participant and time to recover from them

    • Time to set up tasks and time between tasks

    • Estimate will probably be a range, not a single value

  • Listing the resources you need for tasks


Determining the resources you need for each task12 l.jpg

Determining the Resources You Need for Each Task

  • Estimating task times

  • Listing the resources you need for tasks

    • Hardware – participants needs to do the task AND what you need to set up and run the task

    • Software and data files – again for both participants and you

    • Instructions and procedures you will need


Determining the resources you need for each task13 l.jpg

Determining the Resources You Need for Each Task

  • In addition to task times, for most tests, it takes about an hour to

    • Start the test

    • Give the prebriefing

    • Give the participant a break halfway through the test

    • Have the participant fill out the posttest questionnaire

    • Give the debriefing


Creating the final list of resources l.jpg

Creating the Final List of Resources

  • Here’s an example of what a list of resources might look like for testing the task of sending a reply on an email program


Ii creating task scenarios l.jpg

II. Creating Task Scenarios

  • What is a scenario?

  • What makes a good scenario?

  • Do you give participants written scenarios?

  • How do you divide up tasks and scenarios?

  • How do you make participants stop between tasks for timing purposes?


What is a scenario l.jpg

What is a scenario?

  • You use scenarios to tell participants what you want them to do during the test.

  • Scenarios describe the tasks in a way that takes some of the artificiality out of the test.

  • It makes the task more realistic.


What makes a good scenario l.jpg

What makes a good scenario?

  • Short

  • User’s words, not product’s words

  • Unambiguous

  • Enough information to do the task

  • Directly linked to your tasks and concerns


Do you always give participants written scenarios l.jpg

Do you always give participants written scenarios?

Not always. You may:

  • Have test team members pretend to be customers, supervisors, or colleagues and walk into the test room to deliver the scenarios in person

  • Use the product to deliver the scenarios

  • Mix modes, give them writing, but then add visits or calls


How do you divide up the tasks and scenarios for participants l.jpg

How do you divide up the tasks and scenarios for participants?

  • It depends…

  • In deciding how to match up scenarios and tasks, the major issue you have to consider is whether you want to separate measurements of time, errors, or other codes for particular tasks.


How do you make participants stop between tasks l.jpg

How do you make participants stop between tasks?

  • If you want to time tasks separately, you need to get participants to stop between each task.

  • You can give them one task at a time.

  • Or you can just say after each task:

    “Please tell us when you have finished this task.”

    or

    “Please wait for us to tell you to turn the page.”


Iii preparing test materials l.jpg

III. Preparing Test Materials

  • The Legal Form - minimal risk, informed consent

  • Questionnaires

  • Participant Training


The legal form l.jpg

The Legal Form

As a tester it is your responsibility to:

  • Create a form that correctly states each party’s rights

  • Ensure that test participants have read and understood what the form says

  • Observe or witness participants signing the form


Concept of minimal risk l.jpg

Concept of minimal risk

  • Minimal risk means that “the probability and magnitude of harm or discomfort anticipated in the test are not greater, in and of themselves, than those ordinarily encountered in daily life or during the performance of routine physical or psychological examinations or tests.”

  • If, in your judgment, any usability test could put participants at risk, you have three options:

    • Eliminate the risk

    • Follow the federal policies for such tests

    • Don’t do the test


Informed consent l.jpg

Informed consent

  • Key to protecting both parties rights

  • Participants should be given the opportunity to choose what will or will not happen to them


Three elements of any informed consent l.jpg

Three elements of any informed consent:

  • Information

  • Comprehension

  • Voluntariness


Three elements of any informed consent26 l.jpg

Three elements of any informed consent:

  • Information:

    • Procedures you will follow

    • The purpose of the test

    • Any risks to the participant

    • Provide the opportunity to ask questions

    • Provide the opportunity to withdraw at any time

  • Comprehension

  • Voluntariness


Three elements of any informed consent27 l.jpg

Three elements of any informed consent:

  • Information

  • Comprehension

    • You must convey the information clearly and completely.

    • Do not rush through consent procedure, make it seem unimportant, or not allow questions.

  • Voluntariness


Three elements of any informed consent28 l.jpg

Three elements of any informed consent:

  • Information

  • Comprehension

  • Voluntariness

    • Participants must be free from coercion and undue influence

    • If participants are considering withdrawing from a test, remain calm and neutral in your manner so that you do not unduly influence them


Other major legal issues l.jpg

Other Major legal issues:

  • Nondisclosure, if product is still under development

  • Waivers, permission to use tapes and questionnaires

  • Participant’s rights, ex, right to be informed


Questionnaires l.jpg

Questionnaires

  • Pretest questionnaire - gather info about the background of participant

  • Posttask questionnaire - gather judgments and rating after each task

  • Posttest questionnaire - gather judgments after completion of all tasks


Training participants l.jpg

Training Participants

Reasons to train

  • To ensure that all participants have the same level of skill or knowledge before they begin the tasks

  • To provide some groups of participants with training that others do not get


Training participants36 l.jpg

Training Participants

To provide effective training, you need to decide:

  • The purpose of the training

  • What skills and knowledge you want participants to learn

  • How you will conduct the training

  • What criteria of success you will use to measure the effectiveness of training


Iv preparing the test environment l.jpg

IV. Preparing the Test Environment

  • Preparing the Product

  • Data-logging material preparation


Preparing the product l.jpg

Preparing the Product

  • Create the sample data you need.

  • Plan for disaster. Save time and frustration.

  • Preparing for manuals.

    • Legible type and graphics

    • Numbered pages

    • High quality, professional looking binding or binder

    • Put in table of contents, cross-references, and an index


Preparing data logging materials or software l.jpg

Preparing Data-Logging Materials or Software

  • The key to having quantitative measures of performance is recording the duration and frequency of events during the test.

  • You can buy data-logging software or at least have stopwatch and sheet of paper that lists the task numbers.

  • If you decide to videotape the test, keep in mind that it takes twice as long to run test and then watch the tape.


V preparing the test team l.jpg

V. Preparing the Test Team

  • How many people do you need to run a test?

  • Should people switch roles during the test?

  • Who should be on the test team

  • What roles are there for test team members


How many people do you need to run a test l.jpg

How many people do you need to run a test?

  • Three people make a good team. Some have five, including a technical person.

  • Two can make up team if both are skilled usability specialists and participants don’t have to move around much.


Should people switch roles during the test l.jpg

Should people switch roles during the test?

No.

  • You don’t want differences in data based on differences in test team.


Who should be on the test team l.jpg

Who should be on the test team?

  • Everyone on test team should be on the planning team, but not everyone planning has to be on test team.

  • Product developers or usability specialists? Realize obvious tradeoffs and choose one that makes sense for you.

  • Other people, such as managers, developers, writers, can participate in a test as observers.


What roles are there for test team members l.jpg

What roles are there for test team members?

  • Test administrator

  • Briefer

  • Camera operator

  • Data operator

  • Help desk operator

  • Product expert

  • Narrator


Slide45 l.jpg

  • See the text book if you want specific tasks and a checklist for each role


Questions l.jpg

QUESTIONS?

On to Deepika…. During your User Requirements Activity


  • Login