1 / 46

Preparing and Running User Experiments

Preparing and Running User Experiments By Mei Li, Pearl Ho, and Deepika Gandhi How to Prepare and Run Usability Testing How to gather participants and prepare a proposal. Preparing the environment, test materials, and test team. How to measure usability and deal with participants.

Ava
Download Presentation

Preparing and Running User Experiments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Preparing and Running User Experiments By Mei Li, Pearl Ho, and Deepika Gandhi

  2. How to Prepare and Run Usability Testing • How to gather participants and prepare a proposal. • Preparing the environment, test materials, and test team. • How to measure usability and deal with participants.

  3. Preparing Materials, Environment, and Test Team for Usability Tests Presenter: Pearl Ho From Dumas and Redding Chapters 11, 12, 14, 15, 16

  4. Overview • Selecting and organizing tasks to test • Creating task scenarios • Preparing test materials • Preparing the test environment • Preparing the test team

  5. I. Selecting and Organizing Tasks to Test • How do you select which tasks to test? • What resources do you need for each task?

  6. Selecting Tasks • Tasks that probe potential usability problems • Use tasks suggested from concerns and experience • Tasks that users will do with the product

  7. Selecting Tasks • Tasks that probe potential usability problems • The more problems you find, the more successful your test will be • Tasks suggested from concerns and experience • Tasks that users will do with the product

  8. Selecting Tasks • Tasks that probe potential usability problems • Tasks suggested from concerns and experience • People who develop the product will have some sense of what will be difficult to use based on which parts were difficult to design or which parts where they disagreed about the best approach • Tasks that users will do with the product

  9. Selecting Tasks • Tasks that probe potential usability problems • Tasks suggested from concerns and experience • Tasks that users will do with the product • There are other tasks that users can do with a product over and above those that relate to usability problems, concerns, or goals. For example: • New or modified tasks • Tasks that are critical to the operation of the product • Frequently done tasks • Things done under pressure

  10. Determining the Resources You Need for Each Task • Estimating task times • Listing the resources you need for tasks

  11. Determining the Resources You Need for Each Task • Estimating task times • Time it will take to run the task during the test. • Time that users will feel is acceptable for completing a task. • Consider characteristics of participants who will be in the test • Time for experienced person. This forms baseline • Problems for typical participant and time to recover from them • Time to set up tasks and time between tasks • Estimate will probably be a range, not a single value • Listing the resources you need for tasks

  12. Determining the Resources You Need for Each Task • Estimating task times • Listing the resources you need for tasks • Hardware – participants needs to do the task AND what you need to set up and run the task • Software and data files – again for both participants and you • Instructions and procedures you will need

  13. Determining the Resources You Need for Each Task • In addition to task times, for most tests, it takes about an hour to • Start the test • Give the prebriefing • Give the participant a break halfway through the test • Have the participant fill out the posttest questionnaire • Give the debriefing

  14. Creating the Final List of Resources • Here’s an example of what a list of resources might look like for testing the task of sending a reply on an email program

  15. II. Creating Task Scenarios • What is a scenario? • What makes a good scenario? • Do you give participants written scenarios? • How do you divide up tasks and scenarios? • How do you make participants stop between tasks for timing purposes?

  16. What is a scenario? • You use scenarios to tell participants what you want them to do during the test. • Scenarios describe the tasks in a way that takes some of the artificiality out of the test. • It makes the task more realistic.

  17. What makes a good scenario? • Short • User’s words, not product’s words • Unambiguous • Enough information to do the task • Directly linked to your tasks and concerns

  18. Do you always give participants written scenarios? Not always. You may: • Have test team members pretend to be customers, supervisors, or colleagues and walk into the test room to deliver the scenarios in person • Use the product to deliver the scenarios • Mix modes, give them writing, but then add visits or calls

  19. How do you divide up the tasks and scenarios for participants? • It depends… • In deciding how to match up scenarios and tasks, the major issue you have to consider is whether you want to separate measurements of time, errors, or other codes for particular tasks.

  20. How do you make participants stop between tasks? • If you want to time tasks separately, you need to get participants to stop between each task. • You can give them one task at a time. • Or you can just say after each task: “Please tell us when you have finished this task.” or “Please wait for us to tell you to turn the page.”

  21. III. Preparing Test Materials • The Legal Form - minimal risk, informed consent • Questionnaires • Participant Training

  22. The Legal Form As a tester it is your responsibility to: • Create a form that correctly states each party’s rights • Ensure that test participants have read and understood what the form says • Observe or witness participants signing the form

  23. Concept of minimal risk • Minimal risk means that “the probability and magnitude of harm or discomfort anticipated in the test are not greater, in and of themselves, than those ordinarily encountered in daily life or during the performance of routine physical or psychological examinations or tests.” • If, in your judgment, any usability test could put participants at risk, you have three options: • Eliminate the risk • Follow the federal policies for such tests • Don’t do the test

  24. Informed consent • Key to protecting both parties rights • Participants should be given the opportunity to choose what will or will not happen to them

  25. Three elements of any informed consent: • Information • Comprehension • Voluntariness

  26. Three elements of any informed consent: • Information: • Procedures you will follow • The purpose of the test • Any risks to the participant • Provide the opportunity to ask questions • Provide the opportunity to withdraw at any time • Comprehension • Voluntariness

  27. Three elements of any informed consent: • Information • Comprehension • You must convey the information clearly and completely. • Do not rush through consent procedure, make it seem unimportant, or not allow questions. • Voluntariness

  28. Three elements of any informed consent: • Information • Comprehension • Voluntariness • Participants must be free from coercion and undue influence • If participants are considering withdrawing from a test, remain calm and neutral in your manner so that you do not unduly influence them

  29. Other Major legal issues: • Nondisclosure, if product is still under development • Waivers, permission to use tapes and questionnaires • Participant’s rights, ex, right to be informed

  30. Questionnaires • Pretest questionnaire - gather info about the background of participant • Posttask questionnaire - gather judgments and rating after each task • Posttest questionnaire - gather judgments after completion of all tasks

  31. Training Participants Reasons to train • To ensure that all participants have the same level of skill or knowledge before they begin the tasks • To provide some groups of participants with training that others do not get

  32. Training Participants To provide effective training, you need to decide: • The purpose of the training • What skills and knowledge you want participants to learn • How you will conduct the training • What criteria of success you will use to measure the effectiveness of training

  33. IV. Preparing the Test Environment • Preparing the Product • Data-logging material preparation

  34. Preparing the Product • Create the sample data you need. • Plan for disaster. Save time and frustration. • Preparing for manuals. • Legible type and graphics • Numbered pages • High quality, professional looking binding or binder • Put in table of contents, cross-references, and an index

  35. Preparing Data-Logging Materials or Software • The key to having quantitative measures of performance is recording the duration and frequency of events during the test. • You can buy data-logging software or at least have stopwatch and sheet of paper that lists the task numbers. • If you decide to videotape the test, keep in mind that it takes twice as long to run test and then watch the tape.

  36. V. Preparing the Test Team • How many people do you need to run a test? • Should people switch roles during the test? • Who should be on the test team • What roles are there for test team members

  37. How many people do you need to run a test? • Three people make a good team. Some have five, including a technical person. • Two can make up team if both are skilled usability specialists and participants don’t have to move around much.

  38. Should people switch roles during the test? No. • You don’t want differences in data based on differences in test team.

  39. Who should be on the test team? • Everyone on test team should be on the planning team, but not everyone planning has to be on test team. • Product developers or usability specialists? Realize obvious tradeoffs and choose one that makes sense for you. • Other people, such as managers, developers, writers, can participate in a test as observers.

  40. What roles are there for test team members? • Test administrator • Briefer • Camera operator • Data operator • Help desk operator • Product expert • Narrator

  41. See the text book if you want specific tasks and a checklist for each role

  42. QUESTIONS? On to Deepika…. During your User Requirements Activity

More Related