1 / 13

Usability & Satisfaction Testing Pitfalls

Usability & Satisfaction Testing Pitfalls . Rick Rappe’ Vocal Laboratories Inc. Pitfall #1 Not Testing At All. -Big Egos The Hallmark of a Good VUI Designer is Humility -Too Much Vendor Trust -Budget & Time Constraints Testing needs to be an integral part of the cost/design process.

dasha
Download Presentation

Usability & Satisfaction Testing Pitfalls

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Usability & Satisfaction Testing Pitfalls Rick Rappe’ Vocal Laboratories Inc.

  2. Pitfall #1Not Testing At All -Big Egos The Hallmark of a Good VUI Designer is Humility -Too Much Vendor Trust -Budget & Time Constraints Testing needs to be an integral part of the cost/design process

  3. Pitfall #2 Inadequate Testing Methodologies The Focus Group is a design aid, not a method to vet the new application for ease of use and caller satisfaction. A usability problem affecting 5% of calls requires 100 test calls to reliably hit the problem just once. A 1% problem takes 500. If a problem is hit but once, will it be recognized?

  4. Inadequate Testing Methodologies, cont. • The WOZ (Wizard of Oz) test does not exercise speech recognizer accuracy. • The rapid prototype test is a good design aid, but does not address technical performance (load/stress tests) and will not uncover all grammar/ recognizer issues. • Friends and Family (insider) testing is flawed. You are not like your customers. You understand jargon and how the system is supposed to work. • Testing is an ongoing requirement. Things change.

  5. Pitfall #3 Things Change • A well done (satisfied caller) application will degrade over time. • Products, procedures and marketing “changes” result in Application Drift, and • As consumers adopt self service, their expectations rise.

  6. Pitfall #4 Bad Survey Technique • End-Of-Call surveys systematically exclude anyone who hangs up early. (Participation bias) • Follow up surveys reveal few useful specifics on improvement • Too small a sample, too high a margin of error • Insiders and Focus Groups do not emulate a customer base (Sample bias)

  7. Pitfall #5 Poor Survey Usage • Were you happy, or would you like to be run over by a truck? • Over 1/2 of all surveys are said to be written to justify a currently held belief rather than objectively seek truth. • How satisfied were you? • Biased upward question. Should be: “How satisfied or dissatisfied were you?” • Do you like Cookies? Will get different answers if previous Q is: “Do you like milk?”

  8. The Solution (New Apps.) • Consider a usability/satisfaction exam of existing service as a base line from which to measure acceptance of the new service. • Run “mini” usability tests throughout the development process when changes are easy to accomplish. • Do not rely on a pre-cut Pilot test. Why? • You WILL uncover problems when the fix will cause cutover and budget delays; and a tendency to declare the application “good enough” is strong. • Using actual customers as test call guinea pigs is a risk to customer retention and brand loyalty.

  9. The Solution (Existing Apps.) • An IVR/Speech recognition application is NOT a build it and forget it exercise. • Because “things” change, consider an ongoing survey large enough to detect opinion shifts.

  10. Conducting a Quality Survey • Is poor survey technique, delivering slanted results/interpretation, a cause of mediocre customer service? • Yes, if “false positives” lull into a why change attitude.

  11. Five Survey Rules • 1. Sample must reflect a fair cross section of typical users. • Sample bias, participation bias & insider jargon must be avoided • 2. Sample must be large enough to reliably exercise the application. • The trade off between sample size and accuracy must be understood. Situations differ, but beyond a study size yielding a 4-5% margin of error, more surveys yield decreasing levels of accuracy for the cost/effort

  12. Five Survey Rules cont. • 3. Ask meaningful and objective questions. • Specific queries yield actionable results • 4. Understand the responses. • The human desire to please must be taken into account. “Satisfied” in the mind of the consumer is not a positive/loyalty rating. • 5. Do something with the information. • Feedback suggests that results from as many as 2/3 of satisfaction surveys are never acted upon.

  13. Rick Rappe’ rappe@vocalabs.com952-941-6580 X205

More Related