1 / 47

IAT 334 Experimental Evaluation

IAT 334 Experimental Evaluation. ______________________________________________________________________________________ SCHOOL OF INTERACTIVE ARTS + TECHNOLOGY [SIAT] | WWW.SIAT.SFU.CA. Evaluation. Evaluation styles Subjective data Questionnaires, Interviews Objective data

garima
Download Presentation

IAT 334 Experimental Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. IAT 334 Experimental Evaluation ______________________________________________________________________________________SCHOOL OF INTERACTIVE ARTS + TECHNOLOGY [SIAT] | WWW.SIAT.SFU.CA

  2. Evaluation • Evaluation styles • Subjective data • Questionnaires, Interviews • Objective data • Observing Users • Techniques, Recording • Usability Specifications • Why, How IAT 334

  3. Our goal? IAT 334

  4. Evaluation • Earlier: • Interpretive and Predictive • Heuristic evaluation, walkthroughs, ethnography… • Now: • User involved • Usage observations, experiments, interviews... IAT 334

  5. Evaluation Forms • Summative • After a system has been finished. Make judgments about final item. • Formative • As project is forming. All through the lifecycle. Early, continuous. IAT 334

  6. Evaluation Data Gathering • Design the experiment to collect the data to test the hypothesis to evaluate the interface to refine the design • Information we gather about an interface can be subjective or objective • Information also can be qualitative or quantitative • Which are tougher to measure? IAT 334

  7. Subjective Data • Satisfaction is an important factor in performance over time • Learning what people prefer is valuable data to gather IAT 334

  8. Methods • Ways of gathering subjective data • Questionnaires • Interviews • Booths (eg, trade show) • Call-in product hot-line • Field support workers IAT 334

  9. Questionnaires • Preparation is expensive, but administration is cheap • Oral vs. written • Oral advs: Can ask follow-up questions • Oral disadvs: Costly, time-consuming • Forms can provide better quantitative data IAT 334

  10. Questionnaires • Issues • Only as good as questions you ask • Establish purpose of questionnaire • Don’t ask things that you will not use • Who is your audience? • How do you deliver and collect questionnaire? IAT 334

  11. Questionnaire Topic • Can gather demographic data and data about the interface being studied • Demographic data: • Age, gender • Task expertise • Motivation • Frequency of use • Education/literacy IAT 334

  12. Interface Data • Can gather data about • screen • graphic design • terminology • capabilities • learning • overall impression • ... IAT 334

  13. Question Format • Closed format • Answer restricted to a set of choices Characters on screen hard to read easy to read 1 2 3 4 5 6 7 IAT 334

  14. Closed Format • Likert Scale • Typical scale uses 5, 7 or 9 choices • Above that is hard to discern • Doing an odd number gives the neutral choice in the middle IAT 334

  15. Advantages Clarify alternatives Easily quantifiable Eliminates useless answers Disadvantages Must cover whole range All should be equally likely Don’t get interesting, “different” reactions Closed Format IAT 334

  16. Issues • Question specificity • “Do you have a computer?” • Language • Beware terminology, jargon • Clarity • Leading questions • Can be phrased either positive or negative IAT 334

  17. Issues • Prestige bias • People answer a certain way because they want you to think that way about them • Embarrassing questions • Hypothetical questions • “Halo effect” • When estimate of one feature affects estimate of another (eg, intelligence/looks) IAT 334

  18. Deployment • Steps • Discuss questions among team • Administer verbally/written to a few people (pilot). Verbally query about thoughts on questions • Administer final test IAT 334

  19. Open-ended Questions • Asks for unprompted opinions • Good for general, subjective information, but difficult to analyze rigorously • May help with design ideas • “Can you suggest improvements to this interface?” IAT 334

  20. Ethics • People can be sensitive about this process and issues • Make sure they know you are testingsoftware, not them • Attribution theory • Studies why people believe that they succeeded or failed--themselves or outside factors (gender, age differences) • Can quit anytime IAT 334

  21. Objective Data • Users interact with interface • You observe, monitor, calculate, examine, measure, … • Objective, scientific data gathering • Comparison to interpretive/predictive evaluation IAT 334

  22. Observing Users • Not as easy as you think • One of the best ways to gather feedback about your interface • Watch, listen and learn as a person interacts with your system IAT 334

  23. Direct In same room Can be intrusive Users aware of your presence Only see it one time May use semitransparent mirror to reduce intrusiveness Indirect Video recording Reduces intrusiveness, but doesn’t eliminate it Cameras focused on screen, face & keyboard Gives archival record, but can spend a lot of time reviewing it Observation IAT 334

  24. Location • Observations may be • In lab - Maybe a specially built usability lab • Easier to control • Can have user complete set of tasks • In field • Watch their everyday actions • More realistic • Harder to control other factors IAT 334

  25. Challenge • In simple observation, you observe actions but don’t know what’s going on in their head • Often utilize some form of verbal protocol where users describe their thoughts IAT 334

  26. Verbal Protocol • One technique: Think-aloud • User describes verbally what s/he is thinking and doing • What they believe is happening • Why they take an action • What they are trying to do IAT 334

  27. Think Aloud • Very widely used, useful technique • Allows you to understand user’s thought processes better • Potential problems: • Can be awkward for participant • Thinking aloud can modify way user performs task IAT 334

  28. Teams • Another technique: Co-discovery learning • Join pairs of participants to work together • Use think aloud • Perhaps have one person be semi-expert (coach) and one be novice • More natural (like conversation) so removes some awkwardness of individual think aloud IAT 334

  29. Alternative • What if thinking aloud during session will be too disruptive? • Can use post-event protocol • User performs session, then watches video afterwards and describes what s/he was thinking • Sometimes difficult to recall IAT 334

  30. Historical Record • In observing users, how do you capture events in the session for later analysis? IAT 334

  31. Capturing a Session • 1. Paper & pencil • Can be slow • May miss things • Is definitely cheap and easy Task 1 Task 2 Task 3 … Time 10:00 10:03 10:08 10:22 S e S e IAT 334

  32. Capturing a Session • 2. Audio tape • Good for talk-aloud • Hard to tie to interface • 3. Video tape • Multiple cameras probably needed • Good record • Can be intrusive IAT 334

  33. Capturing a Session • 4. Software logging • Modify software to log user actions • Can give time-stamped key press or mouse event • Two problems: • Too low-level, want higher level events • Massive amount of data, need analysis tools IAT 334

  34. Assessing Usability • Usability Specifications • Quantitative usability goals, used a guide for knowing when interface is “good enough” • Should be established as early as possible in development process IAT 334

  35. Measurement Process • “If you can’t measure it, you can’t manage it” • Need to keep gathering data on each iterative refinement IAT 334

  36. What to Measure? • Usability attributes • Initial performance • Long-term performance • Learnability • Retainability • Advanced feature usage • First impression • Long-term user satisfaction Quantitative IAT 334

  37. How to Measure? • Benchmark Task • Specific, clearly stated task for users to carry out • Example: Calendar manager • “Schedule an appointment with Prof. Smith for next Thursday at 3pm.” • Users perform these under a variety of conditions and you measure performance IAT 334

  38. Assessment Technique Usability Measure Value to Current Worst Planned Best poss Observ attributeinstrumentbe measuredlevelacc leveltarget levellevelresults Initial Benchmk Length of 15 secs 30 secs 20 secs 10 secs perf task time to (manual) success add appt on first trial First Quest -2..2 ?? 0 0.75 1.5 impression IAT 334

  39. Summary • Measuring Instrument • Questionnaires • Benchmark tasks IAT 334

  40. Summary • Value to be measured • Time to complete task • Number of percentage of errors • Percent of task completed in given time • Ratio of successes to failures • Number of commands used • Frequency of help usage IAT 334

  41. Summary • Target level • Often established by comparison with competing system or non-computer based task IAT 334

  42. Ethics • Testing can be arduous • Each participant should consent to be in experiment (informal or formal) • Know what experiment involves, what to expect, what the potential risks are • Must be able to stop without danger or penalty • All participants to be treated with respect IAT 334

  43. Consent • Why important? • People can be sensitive about this process and issues • Errors will likely be made, participant may feel inadequate • May be mentally or physically strenuous • What are the potential risks (there are always risks)? • Examples? • “Vulnerable” populations need special care & consideration (& IRB review) • Children; disabled; pregnant; students (why?) IAT 334

  44. Before Study • Be well prepared so participant’s time is not wasted • Make sure they know you are testing software, not them • (Usability testing, not User testing) • Maintain privacy • Explain procedures without compromising results • Can quit anytime • Administer signed consent form IAT 334

  45. During Study • Make sure participant is comfortable • Session should not be too long • Maintain relaxed atmosphere • Never indicate displeasure or anger IAT 334

  46. After Study • State how session will help you improve system • Show participant how to perform failed tasks • Don’t compromise privacy (never identify people, only show videos with explicit permission) • Data to be stored anonymously, securely, and/or destroyed IAT 334

  47. One Model IAT 334

More Related