1 / 48

Evaluation Methods

Evaluation Methods. April 20, 2005 Tara Matthews CS 160. In 160 We’ve Covered…. Task Analysis & Contextual Inquiry Cognitive Walkthrough Heuristic Evaluation WOZ usability study w/ paper prototypes. There are many more methods…. Survey Interview Controlled-lab experiment

zagiri
Download Presentation

Evaluation Methods

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Evaluation Methods April 20, 2005 Tara Matthews CS 160

  2. In 160 We’ve Covered… • Task Analysis & Contextual Inquiry • Cognitive Walkthrough • Heuristic Evaluation • WOZ usability study w/ paper prototypes

  3. There are many more methods… • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

  4. How to chose a method? • Stage of study • formative, iterative, summative • Pros & cons • Metrics • depends on what you want to measure • Qualitative vs. quantitative • Research perspective • CS vs. psychology vs. sociology

  5. Pros & Cons • Realism • Precision • Generalizability • Time & cost • Researcher expertise

  6. Methods • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

  7. Survey • Online / paper questionnaires distributed to target audience • Can be used to • tabulate quantitative data • gather qualitative feedback (opinions, feelings, etc.) • Useful at any time in study

  8. Survey • Pros • Easy to get a large number of responses. • Quick and easy to conduct. • Highly generalizable. • Cons • Self-selection. • Participants often only offer enough information to answer the question. • Can miss details. • Low in realism and precision.

  9. Interview • Evaluators formulate questions on the issues of interest. • Interview representative users, asking them these questions in order to gather information desired. • Interviewer reads questions to user, who replies verbally; interviewer records responses.

  10. Interview • Pros • Quick and easy to conduct. • Gives designer quick feedback on a range of ideas. • Can get a person’s initial reaction to an idea. • Can get detailed information from a person. • Cons • Often takes place away from natural setting. • Question wording or interviewer “body language” can bias answers. • High probability of false positives and missed problems (e.g., users may not have a clear idea of how an app will be used). • Can miss details if interviewer doesn’t know what issues to draw out.

  11. Controlled Lab Experiment • In lab, manipulate one feature of a system to assess the causal effects of the difference in that manipulated feature on other behaviors of the system. • Example: • in lab, show users 4 versions of a website: • blue, yellow, red, and black text • measure time to find specific words • compare

  12. Controlled Lab Experiment • Pros • Provides precise, quantifiable data. • Easier to draw inferences from data. • Relatively quick. • Can get a medium-sized number of participants. • Cons • Short duration of a lab experiment may not be enough to allow users to become accustomed to an app. • Not a natural setting – interaction may not be normal.

  13. In-lab Observation • Participants come to lab to "use" an interface • Given sample tasks to complete with it • Evaluators observe and possibly audio- or videotape • Participants may "think out loud" • Can use lo-fi prototype (for a project in the design stage) to an almost-complete interface • Evaluators note participants’ • emotions, exclamations, facial expressions, and other "qualitative" data • take note of quantitative data such as time to complete a task or number of errors

  14. In-lab Observation • Pros • Relatively quick. • Can get a medium-sized number of participants. • Cons • Observations are subjective and error prone. • Short duration of lab observation is not enough time for user to get accustomed to using the interface. • Not a natural setting – interaction may not be normal.

  15. Controlled Field Experiment • In natural setting, manipulate one feature of a system to assess the causal effects of the difference in that manipulated feature on other behaviors of the system. • Example: • Participants use 3 different input devices in their own office: mouse with 1, 2, or 3 buttons • Perform a set of tasks • Measure differences

  16. Controlled Field Experiment • Pros • Less intrusive than most other evaluation methods. • Provides more precise data than field observation. • Can observe natural behavior of user (though some part of the system will be controlled/unnatural). • Cons • More intrusive than field observation. • Less natural than field observation.

  17. Field Observation Study • Evaluator makes direct observations of “natural” systems • Takes care to not intrude on / disturb those systems • A.K.A. “ethnography”

  18. Field Observation Study • Pros • Only way to observe natural behavior of user & interaction between user & tools. • Cons • Difficult and time consuming. • Hard to get permission to observe people. • Observations are subjective and error prone. • Cannot make strong interpretations from observations. • Not very generalizable.

  19. Heuristic Evaluation • Pros • Quick and easy. • Cons • Nielson’s heuristics may not be as relevant to non-GUIs. • Results in false positives in missed problems, especially when experts are not part of target audience.

  20. Cognitive Walkthrough • Pros • Quick and easy. • Cons • Results in false positives and missed problems when evaluator is different from target audience.

  21. Automate Observation Study • Techniques include • video or audio recording of user • pop-up screens • screen shots • time logging • log users actions (collecting statistics about detailed system use)

  22. Automate Observation Study • Pros • Eases burden on observers for data collection & analysis. • Cons • Setup is often more time-consuming to complete. • Harder to get approved if it involves analysis of videotape or audiotape. • May miss nuanced/interpretive details.

  23. Experimental Simulation • In-lab experiment that is as much like some real situation as possible. • Example: • ground-based flight simulator • behaves as closely as possible to a real flight • still under researcher control

  24. Experimental Simulation • Pros • Still fairly precise. • More realistic than in-lab experiment. • Cons (same as lab exp.) • Short duration of a lab experiment may not be enough to allow users to become accustomed to an app. • Not a natural setting – interaction may not be normal.

  25. Claims Analysis • Claim = statement that a certain aspect (button, scrollbar) of a design has psychological implications reflected in how capable a user is in using that design • UI artifacts are listed along with their design features & pros/cons • Helps • select among alternative designs • clarify questions to be analyzed through user testing by stating how the design should work (in claims)

  26. GOMS • A method to describe user tasks and how a user performs those tasks with a specific interface design • Views humans as information processors • Small number of cognitive, perceptual, and motor operators characterize user behavior • To apply GOMS: • Analyze task to identify user goals (hierarchical) • Identify operators to achieve goals • Sum operator times to predict performance • GOMS = • Goals: What a user wants to accomplish • Operators: Cognitive or physical actions that change the state of the user or the system • Methods: Groups of goals and operators • Selection rules: Determine which method to apply

  27. GOMS • Pros • Predict human performance before committing to a specific design in code or running user studies • Many studies have validated the model (it works) • Cons • Assumes error-free, skilled user behavior • No formal recipe for how to perform analysis • Significant time investment

  28. Computer Simulation • Creating a complete & closed system that models the operation of the concrete system without users. • Example: • geophysical process going on in connection with the eruption of Mount St. Helens

  29. Computer Simulation • Pros • Supposedly high in realism (depends on accuracy of data/system replication) • Cons • Low in precision & generalizability

  30. Formal Theory • Formulating general relations (propositions, hypothesis, or postulates) among a number of variables of interest. • Pros • Relatively generalizable • Cons • Not realistic or precise

  31. How to chose a method? • Stage of study • Pros & cons • Realism • Precision • Generalizability • Time & cost • Researcher expertise • Metrics • Qualitative vs. quantitative • Research perspective

  32. Methods • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

  33. Early Stage • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

  34. Early Stage • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

  35. Iterative & Summative Stages • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

  36. Iterative & Summative Stages • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

  37. Realism • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

  38. Realism • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

  39. Precision • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

  40. Precision • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

  41. Generalizability • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

  42. Generalizability • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

  43. Time & Cost • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

  44. Time & Cost • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

  45. Researcher Perspective • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

  46. Metrics: examples • Traditional GUIs: • efficiency (time to complete task) • accuracy (# of errors) • simplicity • Peripheral Displays: • awareness (recall) • distraction (dual-task behavior) • aesthetics

  47. Peripheral Displays • Survey • Interview • Controlled-lab experiment • In-lab observation • Controlled field experiment • Field observation study • Heuristic Evaluation • Cognitive Walkthrough • Contextual Inquiry • Automated observation user study • Experimental simulation • Claims analysis • GOMS • Computer simulation • Formal theory

  48. Questions?

More Related