1 / 15

Research Design and Evaluation

Research Design and Evaluation. Colin Ware. Goals for empirical research. Uncover fundamental truths and test theories (late stage science).

leisad
Download Presentation

Research Design and Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Research Design and Evaluation Colin Ware

  2. Goals for empirical research • Uncover fundamental truths and test theories (late stage science). • Discover the nature of the world. Find out about how people interact with information artifacts. Find out how people respond to a particular display. (survey of recreational boaters). • Does theory apply in practice. E.g. good continuity applied to flow vis. • Compare two (or more) display or interaction methods. • Is this method adequate?

  3. Continued • Compare two entire systems. (GIS 1, vs GIS 2) • Measure task performance (e.g how fast can we find a certain class of patterns). • Determine user preferences. _________________________________

  4. Non-empirical contributions • Invent a better interface for some particular domain (Social network analysis, interactive flow visualization) • Proof of concept prototype • Constructivist analysis • Invent a better evaluation method • Invent a new “theory” (information foraging theory)

  5. Methods for evaluation • Measure errors (how accurate are we) • Target detection (rapid response) Binary data • Method of adjustment • Measure time to perform a task • How much work can we do per unit time. • Rating scales (Likert scales) • Turn judgments into numbers. • Task analysis + Structural analysis. • A cognitive theory is constructed with observation as evidence (constructivism). (e.g. narrative). • Goes hand-in-hand with proor of concept applications

  6. Semi-stuctured interviews. • Expose the user to a problem + solution in a systematic way. (+questions) • Wizard of Oz • Cognitive walkthroughs. • Have the user perform a task and talk aloud as they do it.

  7. Spiral design method Must document the design process

  8. Spiral design • One way of proceeding is though a spiral design method. • Construct a rapid prototyping environment (tool kit) – demo or die • Construct a prototype • Evaluate (and document findings) • Refine prototype • Evaluate (and document findings)

  9. Methodological Issues • If you want to measure time/error data • Best to have a simple representative task • Justify it. • If you want to get opinions (ratings, interviews). Domain experts often ideal. But they may have established ideas – they like the status quo.

  10. Determine data and problem Determine tasks Create mappings from data to representation such that important patterns are “transparent”. Create interfaces for rapid task performance Create support for hypothesis Information interface design

  11. Things to consider • Aim for important targets. • Keep it simple • Maximize payoffs for (your) time expended • Be sure to have control conditions. • Defacto standard • Alternative display methods. • Choose dependent variable ranges. • Remove all un-necessary variables (e.g. background). • Have sufficient observations.

  12. Statistics • Means. • Standard deviations. • Standard error of the mean = s/sqrt(N); • 2se ~ 95% confidence intervals. • ANOVA e.g. color and motion.

  13. The End

  14. Velocity glyphs Curved Compact Show direction clearly Easily readable: more looks “greater” Capable of 100 levels

  15. Patterns of common low-level tasks Relating patterns- e.g. bioinformatics. Pattern searching Finding paths Finding groups Finding target symbols Visualization design pattern

More Related