1 / 20

The utility of simple prototype tasks in understanding real-world design behavior

The utility of simple prototype tasks in understanding real-world design behavior. John C. Thomas IBM T.J. Watson Research Center CHI 2007 Design Workshop April 29, 2007. Design Science from Design Behavior. So costly to do controlled studies of real world scope that it is seldom done

lotte
Download Presentation

The utility of simple prototype tasks in understanding real-world design behavior

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The utility of simple prototype tasks in understanding real-world design behavior John C. Thomas IBM T.J. Watson Research Center CHI 2007 Design Workshop April 29, 2007

  2. Design Science from Design Behavior • So costly to do controlled studies of real world scope that it is seldom done • Evaluating real-world designs is difficult • Multi-dimensional • Time scale • Multiple stakeholders • Real world designs may often succeed or fail for “uninformative” reasons • Equating design experience relevant to task at hand is very difficult • Worked with J. Carroll in the 1970’s to study “Design” – people almost universally asked, “Design of What?” • If content trumps process, then design as an activity may change much faster than our ability to develop a “science” of design

  3. An Alternative: Prototypical Tasks • Prototypical tasks are derived from observations of real world design issues • Prototypical tasks are cheap enough to be run multiple times • Prototypical tasks can be completely quickly • Prototypical tasks can have measurable outcomes • Relevant experience can be relatively controlled

  4. Examples • Constraint combination

  5. Example: • “There are two locked boxes, each containing the others one and only unique key. There is no way to open either box except by using the key. Yet I am able to open both boxes. How is this possible?”

  6. Possible Phases to Model • Problem Finding • Problem Formulation • Requirements Prioritization • Requirements Organization • Idea Generation • Idea Integration • Testing Subsolutions • Iterative Refinement

  7. Cautions • Finding an effect on a prototype task is like an in vitro test; still needs to be verified in larger, more realistic contexts. • Prototypical tasks will not cover all aspects of design; e.g., tools whose purpose is to help coordinate across the phases of design or to facilitate long-term teamwork. • Need to develop “alternative forms” or keep answers from becoming too well known. • Implicitly thinking of “running shoes” but could be applied to “barbells”; suggest “stories” and non-verbal expressions as ways to test “skis.” • Example: in experiment on Dynamic Learning System, people who used query language spoke much more in terms of navigation and finding; people who used Learning System used many more terms concerning learning, cognition, and creativity. • Think about avoiding ecological “Gaps” (Thomas & Kellogg, 1989).

  8. To be used within a spectrum of methods • Open-ended observation of real design problems through invitations; e.g., dialogue captured of librarian and IT specialist • Multiple people solving design problems described with scenarios; e.g., churchrestaurant; library processes; web service • Well-structured but open-ended design problems; e.g., office layout, process scheduling • Prototypical tasks (new)

  9. Scoring open-ended designs • Experts rank order overall design and on various scales • Match score with expert solutions (“practicality” of design) • Match scores with other people’s solutions (“originality” of design) • In restaurant design task, e.g., SAT scores predicted overall “goodness”; within, a tradeoff between “practicality” and “originality.”

  10. References • Malhotra, A., Thomas, J., & Miller, L. (1980). Cognitive processes in design. International Journal of Man-Machine Studies, 12, 119-140. • Carroll, J., Thomas, J., & Malhotra, A. (1980). Presentation and representation in design problem solving. British Journal of Psychology, 71(1), 143-155. • Carroll, J., Thomas, J., & Malhotra, A. (1979). A clinical-experimental analysis of design problem solving. Design Studies, 1(2), 84-92. • Thomas, J. (1978). A design-interpretation analyusis of natural English. International Journal of Man-Machine Studies, 10, 651-668. • Thomas, J. & Carroll, J. (1978). The psychological study of design. Design Studies. 1(1), 5-11. • Thomas, J.C. and Kellogg, W.A. (1989). Minimizing ecological gaps in interface design, IEEE Software, January 1989. • Thomas, J., Lee, A., and Danis, C. Enhancing creative design via software tools, CACM, 45(10), 112-115. • Thomas, J. & Farrell, R. HCI techniques from Idea to Deployment: A case study for a dynamic learning environment. CHI 2006, Montreal.

  11. On-demand Custom Courseware Describe Assemble Capture Cleave Custom Course Learning Objects Customize Share Original Sources

  12. Design “Values” • Heard of as technique for making a story coherent in Story by Robert McKee • Also heard as technique used in European car company for design integrity • Design Values prominently posted above my ThinkPad and referenced during design decisions, interviews, and looking at test results

  13. E-Learning Personalized Modern Useful Fun Flexible Targeted Efficient Correct Authoritative Under my (i.e., the user’s) control Easy to Use Worthwhile Secure Private Forest Glen Waterfall Springtime Sunny Dynamic Safe Active Design “Values”

  14. Initial Iterations • Made initial design in presentation SW • Three iterations, each with a small number of users (3-6) • Gave each user a motivating scenario for learning based on dilemma for “Joe” • For each screen, asked what they saw, what each button would do, which action they would take, then “reset” to sequence by saying what Joe actually did • Asked HCI experts in company for “heuristic evaluation”

  15. Design Results • CCC produced more design behavior; e.g., 6.5 pages vs. 1.83 (t(24)), p < .05. • More experienced subjects produced significantly more design behavior (pages, words, boxes, arrows) • CCC designs had more features matching an expert solution (n.s.); correlated with experience r=.463, p<.05; if corrected for experience, CCC t(22)=2.35, p<.03.

  16. Qualitative Design Results • Three experts rated each design on Presentation, Level of Detail, Accuracy, Completeness and Depth of Understanding • Graders agreed, r=.72, p<.0001 • CCC produced “better” designs in the opinion of each rater, but not significantly so • If adjusted for experience, more “good” designs in CCC than Query Only Chi Square(1)=4.43, p<.05.

  17. Other Findings • Custom Course Diversity • 29 LO visited by one subject each • No LO visited by every subject • Every course different, but every course relevant to the scenario • Subjectively, it appeared that choices were related to experience level • Behavior during experiment • CCC used significantly more words related to cognition • Query Only used significantly more words related to navigation • CCC used fewer queries (5.0 vs. 12.5, t(20)=2.16, p<.05) based on server log data • Query Only made longer free form comments than CCC 60.6 words vs. 28.9 t(21)=2.13, p<.05 • Groups did not differ in subjective scale; however, subjective satisfaction correlated with experience • r = .601, t(21)=3.445, p<.0024 (two-tailed)

More Related