Thinking with Visualizations: Cognitive Execution of Visual Queries

1. Thinking with Visualizations:Cognitive Execution of Visual Queries Colin Ware Data Visualization Research Lab University of New Hampshire Designing with cyborgs in mind

3. Architecture for visual thinking 10+ billion neurons Parallel,automatic Interaction Loop

5. Stage 2 Pattern perception 10 billion neurons Parallel,automatic Interaction Loop

6. Stage 2 Pattern perceptionVisual queries are executed by finding patterns in displays • Gestalt principles • Proximity • Continuity • Connectedness • Closure

7. Connection or Relationship Basic patterns Outlier Overlap Groups Uptrend Identity Contrast = quantity Size = quantity

8. Dual Coding Theory Visual Structural Information Verbal-Propositional Information Baddeley Verbal Working Memory Central Executive Visual Working Memory Visuo-Spatial Sketchpad Visual input Verbal input Words and symbols

9. Pictures and Words • When should we use a visual display? • What is a visual language?

10. Visual and verbal pseudo-code • While letters in stack • Take a letter • Put a stamp on it • Put it in the ‘out tray’ While (text in input) { getline; change characters to upper case; write to output; } Visual programming languages have a history of failure

11. Capacity of visual working memory (Vogal, Woodman, Luck, 2001) 1 second • Task – change detection • Can see 3.3 objects • Each object can be complex

12. Sequential comparison task

17. Change Blindness Simons and Levin

18. Architecture for visual thinking 10 billion neurons Parallel,automatic Selective Tuning Tsotsos Object Files Interaction Loop

19. Gist Semantic content

20. Visual search

21. Eye movements • Two or three a second • Preserves Context • The screen is a kind of buffer for visual ideas – we cannot see it all at once but we can sample it rapidly

22. Thinking visuallyEmbedded processes • Define problem and steps to solution • Formulate parts of problem as visual questions/hypotheses • Setup search for patterns • Eye movement control loop • IntraSaccadic Scanning Loop (form objects)

24. Problem • Trip Port Bou- Calais (5 days 3 cities) • Visual Problem Mayor Highways – Distance < 1.2 min = red smooth path • Eye movements to identify major candidate pathways • Pattern Identification: smooth, red, connected segments / reject non-red-wrong direction • Part solutions into vwm – spatial markers • Parts may be handed to verbal wm

25. Software Engineering Example - with Graph Representation • Segment Big Module into parts • High Cohesion (semantics) • Low Coupling • Find highly connected subgraphs with minimal links • Scan for candidate patterns • Look for Low connectivity • Look for Semantic similarity (symbols) • Important question: what are relevant pattern that can fit in vwm

26. Cost of Knowledge • Intra-saccade (0.04 sec) (Query execution) • An eye movement (0.5 sec) < 10 deg : 1 sec> 20 deg. • A hypertext click (1.5 sec but loss of context) • A pan or scroll (3 sec but we don’t get far) • Walking (30 sec. we don’t get far) • Flying (faster can be tuned) • Zooming, fisheye, DragMag

27. Walking Flying (30 sec +) Naïve view that does not take perception or the cost of action into account. Semnet GraphVisualizer3D ConeTrees

28. How to navigate large 2 ½D spaces? (Matt Plumlee) Zooming Vs Multiple Windows • Key problem: How can we keep focus and maintain context. • Focus is what we are attending to now. • Context is what we may wish to attend to. • 2 solutions: Zooming, multiple windows

29. When is zooming better thanmultiple windows • Key insight: Visual working memory is a very limited resource. Only 3 objects GeoZui3D

30. Task: searching for target patterns that match

31. Cognitive Model (grossly simplified) • Time = setup cost + number of visits*time per visit • Number of visits is a function of number of objects (& visual complexity) • When there are too many multiple visits are needed

32. Prediction Results As targets (and visual working memory load) increases, multiple Windows become more attractive.

33. Lessons for design • Tells us when we need extra windows • What do we need to keep on the screen at the same time • Simultaneous vs successive views

34. Need low cost and low cognitive cost interactions Constellation: Hover queries (Munzner) MEGraph Brushing Dynamic Queries

35. Architecture for visual thinking Low -Medium level – pattern perception High level vwm and cognitive costs Appropriate representation (words vs images) Navigation costs Interaction Loop

36. Cognitive Systems • Slogan: “Tighten the loop” • Take a systems approach • Visualization for pattern finding • Take into account the non-homogeneity of space. • Algorithms for pre-filtering • Optimize navigation – brushing, dynamic queries – breadth first range searches • Large high resolution displays

37. Acknowledgements • NSERC (Canada) • NSF (USA) • NOAA • ARDA

39. 3D versus 2D 11.4% errors 3.7 sec sub-structure 21% errors 5.1 sec sub-structure 20% memory errors 34% memory errors

40. Research topicWhat are easy visual queries Easy= single object comparison in vwm