Inferential and Expressive Capacities of Graphical Representations

1. Tutorial Inferential and Expressive Capacities of Graphical Representations Survey and Some Generalizations Diagrams 2004University of Cambridge March 23, 2004

2. Atsushi Shimojima School of Knowledge Science Japan Advanced Institute of Science & Technology ATR Media Information Science Labs

3. To understand three concepts useful to capture the inferential-expressive capacities of many graphical systems. In my personal terminology: Free ride Over-specificity Derived meaning Purpose

4. but Their exact contents seldom defined thus Different people used different terms to refer to them, sometimes missing important connections of their ideas and findings Why Important? These concepts are very often alluded to in the literature Their ranges of application never explicated in full

5. Plan for the hour Free ride Examples Analysis & definition Connections Over-specificity Derived meaning Outstanding questions

6. 1. Free Ride

7. A toy example: • Jon, Ken, Gil, Bob, and Ron run races • of the kind with no “ties” in arrival Suppose:

8. Defeated(Jon,Bob) Jon Bob Jon defeated Bob. Compare: • Different ways of expressing the information that Jon defeated Bob :

9. Defeated(Jon,Bob) Jon Bob Jon defeated Bob. • Atomic sentence of a first-order language (FOL) with: • two-place predicate Defeated • its arguments Jon and Bob

10. Defeated(Jon,Bob) Jon Bob Jon defeated Bob. • Representation of PD system(position diagrams) where: • Horizontal relation of names indicate arrival order of people.

11. Defeated(Jon,Bob) Jon Bob Jon defeated Bob. Sentence of English describing the arrival order of two people.

12. Syntactic rules: Two or more of the names “Jon”, “Ken”, “Gil”, “Bob”, and “Ron” appear in a horizontal row. The same name appears at most once. Semantic rules: If the name X appears to the left of the name Y, the bearer of X defeated the bearer of Y. PD system (a bit more precisely)

13. FOL Defeated(Jon,Bob) PD Jon Bob English Jon defeated Bob. Look Similar But behave quite differently when more information expressed

14. FOL Defeated(Jon,Bob) & Lost_to(Ken,Bob) PD Jon Bob Ken Difference 1 Express information: Jon defeated Bob. Ken lost to Bob. English Jon defeated Bob and Ken lost to Bob.

15. PD Jon Bob Ken The PD system expresses an additional piece of information… Express information: Jon defeated Bob. Ken lost to Bob.

16. FOL Defeated(Jon,Bob) & Lost_to(Ken,Bob) Express information: Jon defeated Bob. Ken lost to Bob. English Jon defeated Bob and Ken lost to Bob. …while FOL and English don’t.

17. In PD, expressing certain sets of information results in the expression of additional, consequential information. = Free rides

18. As As As Bs Bs Bs Cs Cs Cs Another example: Venn diagrams Express information: All As are Bs. No Bs are Cs. Expressing certain sets of information results in the expression of additional, consequential information

19. C A A B B Another example: Euler diagrams Express information: A ⊂ B C ∩ B = φ Expressing certain sets of information results in the expression of additional, consequential information

20. Another example: Maps Express information: B’s house is in front of F’s house across the river. Expressing certain sets of information results in the expression of additional, consequential information

21. Sloman (1971) Of course we cannot always do the manipulations in our heads: we may have to draw a diagram on paper, or re-arrange parts of a scale model, in order to see the effects.…The main point is that the ability to apply such subroutines to parts of analogical configurations enables us to generate, and systematically inspect, ranges of related possibilities, and then…to make valid inferences, for instance about the consequences of such possibilities. (p. 220.)

22. Barwise & Etchemendy (1990) Diagrams are physical situations….As such, they obey their own set of constraints…By choosing a representational scheme appropriately, so that the constraints on the diagrams have a good match with the constraints on the described situation, the diagram can generate a lot of information that the user never need infer. Rather, the user can simply read off facts from the diagram as needed.

23. Larkin & Simon (1987) We have seen that formally producing perceptual elements does most of the work of solving the geometry problem. But we have a mechanism---the eye and the diagram---that produces exactly these “perceptual” results with little effort. We believe the right assumption is that diagrams and the human visual system provide, at essentially zero cost, all of the inferences we have called “perceptual.” As shown above, this is a huge benefit. (p. 99.)

24. Non-deductive representation systems where the operation of the construction process entails the “making” of the inferences (Lindsay 1988, p. 112) Inference by recognition(Novak 1995) Inference by inspection and transformation(Olivier 2002, p. 72--74) Emergence effect(Kulpa 2003, p. 90) Emergent properties(Koedinger 1992, as cited by Olivier 2001) Emergent relations(Chandrasekaran, Kurup, and Banerjee 2004) Other conceptions

25. But What, more exactly, is the “free-ride” capacity? What is the general condition---semantic mechanism---for a system to have that property?

26. Basic Assumption A representation X expresses information about the represented object Y by having a property that indicates the corresponding property of Y. a property Represented object Y indicates Representation X a property

27. Example: PDs [Jon defeated Bob] a particular running race indicates a particular position diagram [the name “Jon” appears to the left of the name “Bob”] Jon Bob

28. A B Example: Euler diagrams [A ⊂ B] a particular group of objects indicates a particular Euler diagram [the circle “A” appears inside the circle “B”]

29. constraint Ken lost to Bob. Jon defeated Bob. indicates indicates indicates constraint Condition for Free Ride: PD system Jon defeated Ken. The name “Jon” appears to the left of the name “Bob” The name “Ken” appears to the right of the name “Bob” The name “Jon” appears to the left of the name “Ken”.

30. constraint indicates indicates indicates constraint Condition for Free Ride: Euler Diagram C ∩ B = φ A ⊂ B C ∩ A = φ The circle “C” and the circle “A” has no overlap. A circle “C” and the circle “B” has no overlap. A circle “A” appears inside a circle “B”.

31. indicates indicates indicates Condition for Free Ride: General (Shimojima 1996a, 1996b) constraint ……… ……… constraint Constraints on representations themselves track constraints in the represented domain.

32. Thus: A system with a free-ride property supports deductive inference through physical manipulation of representations on an external display, not in the head. A (paradigm) case of distributed cognition

33. Connection: AI systems Some AI systems utilize the free-ride capacities of graphical systems by installing some manipulation-inspection abilities on diagrams. • WHISPER for the prediction of the collapsing of objects (Funt 1980) • REDRAW I & II for the deflection shape problem (Tessler, Iwasaki, and Law 1995a, 1995b) • KAP for the prediction of the movements of cam-follower pairs and meshing gears (Olivier, Ormsby, and Nakata 1996) • DRS component (Chandrasekaran et al. 2004)

34. Connection: graphical simulations Some graphical simulations can be considered free rides in an extended sense, combining computer-controlled dynamic constraints with geometrical-topological constraints on graphics. • Dynamic behaviors of strings, flexible rods, and rings, falling in free space, etc. (Gardin and Meltzer 1995) • Liquid behaviors in and out of containers with complex shapes (Decuyper, Keymeullen, and Steels 1995)

35. Connection: studies of sketching in design Free-ride capacities may be an essential factor of the utility of pictorial sketches in design process.

36. Schoen (1982) Each move is a local experiment which contributes to the global experiment of reframing the problem. Some moves are resisted (the shape cannot be made to fit the contours), while others generate new phenomena.As Quist reflects on the unexpected consequences and implications of his moves, he listens to the situation’s back talk, forming new appreciations which guide his further moves. (p. 94.)

37. Lawson (1997) Thus the drawing represents a sort of hypothesis or “what if” tool. With a plan, for example, the architect can say, what if the kitchen were here, the dining-room next to it and the living-room there? How could I then organize the entrance and the stairs? (p. 242 , colored emphasis by me.)

38. Also: • Goldschmidt (1994) • “One reads off the sketch more information than was invested in its making” (p. 164) • Such reading-off of unexpected reading-off as an essential step in “interactive imagery” in design • Suwa, Gero, and Purcell (2000) • Relationship between unexpected discoveries in sketches and invention of new design requirements

39. Warning: Recognition problem Free rides only guarantee the expression of consequential information in the representation, not its recognition by the user. • “Cheap rides” (Gurr, Lee, and Stenning 1998, Gurr 1999) • Expertise in diagram construction to facilitate the recognition of useful consequences (Novak 1995)

40. 2. Over-Specificity

41. FOL Defeated(Jon,Bob) & Defeated(Ken,Bob) Difference 2 Express information: Jon defeated Bob. Ken defeated Bob. ? ? PD Jon Ken Bob Ken Jon Bob English Jon defeated Bob and Ken defeated Bob.

42. Express information: Jon defeated Bob. Ken defeated Bob. ? ? LD Jon Ken Bob Ken Jon Bob The PD system can’t express the info without additional info…

43. FOL Defeated(Jon,Bob) & Defeated(Ken,Bob) Express information: Jon defeated Bob. Ken defeated Bob. English Jon defeated Bob and Ken defeated Bob. …while FOL and English can.

44. In PD, certain sets of information cannot be expressed without expressing additional, non-warranted information. = Over-specificity

45. C C C C Another example: Euler diagrams Express information: A ⊂ B C ∩ B ≠ φ How do you place This circle? A A B B Certain sets of information cannot be expressed without expressing additional, non-warranted information.

46. K Another example: Maps Express information: K’s house is between A’s house and B’s house. Where do you place this icon? Certain sets of information cannot be expressed without expressing additional, non-warranted information.

47. Stenning and Oberlander (1995) Specificity = ``the demand by a system of representation that information in some class be specified in any interpretable representation'' (p. 98)

48. Lawson (1997) However, there are some ways in which a picture can often carry too much information or indicate a degree of precision which may be inappropriate….It would be difficult to construct a drawing which did not suggest other features of the form of the finished product which might restrict a future designer. (p. 242.)

49. Aristotle (350 B. C. E.) [In geometrical proofs,] though we do not for the purpose of the proof make any use of the fact that the quantity in the triangle (for example, which we have drawn) is determinate, we nevertheless draw it determinate in quantity (As cited by Kulpa 2003, p. 101.)

50. Analog property of representation systems as opposed to digital property (Dretske 1981) Smaller degree of discretion(Norman 2000, p. 110) Particularity feature(Kulpa 2003, p. 96, p. 101) Other conceptions Not in the sense of Goodman (1982)!