Keeping track of objects while exploring an informationally impoverished environment:

1. Keeping track of objects while exploring an informationally impoverished environment: Local deictic versus global spatial strategies Nicolas J. Bullot a, b, c , Jacques Droulez c , Zenon W. Pylyshyn a a Rutgers Center for Cognitive Science, Rutgers University, 152 Frelinghuysen Road,Piscataway, NJ 08854-8020, USA. b Institut Jean Nicod, CNRS-EHESS-ENS, 1 bis avenue Lowendal, 75007 Paris, France. cLaboratoire de Physiologie de la Perception et de l’Action, Collège de France, 11 place Marcelin Berthelot, 75005 Paris, France. - Corresponding author: Nicolas J. Bullot. E-mail address: nicolas.bullot@ruccs.rutgers.edu. - Internet site: http://www.objectcognition.net/NJB/index.html - MTSP demos (small movies) are available at : http://nicolas.bullot.free.fr/Nicolas.J.Bullot/MTSP_Base1/MTSPd1.html

2. 1. Abstract • This study investigates a new experimental paradigm called the Modified Traveling Salesman Problem (Bullot & Droulez, submitted). This task requires subjects to visit once and only once each of n invisible targets in a 2D display, using a virtual vehicle controlled by the subject. Subjects can only see the directions of the targets from the current location of the vehicle, displayed by a set of oriented segments that can be viewed inside a circular window surrounding the vehicle. • Two conditions were compared. In the “allocentric” condition, subjects see the vehicle move across the screen and change orientation under their command. The “egocentric” condition is similar except for how the information is provided: the position and orientation of the vehicle icon remains fixed at the center of the screen and only target directions, as indicated by the oriented segments, change as the subject “moves” the vehicle. The unexpected finding was that this task can be performed, in either condition, for up to 10 targets.

3. Abstract (continued) • We consider two possible strategies that might be used, a location-based strategy and a segment strategy. The location-based strategy relies on spatial memory and attempts to infer the locations of all the targets. The segment strategy is more local and focuses on the directional segments themselves, keeping track of the ones that represent to-be-visited or already-visited targets. • A number of observations suggest that the segment strategy was used, at least for larger numbers of targets. According to our hypothesis, keeping track of the segments requires one to use indexical reference for associating the segments with their status in the task – given by current status predicates Visited(x) or Not-visited(x) –, perhaps using visual indexes (Pylyshyn 2001), deictic pointers (Ballard et al. 1997), or object files (Kahneman et al. 1992).

4. 2. Motivation • This work was carried out in the context of research on spatial memory and visual inferences (Allwein & Barwise 1996; Campbell 2002; Evans 1982; Ullman 1984). • More specifically, it relates to research on: • The use of visual indexicals or indexes linking with distal token individuals (Ballard et al. 1997; Pylyshyn 2001) in situated tasks. • The ability to keep track of distal targets through recall of target locations (McNamara in press; Posma & De Haan 1996).

5. Motivation (continued) • We frequently have to maintain cognitive contact with objects around us even though we cannot see them. • How do we keep track of target objects when information about the objects’ location is partial or indeterminate? • How many (stationary) targets can one keep track of while moving among them, when only directions of the targets are known (and an indication of when the targets are actually encountered) ? • We studied these questions with the new Modified Traveling Salesman Problem (MTSP) paradigm.

6. 3. The Modified Traveling Salesman Paradigm • In the MTSP experimental paradigm subjects are instructed to (1) move a virtual vehicle so as to visit each invisible targetonce and only once, and (2) try to minimize the distance traveled. • Only target directions from the current vehicle location are shown by radial segments displayed inside a circular window (see figure, next frame). When a target is encountered, the segment changes color. • Subjects can draw different kinds of visual inferences by scrutinizing the motion of directional segments, which co-vary with their motor commands.

7. The MTSP task : window surrounding the vehicle within which target directions are visible Instructions: during a simulated navigation with the vehicle, (1) visit once and onlyonce each of 6, 8 or 10 invisible targets and (2) try to minimize traveling distance : visible vehicle controlled by subjects : direction of vehicle translation : visible target direction : invisible target to be visited : invisible target direction

8. 4. Visual inferences From the information provided to subjects they may: • Select segments and/or target locations in a deictic manner (c.f., Ballard, Hayhoe, Pook & Rao, 1997). • Draw inferences concerning possible locations of targets (in the allocentric condition this would be in screen coordinates). • Tag, or assigncurrent-status predicatesto the representation of targets x, e.g., Visited(x) or Not-Visited(x). • Estimate the relative distance and perhaps even the time-to-contact of a particular target x, on the basis of the angular velocity of the directional segment pointing toward x. The inference rule might be something like: if this s1 which points towards x1 moves faster than s2, s3, s4, then the vehicle is closer to x1than to x2, x3, and x4.* s * If Θ is the angle between straight-ahead and the direction of the segment pointing to some target, and if the vehicleis moving at a constant velocity v, then the closer the object is the faster will the angle be changing. Although this does not tell the observer how far the vehicle is from the object, it does provide some guidance about the relative distance to various objects (for small nonzero Θ). r θ

9. 5. Conditions To study the visual inferences adopted by subjects to solve the MTSP task, we examined two different conditions. • In the ”allocentric condition”, targets are fixed with respect to the screen (see figure, not to scale), and the vehicle icon moves both in translation and rotation according to the subject’s commands. In this condition, subjects may (a) use the screen borders as the relevant landmarks(screen-target and inter-target spatial relations are fixed), (b) infer target positions on the screen from the cues provided and (c) construct in memory a spatial representation of target locations with respect to the screen reference-system.

10. In the “egocentric condition”, the vehicle icon and arrow remain fixed at the center of the screen (frontal direction upward). Only directional segments move according to subject’s commands, as if subjects were directly moving the display beneath their apparently stationary vehicle. In this condition, subjects cannot use the screen borders as the main relevant landmark, because target spatial locations on the screen are no longer fixed (even though target-target spatial relations were stable and coherent). • Thus, available strategies are different in the two conditions. • Because the egocentric condition does not allow recovery of target locations in the screen frame of reference one would expect this condition to be very difficult, if not impossible. At the very least one would expect very different performance in the two conditions.

11. 6. Method • Subjects controlled the movement of a vehicle displayed on a computer screen, in order to visit a set of n targets (n = 6, 8, 10). • 2 vehicle movements were possible: rotations (CW or CCW) and translation, indicated by the arrow. Only the target directions with respect to the controlled vehicle were displayed, so subjects had no direct way to evaluate target distances or locations (e.g. in x,y coordinates). • 9 subjectstook part in the experiment, 5 males and 4 females. • This task was performed under 6 experimental conditions: 3 target set sizes (6, 8 or 10) and two different reference frames: “allocentric” or “egocentric”.

12. 7. Experimental Results of the MTSP task

13. 8. Discussion • We found that subjects were able to carry out the task correctly on more than 70% of the trials, in both conditions! • Question: How do they do it? • With 8 and 10 targets the task should have been beyond their memory capacity! • Consider two possible strategies : A global and a local strategy. • Both strategies are what Ballard et al. refer to as Deictic Strategies.

14. The global strategy (location-based): • In the Global Spatial Strategy (GSS) subjects infer and store distal target locations in an allocentric frame of reference. • But, given working memory limitations (e.g., Cowan 2000, Logie 1995), and limitation in visual tracking (Pylyshyn & Storm 1988, Pylyshyn 1989), GSS should fail for high number of targets (n = 8, 10). • Moreover, GSS would not be available at all for solving the task in egocentric condition (see the description of the egocentric condition).

15. The local strategy: • In the Local Deictic Strategy (LDS), subjects do not compute target locations. Instead they group and track local/proximal cues, i.e. subsets of oriented segments. • To overcome tracking and working memory limits, they may group subsets of segments. • Such grouping or “chunking” strategies are often used when storage capacity is exceeded (Miller, 1956; Mandler & Shebo, 1982; Cowan 2000). Because vehicle motion is under subjects’ control it may be possible to switch between the chunks and the individual segments.

16. The local strategy (continued) • For example, • With 8 targets (x1 to x8) and 8 segments (s1 to s8), a subject might first group 4 adjacent segments on one side, or segments that happen to be close together into one chunk (set A, s1 to s4), and track the 4 others (set B, s5 to s8). • Subjects might then switch to tracking the individuals in set A, treating set B (s5 to s8) as a group. • By switching between groups and individual targets subjects might be able to use a time-sharing strategy to overcome the limit on how many targets can be tracked (Pylyshyn & Storm 1988). • To avoid losing track of segments that cross over one another, subjects would have to switch between groups and individuals sufficiently quickly to anticipate that a pair of segments were about to cross and select those segments to track.

17. 9. Conclusions • Subjects were able to carry out the MTSP task on more than 70% of the trials, in both conditions, with 6, 8 and 10 targets. • Visual inferences based on Global (or location-based) strategy might explain performance for the smallest number of targets (n = 6), but does not explain performance for high number of targets (n = 8, 10) and for the egocentric condition. • Visual inferences based on the Local (or the segment-based deictic) strategy might explain performance for high number of targets (n = 8, 10) in the large-nallocentric condition and in the egocentric condition, although it would require a chunking and time-sharing process that has not been modeled in detail.

18. 10. References • Allwein, G., & Barwise, J. (Eds.). (1996). Logical reasoning with diagrams. New York: Oxford University Press. • Ballard, D. H., Hayhoe, M. M., Pook, P. K., & Rao, R. P. N. (1997). Deictic codes for the embodiment of cognition. Behavioral and Brain Sciences, 20(4), 723-767. • Campbell, J. (2002). Reference and Consciousness. Oxford: Clarendon Press. • Cowan, N. (2000). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185. • de Vega, M., & Rodrigo, M. J. (2001). Updating spatial layouts mediated by pointing and labeling under physical and imaginary rotation. European Journal of Cognitive Psychology, 13(3), 369-393. • Evans, G. (1982). The Varieties of Reference. Oxford: Oxford University Press. • Kahneman, D., Treisman, A., & Gibbs, B. J. (1992). The reviewing of object files: Object-specific integration of information. Cognitive Psychology, 24(2), 175-219. • Karn, K. S., & Hayhoe, M. M. (2000). Memory representations guide targeting eye movements in a natural task. Visual Cognition, 7(6), 673-703. • Logie, R. H. (1995). Visuo-Spatial Working Memory. Hove, Hillsdale: Lawrence Erlbaum Associates. • Mandler, G., & Shebo, B. (1982). Subitizing: An analysis of its component processes. Journal of Experimental Psychology: General, 111, 1-22. • McNamara, T. P. (in press). How are the locations of objects in the environment represented in memory? In C. Fresksa & C. Brauer & C. Habel & S. Wender (Eds.), Spatial Cognition III (tentative title): Springer. • Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81-97. • Posma, A., & De Haan, E. H. F. (1996). What was where? Memory for object locations. The Quarterly Journal of Experimental Psychology, 49A(1), 178-199. • Pylyshyn, Z. W. (1989). The role of location indexes in spatial perception: a sketch of the FINST spatial-index model. Cognition(32), 65-97. • Pylyshyn, Z. W. (2001). Visual indexes, preconceptual objects, and situated vision. Cognition, 80, 127-158. • Pylyshyn, Z. W., & Storm, R. W. (1988). Tracking multiple independents targets: Evidence for a parallel tracking mechanism. Spatial Vision, 3, 179-197. • Ullman, S. (1984). Visual routines. Cognition, 18, 97-159.

19. Acknowledgements • We wish to thank the following persons and researchers for their help with the MTSP paradigm: Roberto Casati, Jérôme Dokic, Jeslan Hopkins, Joëlle Proust, Mark Wexler and the members of the ‘Visual Attention Laboratory’ at Rutgers Center for Cognitive Science. • This research was supported in part by “Action Incitative Cognitique” grant n°0693 from the French Ministry of Research.