Conscious experience and the picture-theory

1. Given the similarly of vision and visual imagery, does this suggest that imaging involves a mental picture? • Reasoning from the involvement of the visual system in mental imagery to the conclusion that the format of the image must be pictorial, is based on a thoroughly discredited theory of vision (the picture theory of vision). • There are serious reasons to doubt that vision involves the construction of a pictorial display

2. Conscious experience and the picture-theory The picture theory was meant to explain why our perceptual experience is panoramic and stable while the visual inputs are partial and constantly changing. This assumes that the content of experience is represented. • But the picture theory of vision has been thoroughly discredited: There is no rich panoramic display in vision (e.g., see change blindness, superposition studies, …)

3. Vision appears to fill in missing parts of a familiar pattern Find three of these … In here….

4. But superposition of images has been discredited O'Regan & Lévy-Schoen, 1983

5. We encode far less visual information than we generally believe we do – far from what any picture would hold Harborside Airplane

6. Signature properties of vision do not apply to off-retinal information

7. Signature properties of vision do not apply to off-retinal information

8. Off-retinal information uses memory Effectiveness of anorthoscopic presentation depends on memory load

9. There are many questions about what goes on when we have the experience of “seeing with the mind’s eye” • Is mental imagery a special form of thought? If so, in what way is it special? • Are mental images sensory-based and modality-specific? • Are mental images like pictures? In what respect? • Are images different from other forms of thought? Do they resemble what they represent? • Does mental imagery use the visual system? If so, what does that tell us about the format of images? • Is there neurophysiological evidence for a pictorial “display” in visual cortex? • What if a display were found in human visual cortex?

10. A central methodological point:The difference between appeals to properties of the architecture or to properties of what is represented Suppose we observe some robust behavioral regularity. What might it tell us about the nature of the system that exhibits the regularity? • e.g., in the case of human behavior, what does it tell us about the cognitive architecture or the format of representations?

11. An illustrative example: Behavior of a mystery box What does this behavior pattern tell us about the nature of the box?

12. The moral of this example: Regularities in behavior may be due to one of two very different causes: The inherent nature of the system (to its structure or wiring), or The nature of what the system represents (what it “knows”).

13. The “imagery debate” The major difference between picture-theorists and the rest of us (me) is in how we answer the following question: • Do experiments on mental imagery (such as those I will review) tell us anything about the format of images or about the properties of a special imagery architecture? Or … • Do the experiments tell us about the knowledge that people have about how things would look if they were actually to see them (together with some common psychophysical skills)? • While these are the main alternatives, there are also other reasons why experiments come out as they do.

14. Examples to probe your intuition Imagine various events unfolding before your “mind’s eye” – • Imagine a bicyclist racing up a hill. Down a hill? • Imagine turning a large heavy wheel. A light wheel. • Imagine a baseball being hit. What shape trajectory does it trace out? Where would you run to catch it? • Imagine a coin dropping and whirling on its edge as it eventually settles. How does it behave? • Imagine a heavy ball (a shot-put) being dropped at the same time as a light ball (a tennis ball). Indicate when they hit the floor. Repeat for different heights.

15. Why did things unfold the way they did in your imagination? • Was it because of the format of your image or your cognitive architecture? Or because of what you know? • Did it reveal a capacity of mind? • Or was it because you made it do what it did? • Could you make your image have any properties you choose? Or behave in any way you want? Why not? • How about imagining an object from all directions at once, or from no particular direction? • How about imagining a 4-dimensional object? • Can you imagine a printed letter which is neither upper nor lower case? A triangle that is not a particular type?

16. More examples of the behavior of images: Do they tell us about tacit knowledge or about the nature of the architecture or format • Color mixing • Conservation of volume Or, more relevant to the present discussion: • The effect of image size • Scanning mental images

17. Studies of mental scanningDoes it show that images have metrical space? (Pylyshyn & Bannon. See Pylyshyn, 1981) • Conclusion: The image scanning effect is Cognitively Penetrable • i.e., it depends on goals and beliefs, or on Tacit Knowledge.

18. Another methodological note • What is tacit knowledge? • A central construct in cognitive science: Cognitive phenomena turn on goals and beliefs • Tacit knowledge need not be consciously accessible but may be inferred indirectly • Access to tacit knowledge is context-dependent • Tacit knowledge of what? • It does not refer to knowledge of what the experimenter wants nor the expectation of how the experiments will turn out (cf Dennis). It refers to knowledge of what it would be like to see the relevant situation as it unfolds. • Cognitive penetrability as a litmus

19. Major Question 2: Does visual imagery use the visual system? • It depends on what you mean by “visual” and by “use” • vision can involve all of cognition • use vision can mean actually perceive the image • The reason this question is of interest to picture theorists is that a positive answer is thought to entail that images are picture-like. • Let’s assume that “the visual system” is “active” in some way during episodes of mental imagery. • What follows from that? • Why should we think that vision may be involved? • Note that the main support for the assumption that vision is involved comes from neuroscience – to be considered later.

20. Reasons for thinking that images are interpreted through the visual system • Similar phenomenology of imagining & seeing • This is the absolutely central reason which overshadows all others • Superposition & interference studies • Visual illusions with projected images • The ubiquitous role of attention • Reperceiving and novel reconstruals • A large but very problematic literature • Chambers, Finke, Peterson, Slezak

21. More persuasive demonstrations • Images constructed from descriptions • The umbrella example(s) • The two-parallelogram example • Amodal completion • Reconstruals: Peterson vs Slezak

22. Imagine two parallelograms

23. Imagine two parallelograms

24. Imagine two parallelograms

25. Amodal completion by imagery?

26. Amodal completion by imagery?

27. Vision is involved when images are superimposed onto visual displays • Many experiments show that when you project an image onto a display the image acts very much like a superimposed display • Shepard & Podgorny, Hayes, … • Interference effects (Brooks) • Mental scanning • Interaction with the motor system (Finke)

28. Can images be visually reinterpreted? • There have been many claims that people can visually reinterpret images (but remember the parallelogram example!) • These have all been cases where one could easily figure out what the combined image would look like without actually seeing it (e.g., the J – D superposition). • Pederson’s careful examination of visual “reconstruals” showed (contrary to her own conclusion) that images are never ambiguous (no Necker cube or figure-ground reversals) and when new construals were achieved from images they were quite different from the ones achieved in vision (more variable, more guessing from partial cues, etc). • The best evidence comes from a philosopher, Peter Slezak!

29. Slezak figures Pick one (or two) of these animals and memorize what they look like. Now rotate it in your mind by 90 degrees clockwise and see what it looks like.

30. Slezak figures rotated 90o

31. And what if visual imagery did involve the “visual system”? • Even if there was credible evidence that vision is involved in visual imagery, it would tell us nothing about the format of imagery because, • There is strong evidence that visual representations are not pictorial either!

32. The picture theory of visionis a non-starter, even for cats (Cartoon by Kliban)

33. A more plausible theory of vision(even for cats)