Images, Words, and Gestures

1. Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014

2. Introduction: Pictures and Words • Addresses relationships between visual information and verbal or textual information • Dual coding theory • When should we use a visual display? • What is a visual language? • And does it make sense to use one to program a computer • How to integrate images and words

3. A Memory Demonstration • Remembering words (and lots of things) is not easy • Given a is a list of 10 words • Try to remember them

4. Method of Loci - a Mnemonic • Remembering things, especially long sequences, not a new problem • Speech, tasks to do, names, … • Dating back to Greek times, have been various mnemonic devices • Method of loci, or places • Uses places and encoding of items • E.g., take a walk through your apartment, or across campus, or your parents’ house – some well known place • A “memory palace” • In Renaissance, mol widely used, cathedrals served nicely • Identify several points • E.g., 10 for the demonstration • When a sequence is given, place each element at a (physical) point on the path • The more “memorable”, or bizarre, the image the better • To retrieve the elements, walk through the physical place • Memory demo again, but will do it using method of loci

5. “How to Build a Memory Palace”http://www.wikihow.com/Build-a-Memory-Palace • Decide on a blueprint for your palace. • The larger or more detailed the real place, the more information you can store in the corresponding mental space. • Define a route. • If you will need to remember things in a certain order, it is essential that you follow a specific route through your palace, both in the real world and in you mind. • Identify specific storage locations in your palace or along your route. • Identify as many locations as you think you will need. • Walk through your structure or along your route and really observe it. • Memorize your memory palace. • Place things to be remembered in your palace. • Put a manageable amount of information in each place. • Use symbols. • Generally, all you need to store in each location is something that will jog your memory, something that will lead you to the actual idea you’re trying to remember. • Be creative. • Generally, images will be more memorable if they are absurd (out of the ordinary • Stock your palace with other mnemonics. • “Every Good Boy Deserves Fudge," which would in turn allow you to recall the order of notes on the lines in treble clef (EGBDF). • Explore your palace. • Go through it and look at them. • Use your palace. • just follow your route in order as you do so. • Build new palaces. • If you need only commit things to memory for a short time. Just replace the existing contents with new ones • If you need to remember the contents of your palace for a long time, you can keep that palace as it is and create new ones in which to store other information as needed

6. Memory Demo, Again • Same song – new technique

7. Coding Words and Images: Two Systems • Bertin, Semiology of Graphics (1983) • Two distinct sign systems • Associated with auditory information processing • Includes mathematical symbols, natural language, music • Based on visual information processing • Includes graphics, abstract and figurative imagery • Pavio (1987), dual coding theory • Two different types of information in working memory • Imagens – mental representation of visual information • Objects, natural groupings of objects, whole parts of objects (e.g., arm), spatial information about layout/arrangement • Logogens – mental representation of language information • Basic information pertaining to language, but not sounds of words • Processed by a set of functional subsystems that provide support for reading and writing, understanding and producing speech, logical thought • Not necessarily tied to speech, e.g., deaf and Braille and sign language

8. Pavio’s Dual Coding Theory, 1 • Account of memory (with implications for perception) • Refines/extends basic approach of 3-stage model of human memory • Sharpens ideas about object recognition • “Makes sense” – split between visual and verbal processing • Long known are different neural processing centers for: • Verbal information (speech areas of temporal cortex) • Visual information (visual cortex) • Core of rationale for use of visual representations • Especially to facilitate learning • E.g., concrete mnemonic devices • Greek orators … • Walk through your house and put … • Primary rationale for “multimedia”! • Provides account of why it works • For our purposes, informs use of visual representation • With “words”, in larger context of diagrams, etc. • Sharpening up relation of “visual” with “semantic, …” • Processing: • Visual-spatial information • Visual text • Acoustic verbal stimuli

9. Pavio’s Dual Coding Theory, 2 • Processing: • Visual-spatial information • Enters through visual system • Fed into association structures in nonverbal imagen system • Visual text • Processed, • But then fed into association structures of logogens • Acoustic verbal stimuli • Processed in auditory system • Then fed into logogen system • Logogens and imagens can be strongly interlinked • E.g., word “cat” and language-based concepts related to cats will be linked to visual information related to the appearance of cats • Method of loci, multimedia, etc. • Imagens – mental representation of visual information • Objects, natural groupings of objects, whole parts of objects (e.g., arm), spatial information about layout/arrangement • Logogens – mental representation of language information • Basic information pertaining to language, but not sounds of words • Processed by a set of functional subsystems that provide support for reading and writing, understanding and producing speech, logical thought • Not necessarily tied to speech, e.g., deaf and Braille and sign language

10. “Thinking” Visually • Usually associate processes of “thinking”, e.g., greater than, follows from, “logic”, etc., with “verbal” (logogen based system) • More recently, such operations as part of the “visual” (imagen based system) are finding evidence • E.g., Subjects report using imagery when compare sizes of light bulb and tennis ball, color (green) of pea and Christmas tree • Positive emission tomography (PET) evidence • Visual processing centers active when imagery invoked • Buy, when mentally change size and position of imagined object, different visual areas of brain activated • Imagery alone vs. operations on (processing, thinking with) imagery • When see a cow and imagine a cow same neural pathways (in part) excited • In accord with accounts of object recognition in which object is “recognized” through interaction of stimuli (bottom up) and memory (top down) • See figure at right • Visual memory traces stored as part of processing • Hence, recognition is easier than recall • Matching with something stored (which in part drives recognition) • Vs. reconstructing all pieces from input of event alone

11. The Nature of Language(s) • Noam Chomsky: • Innate deep structures (meaning) • Surface structure (syntax, form) • That nature of natural (everyday, spoken) language is quite similar to formal description of languages appropriate to describe all languages (natural and formal) is among larger advances of 20th century • Universality of human language • And, e.g., Chomsky hierarchy of (deep) interest to computer scientists • Critical period for language development • But being verbal is not essential to language development • Sign languages for the deaf are the most perfect examples of visual language

12. What is Language? • Language provides: • Description • Communication of intention • Ability to communicate procedures and sequences of operations – including logic • if, but, causes, do a then b then c • Sign Languages • Arose spontaneously • Are not related to verbal languages • Have syntax • Become more abstract • To be fluent in visual (or any) language we should be trained from early in life

13. Again, Visual Languages (or Not) • Recall, some fundamental questions addressed in first of class • Sensory and arbitrary symbols, etc. • Visual language • Consider that hieroglyphs gave way to more abstract symbols • Why turn back the clock? • Can there be a true visual language? • Yes, but not for most of us! • A critical (developmental) period • ontogenetic • Consider verbal language • Abstraction, logic • (if, while, perhaps) • Based on speech • Sign languages are true visual languages

14. Visual and Verbal Pseudo-code • Ware argues that visual languages in fact not good for expressing sequential operations common to computer programming, • Sequence below: • Get a line of text • Change characters to all upper case • Write line to output file • While there is more input • Better expressed by “verbal psuedo-code”: Repeat Get a line of text Change characters to all upper case Write line to output file Until there is more input • Flow charts were wrong • Probably just wrong paradigm • But, billions of dollars spent • Cause inappropriate/unnatural focus on detail • Visual programming languages have history of failure

15. Still, Certainly Uses for Visual Represt. • Gives rapid recognition and pattern finding (again) • Has distinct advantages over text (linear, serial) for conveying some kinds of information • Consider the text below: • Jane is Jim’s boss • Jim is Joe’s boss • Anne works for Jane • Mark works for Jim • Anne is Mary’s boss • Anne is Mike’s boss • And it’s visual representation

16. When to Use Static Images vs. Words • In general (rules of thumb) • Images are better than words for: • Spatial structures • Location • Detail • Words better for: • Procedural information • Logical condition • Abstract verbal concepts • Images best for showing structural relationships • Links between entities and groups of entities • E.g., bus routes shown as graphical representation led to better performance in trip planning than with tables • Visual information generally remembered better than verbal, but not for abstract images • Visual information need be meaningful and capable of incorporation into a cognitive framework for this advantage • Image memory can’t be so relied on if information is new and out of context • Images best for providing detail and appearance • Amount of detail extracted (and remembered) depends on time to study • Recall, silhoettes first, so line drawing best for rapid extraction • Information that specifies conditions under which something should or should not be done is better provided using text or spoken language

17. Gestures, and Linking Images and Words • Deixis and the deictic gesture • Gesture that links subject of a spoken sentence with a visual reference • Can be a glance or a nod • Pre-speech • Shown to disambiguate verbal communications • Why the mouse is so powerful … • Other kinds of gestures • Beat gestures for emphasis • Verb gestures showing how to do something • McNeil Hand and mind • Issues in shared environments • Speech + Pointer + Visuals – most important components • Subtle ways of directing attention also important in meeting dynamics. • Linking images and words • Deixis • Pointing is an elementary speech act. • Pointing links images and words • Put that (points) there (points) • Subject verb predicate

18. Attaching Words to Images • Turns out the use of words and images together work just fine • Or even quite well • “Association” (grouping) can be explained by Gestalt principles

19. Examples of Integrated Pictures and Words • Examples

20. Another Ex. of Integrated Pictures and Words • More examples

21. Yet Another Ex. of Integrated Pics. and Words

22. Examples of visual languages • As noted, limited success of visual languages, especially for programming • Still, some examples • Sanscrit , Petri-nets, Khoros

23. Sanscrit • To program: • Count from 1 to 3 • for i = 1 to 3 do

24. Petri Net Language • Petri nets are stochastic • timed attributed (tokens on nodes, transitions)

25. Khoros • Combine operations

26. Conclusions • Interesting, and not fully understood interaction between the two processing subsystems • “Visual”, images, imagens • “Verbal”, auditory, logogens • Though certainly there can be “visual languages”, finding useful applications has been elusive

27. End • .