Languages, Images and Words. Week 7 :CCT370 – Introduction to Computer Visualization. Addresses relationships between visual information and verbal or textual information Dual coding theory When should we use a visual display? What is a visual language?
Week 7 :CCT370 – Introduction to Computer Visualization
Dual coding theory
When should we use a visual display?
What is a visual language?
And does it make sense to use one to program a computer
How to integrate images and wordsIntroduction: Pictures and Words
Given a is a list of 10 words
Try to remember themA Memory Demonstration
Speech, tasks to do, names, …
Dating back to Greek times, have been various mnemonic devices
Method of loci, or places
Uses places and encoding of items
E.g., take a walk through your apartment, or across campus, or your parents’ house – some well known place
A “memory palace”
In Renaissance, mol widely used, cathedrals served nicely
Identify several points
E.g., 10 for the demonstration
When a sequence is given, place each element at a (physical) point on the path
The more “memorable”, or bizarre, the image the better
To retrieve the elements, walk through the physical place
Memory demo again, but will do it using method of lociMethod of Loci - a Mnemonic
The larger or more detailed the real place, the more information you can store in the corresponding mental space.
Define a route.
If you will need to remember things in a certain order, it is essential that you follow a specific route through your palace, both in the real world and in you mind.
Identify specific storage locations in your palace or along your route.
Identify as many locations as you think you will need.
Walk through your structure or along your route and really observe it.
Memorize your memory palace.
Place things to be remembered in your palace.
Put a manageable amount of information in each place.
Generally, all you need to store in each location is something that will jog your memory, something that will lead you to the actual idea you’re trying to remember.“How to Build a Memory Palace”http://www.wikihow.com/Build-a-Memory-Palace
Two distinct sign systems
Associated with auditory information processing
Includes mathematical symbols, natural language, music
Based on visual information processing
Includes graphics, abstract and figurative imagery
Pavio (1987), dual coding theory
Two different types of information in working memory
Imagens – mental representation of visual information
Objects, natural groupings of objects, whole parts of objects (e.g., arm), spatial information about layout/arrangement
Logogens – mental representation of language information
Basic information pertaining to language, but not sounds of words
Processed by a set of functional subsystems that provide support for reading and writing, understanding and producing speech, logical thought
Not necessarily tied to speech, e.g., deaf and Braille and sign languageCoding Words and Images: Two Systems
Refines/extends basic approach of 3-stage model of human memory
Sharpens ideas about object recognition
“Makes sense” – split between visual and verbal processing
Long known are different neural processing centers for:
Verbal information (speech areas of temporal cortex)
Visual information (visual cortex)
Core of rationale for use of visual representations
Especially to facilitate learning
E.g., concrete mnemonic devices
Greek orators …
Walk through your house and put …
Primary rationale for “multimedia”!
Provides account of why it works
For our purposes, informs use of visual representation
With “words”, in larger context of diagrams, etc.
Sharpening up relation of “visual” with “semantic, …”
Acoustic verbal stimuliPavio’s Dual Coding Theory, 1
Enters through visual system
Fed into association structures in nonverbal imagen system
But then fed into association structures of logogens
Acoustic verbal stimuli
Processed in auditory system
Then fed into logogen system
Logogens and imagens can be strongly interlinked
E.g., word “cat” and language-based concepts related to cats will be linked to visual information related to the appearance of cats
Method of loci, multimedia, etc.Pavio’s Dual Coding Theory, 2
More recently, such operations as part of the “visual” (imagen based system) are finding evidence
E.g., Subjects report using imagery when compare sizes of light bulb and tennis ball, color (green) of pea and Christmas tree
Positive emission tomography (PET) evidence
Visual processing centers active when imagery invoked
Buy, when mentally change size and position of imagined object, different visual areas of brain activated
Imagery alone vs. operations on (processing, thinking with) imagery
When see a cow and imagine a cow same neural pathways (in part) excited
In accord with accounts of object recognition in which object is “recognized” through interaction of stimuli (bottom up) and memory (top down)
See figure at right
Visual memory traces stored as part of processing
Hence, recognition is easier than recall
Matching with something stored (which in part drives recognition)
Vs. reconstructing all pieces from input of event alone“Thinking” Visually
Innate deep structures (meaning)
Surface structure (syntax, form)
That nature of natural (everyday, spoken) language is quite similar to formal description of languages appropriate to describe all languages (natural and formal) is among larger advances of 20th century
Universality of human language
And, e.g., Chomsky hierarchy of (deep) interest to computer scientists
Critical period for language development
But being verbal is not essential to language development
Sign languages for the deaf are the most perfect examples of visual languageThe Nature of Language(s)
Communication of intention
Ability to communicate procedures and sequences of operations – including logic
if, but, causes, do a then b then c
Are not related to verbal languages
Become more abstract
To be fluent in visual (or any) language we should be trained from early in lifeWhat is Language?
Sensory and arbitrary symbols, etc.
Consider that hieroglyphs gave way to more abstract symbols
Why turn back the clock?
Can there be a true visual language?
Yes, but not for most of us!
A critical (developmental) period
Consider verbal language
(if, while, perhaps)
Based on speech
Sign languages are true visual languagesAgain, Visual Languages (or Not)
Get a line of text
Change characters to all upper case
Write line to output file
While there is more input
Better expressed by “verbal psuedo-code”:
Get a line of text
Change characters to all upper case
Write line to output file
Until there is more input
Flow charts were wrong
Probably just wrong paradigm
But, billions of dollars spent
Cause inappropriate/unnatural focus on detail
Visual programming languages have history of failureVisual and Verbal Pseudo-code
Has distinct advantages over text (linear, serial) for conveying some kinds of information
Consider the text below:
Jane is Jim’s boss
Jim is Joe’s boss
Anne works for Jane
Mark works for Jim
Anne is Mary’s boss
Anne is Mike’s boss
And it’s visual representationStill, Certainly Uses for Visual Represt.
Images are better than words for:
Words better for:
Abstract verbal concepts
Images best for showing structural relationships
Links between entities and groups of entities
E.g., bus routes shown as graphical representation led to better performance in trip planning than with tables
Visual information generally remembered better than verbal, but not for abstract images
Visual information need be meaningful and capable of incorporation into a cognitive framework for this advantage
Image memory can’t be so relied on if information is new and out of context
Images best for providing detail and appearance
Amount of detail extracted (and remembered) depends on time to study
Recall, silhoettes first, so line drawing best for rapid extraction
Information that specifies conditions under which something should or should not be done is better provided using text or spoken languageWhen to Use Static Images vs. Words
Gesture that links subject of a spoken sentence with a visual reference
Can be a glance or a nod
Shown to disambiguate verbal communications
Why the mouse is so powerful …
Other kinds of gestures
Beat gestures for emphasis
Verb gestures showing how to do something
McNeil Hand and mind
Issues in shared environments
Speech + Pointer + Visuals – most important components
Subtle ways of directing attention also important in meeting dynamics.
Linking images and words
Pointing is an elementary speech act.
Pointing links images and words
Put that (points) there (points)
Subject verb predicateGestures, and Linking Images and Words
Or even quite well
“Association” (grouping) can be explained by Gestalt principlesAttaching Words to Images
Still, some examples
KhorosExamples of visual languages