languages images and words n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Languages, Images and Words PowerPoint Presentation
Download Presentation
Languages, Images and Words

Loading in 2 Seconds...

play fullscreen
1 / 27

Languages, Images and Words - PowerPoint PPT Presentation


  • 127 Views
  • Uploaded on

Languages, Images and Words. Week 7 :CCT370 – Introduction to Computer Visualization. Addresses relationships between visual information and verbal or textual information Dual coding theory When should we use a visual display? What is a visual language?

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Languages, Images and Words' - berny


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
languages images and words

Languages, Images and Words

Week 7 :CCT370 – Introduction to Computer Visualization

introduction pictures and words
Addresses relationships between visual information and verbal or textual information

Dual coding theory

When should we use a visual display?

What is a visual language?

And does it make sense to use one to program a computer

How to integrate images and words

Introduction: Pictures and Words
a memory demonstration
Remembering words (and lots of things) is not easy

Given a is a list of 10 words

Try to remember them

A Memory Demonstration
method of loci a mnemonic
Remembering things, especially long sequences, not a new problem

Speech, tasks to do, names, …

Dating back to Greek times, have been various mnemonic devices

Method of loci, or places

Uses places and encoding of items

E.g., take a walk through your apartment, or across campus, or your parents’ house – some well known place

A “memory palace”

In Renaissance, mol widely used, cathedrals served nicely

Identify several points

E.g., 10 for the demonstration

When a sequence is given, place each element at a (physical) point on the path

The more “memorable”, or bizarre, the image the better

To retrieve the elements, walk through the physical place

Memory demo again, but will do it using method of loci

Method of Loci - a Mnemonic
how to build a memory palace http www wikihow com build a memory palace
Decide on a blueprint for your palace.

The larger or more detailed the real place, the more information you can store in the corresponding mental space.

Define a route.

If you will need to remember things in a certain order, it is essential that you follow a specific route through your palace, both in the real world and in you mind.

Identify specific storage locations in your palace or along your route.

Identify as many locations as you think you will need.

Walk through your structure or along your route and really observe it.

Memorize your memory palace.

Place things to be remembered in your palace.

Put a manageable amount of information in each place.

Use symbols.

Generally, all you need to store in each location is something that will jog your memory, something that will lead you to the actual idea you’re trying to remember.

“How to Build a Memory Palace”http://www.wikihow.com/Build-a-Memory-Palace
  • Be creative.
    • Generally, images will be more memorable if they are absurd (out of the ordinary
  • Stock your palace with other mnemonics.
    • “Every Good Boy Deserves Fudge," which would in turn allow you to recall the order of notes on the lines in treble clef (EGBDF).
  • Explore your palace.
    • Go through it and look at them.
  • Use your palace.
    • just follow your route in order as you do so.
  • Build new palaces.
    • If you need only commit things to memory for a short time. Just replace the existing contents with new ones
    • If you need to remember the contents of your palace for a long time, you can keep that palace as it is and create new ones in which to store other information as needed
coding words and images two systems
Bertin, Semiology of Graphics (1983)

Two distinct sign systems

Associated with auditory information processing

Includes mathematical symbols, natural language, music

Based on visual information processing

Includes graphics, abstract and figurative imagery

Pavio (1987), dual coding theory

Two different types of information in working memory

Imagens – mental representation of visual information

Objects, natural groupings of objects, whole parts of objects (e.g., arm), spatial information about layout/arrangement

Logogens – mental representation of language information

Basic information pertaining to language, but not sounds of words

Processed by a set of functional subsystems that provide support for reading and writing, understanding and producing speech, logical thought

Not necessarily tied to speech, e.g., deaf and Braille and sign language

Coding Words and Images: Two Systems
pavio s dual coding theory 1
Account of memory (with implications for perception)

Refines/extends basic approach of 3-stage model of human memory

Sharpens ideas about object recognition

“Makes sense” – split between visual and verbal processing

Long known are different neural processing centers for:

Verbal information (speech areas of temporal cortex)

Visual information (visual cortex)

Core of rationale for use of visual representations

Especially to facilitate learning

E.g., concrete mnemonic devices

Greek orators …

Walk through your house and put …

Primary rationale for “multimedia”!

Provides account of why it works

For our purposes, informs use of visual representation

With “words”, in larger context of diagrams, etc.

Sharpening up relation of “visual” with “semantic, …”

Processing:

Visual-spatial information

Visual text

Acoustic verbal stimuli

Pavio’s Dual Coding Theory, 1
pavio s dual coding theory 2
Processing:

Visual-spatial information

Enters through visual system

Fed into association structures in nonverbal imagen system

Visual text

Processed,

But then fed into association structures of logogens

Acoustic verbal stimuli

Processed in auditory system

Then fed into logogen system

Logogens and imagens can be strongly interlinked

E.g., word “cat” and language-based concepts related to cats will be linked to visual information related to the appearance of cats

Method of loci, multimedia, etc.

Pavio’s Dual Coding Theory, 2
  • Imagens – mental representation of visual information
  • Objects, natural groupings of objects, whole parts of objects (e.g., arm), spatial information about layout/arrangement
  • Logogens – mental representation of language information
  • Basic information pertaining to language, but not sounds of words
  • Processed by a set of functional subsystems that provide support for reading and writing, understanding and producing speech, logical thought
  • Not necessarily tied to speech, e.g., deaf and Braille and sign language
thinking visually
Usually associate processes of “thinking”, e.g., greater than, follows from, “logic”, etc., with “verbal” (logogen based system)

More recently, such operations as part of the “visual” (imagen based system) are finding evidence

E.g., Subjects report using imagery when compare sizes of light bulb and tennis ball, color (green) of pea and Christmas tree

Positive emission tomography (PET) evidence

Visual processing centers active when imagery invoked

Buy, when mentally change size and position of imagined object, different visual areas of brain activated

Imagery alone vs. operations on (processing, thinking with) imagery

When see a cow and imagine a cow same neural pathways (in part) excited

In accord with accounts of object recognition in which object is “recognized” through interaction of stimuli (bottom up) and memory (top down)

See figure at right

Visual memory traces stored as part of processing

Hence, recognition is easier than recall

Matching with something stored (which in part drives recognition)

Vs. reconstructing all pieces from input of event alone

“Thinking” Visually
the nature of language s
Noam Chomsky:

Innate deep structures (meaning)

Surface structure (syntax, form)

That nature of natural (everyday, spoken) language is quite similar to formal description of languages appropriate to describe all languages (natural and formal) is among larger advances of 20th century

Universality of human language

And, e.g., Chomsky hierarchy of (deep) interest to computer scientists

Critical period for language development

But being verbal is not essential to language development

Sign languages for the deaf are the most perfect examples of visual language

The Nature of Language(s)
what is language
Language provides:

Description

Communication of intention

Ability to communicate procedures and sequences of operations – including logic

if, but, causes, do a then b then c

Sign Languages

Arose spontaneously

Are not related to verbal languages

Have syntax

Become more abstract

To be fluent in visual (or any) language we should be trained from early in life

What is Language?
again visual languages or not
Recall, some fundamental questions addressed in first of class

Sensory and arbitrary symbols, etc.

Visual language

Consider that hieroglyphs gave way to more abstract symbols

Why turn back the clock?

Can there be a true visual language?

Yes, but not for most of us!

A critical (developmental) period

ontogenetic

Consider verbal language

Abstraction, logic

(if, while, perhaps)

Based on speech

Sign languages are true visual languages

Again, Visual Languages (or Not)
visual and verbal pseudo code
Ware argues that visual languages in fact not good for expressing sequential operations common to computer programming,

Sequence below:

Get a line of text

Change characters to all upper case

Write line to output file

While there is more input

Better expressed by “verbal psuedo-code”:

Repeat

Get a line of text

Change characters to all upper case

Write line to output file

Until there is more input

Flow charts were wrong

Probably just wrong paradigm

But, billions of dollars spent

Cause inappropriate/unnatural focus on detail

Visual programming languages have history of failure

Visual and Verbal Pseudo-code
still certainly uses for visual represt
Gives rapid recognition and pattern finding (again)

Has distinct advantages over text (linear, serial) for conveying some kinds of information

Consider the text below:

Jane is Jim’s boss

Jim is Joe’s boss

Anne works for Jane

Mark works for Jim

Anne is Mary’s boss

Anne is Mike’s boss

And it’s visual representation

Still, Certainly Uses for Visual Represt.
when to use static images vs words
In general (rules of thumb)

Images are better than words for:

Spatial structures

Location

Detail

Words better for:

Procedural information

Logical condition

Abstract verbal concepts

Images best for showing structural relationships

Links between entities and groups of entities

E.g., bus routes shown as graphical representation led to better performance in trip planning than with tables

Visual information generally remembered better than verbal, but not for abstract images

Visual information need be meaningful and capable of incorporation into a cognitive framework for this advantage

Image memory can’t be so relied on if information is new and out of context

Images best for providing detail and appearance

Amount of detail extracted (and remembered) depends on time to study

Recall, silhoettes first, so line drawing best for rapid extraction

Information that specifies conditions under which something should or should not be done is better provided using text or spoken language

When to Use Static Images vs. Words
gestures and linking images and words
Deixis and the deictic gesture

Gesture that links subject of a spoken sentence with a visual reference

Can be a glance or a nod

Pre-speech

Shown to disambiguate verbal communications

Why the mouse is so powerful …

Other kinds of gestures

Beat gestures for emphasis

Verb gestures showing how to do something

McNeil Hand and mind

Issues in shared environments

Speech + Pointer + Visuals – most important components

Subtle ways of directing attention also important in meeting dynamics.

Linking images and words

Deixis

Pointing is an elementary speech act.

Pointing links images and words

Put that (points) there (points)

Subject verb predicate

Gestures, and Linking Images and Words
attaching words to images
Turns out the use of words and images together work just fine

Or even quite well

“Association” (grouping) can be explained by Gestalt principles

Attaching Words to Images
examples of visual languages
As noted, limited success of visual languages, especially for programming

Still, some examples

Sanscrit

Petri-nets

Khoros

Examples of visual languages
sanscrit
To program:

Count from 1 to 3

for i = 1 to 3 do

Sanscrit
petri net language
Petri nets are stochastic

timed attributed (tokens on nodes, transitions)

Petri Net Language
conclusions
Conclusions
  • Interesting, and not fully understood interaction between the two processing subsystems
    • “Visual”, images, imagens
    • “Verbal”, auditory, logogens
  • Though certainly there can be “visual languages”, finding useful applications has been elusive
next class
Next class
  • Topic: Thinking Visualizations
  • Readings:
    • Ware, Chapter 11