multimodal communication n.
Skip this Video
Download Presentation
Multimodal Communication

Loading in 2 Seconds...

play fullscreen
1 / 22

Multimodal Communication - PowerPoint PPT Presentation

  • Uploaded on

Multimodal Communication. Jens Allwood Swecog National Swedish graduate school in cognitive science. Multimodal Communication. 1. Why study multimodal communication? 2. What is multimodal communication? Multimodality in Human-Human Communication Production Types of Representation

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Multimodal Communication' - odysseus-patton

Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
multimodal communication

Multimodal Communication

Jens Allwood


National Swedish graduate school in cognitive science

multimodal communication1
Multimodal Communication

1. Why study multimodal communication?

2. What is multimodal


  • Multimodality in Human-Human Communication
    • Production
    • Types of Representation
    • Perception, Understanding
    • Contextual Dependence

4. Some Theoretical issues

5. How study multimodal


6. Multimodality and ICT (Information & Communication Technology)

why study multimodal communication
Why study multimodal communication?
  • Interesting in itself.
  • What we are built for and are used to.

- very much is still unknown

- need for multimodal communication corpora

- provides a background for what can be simulated or made use of in technical systems

  • Source for design of computer

supported applications

- gives redundance, robustness, synergy

- more natural and efficient HMI

- more efficient and enjoyable Human-Human Communication


- New interactive search machines

(internet & local DB)

- VR applications (games, education, trade)

- Other educational support (Tutoring etc.)

- Public information

  • Commercial systems
  • Multimodality needs to be combined with multiculturality and multifunctionality
what is multimodal communication
What is multimodal communication?

Communication = def. Coactivation, sharing and coconstruction of information

Mode/modality = manner, way

Multimodal =

Several alternatives of interpretation

Main alternative

1. Modalities of perception and production

Visual - gestures

Auditory - sounds, speech

Tactile - touch

Olfactory - smell (molecules)

Gustatory - taste (molecules)

? Submodalities color, shape & movement

other alternatives
Other alternatives

2. Medium = physical carrier

Light waves, sound wave molecules,


waves etc.

3. Type of representation = index, icon,


4. Communication aids writing,

radio, TV etc.

Suggestion for definition:

Multimodal Communication

= coactivation, sharing and coconstruction of information simultaneously and sequentially through several modes of perception & production

multimodal communication often becomes also
Multimodal communication often becomes also
  • Multimedial ( always)
  • Multirepesentational
  • Using several aids of communiction
types of representation
Types of representation

3 ways of carrying, conveying and

sharing information

  • Index
  • Icon
  • Symbol

All 3 are used simultaneously in

ordinary human communication.

E.g. Words (symbols)

Gesture (icons)

Voice quality (index)

“I’m small”

Variation in awareness and intentionality

Bodily communication often more

spontaneous, unaware, unintended (index,

icon) than words (symbols).

analysis of relation between modalities
Analysis of relation between Modalities

necessary both for

1. production, planning of different simultaneous expressions


2. perception, understanding of different simultaneous expressions

theoretical issues
Theoretical Issues
  • How is (should) information (be) distributed (fission) in production across modalities, e g text, picture, voice, diagram, gesture

2. How is information from different modalities integrated (fusion) in reception? What function do the modalities have in relation to each other?

3. Which modes of production and perception are used in human-human communication (HHC)?

4. Are some forms of representation modality neutral and others modality specific?

spoken (& written) language vs


theoretical issues continued
Theoretical issues continued

5. Is it possible to translate

between modalities?

  • vision <-> hearing,
  • Smell < -> touch
  • touch <-> vision

Broad conception of translation and context to determine meaning

6. Is it possible to translate

between types of representation?

Speech (symbol) <-> picture (icon)

Speech (symbol) <-> freq diagram (index)

7. What is synesthesia? - activated complementary?

8. How can cognitive development in general/in different tasks best be supported by multimodal resources?

theoretical issues continued1
Theoretical issues continued

9. How do we formally represent symbolic non-vocal and non-verbal types of representation?

10. Problem

Tradition of neglect of multimodal issues

11. How can the modes of production and perception be made use of, emulated, simulated in HMI (human-machine interaction) and computer supported HHC?

how should multimodal communication be studied
How should multimodal communication be studied?

1. Descriptive and explanatory (audio, video recorded) studies of human-human communication in naturalistic settings

  • establishment of multimodal corpora and techniques for establishing and analyzing these

2. Experimental studies of human-human communication

3. Simulation of human-human communication (VR and robot)

4. Creation of multimodal application using Wizard of Oz techniques or “Dialog distilling”

primary means
Primary means

Multimodal Face-to-face


Production & perception

communicative body movements
Communicative Body Movements

1) Facial gestures (nose, eyebrows, cheek, forehead, chin etc)

2) Head movements

3) Gaze direction, mutual gaze

4) Pupil size

5) Lip movements

6) Hand and arm movements

7) Leg and foot movements

8) Body posture

9) Distance btw communicators

10) Spatial orientation

11) Clothing and bracelets

12) Touch

13) Smell

14) Taste

15) Non-linguistic sounds

three levels of intentionality in communication
Three levels of intentionality in communication




All three occur simultaneously in normal human communication

All combinations are, in principle, possible,

but the three x:s are more common.

One can increase awareness and intentionality through practice.

indicate -> display -> signal

What has been practiced can become a habit

conventional signal -> automatic

ways of processing information reception
Ways of processing information (reception)

1. Unaware reaction

  • Perception


3. Understanding

4. Other attitudes

5. Behavioral response

6. Other actions

concluding remarks
Concluding remarks
  • Overview of multmodality in human communication