Multimodal communication
This presentation is the property of its rightful owner.
Sponsored Links
1 / 22

Multimodal Communication PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Multimodal Communication. Jens Allwood Swecog National Swedish graduate school in cognitive science. Multimodal Communication. 1. Why study multimodal communication? 2.What is multimodal communication? Multimodality in Human-Human Communication Production Types of Representation

Download Presentation

Multimodal Communication

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Multimodal communication

Multimodal Communication

Jens Allwood


National Swedish graduate school in cognitive science

Multimodal communication1

Multimodal Communication

1. Why study multimodal communication?

2.What is multimodal


  • Multimodality in Human-Human Communication

    • Production

    • Types of Representation

    • Perception, Understanding

    • Contextual Dependence

      4.Some Theoretical issues

      5.How study multimodal


      6.Multimodality and ICT (Information & Communication Technology)

Why study multimodal communication

Why study multimodal communication?

  • Interesting in itself.

  • What we are built for and are used to.

    -very much is still unknown

    -need for multimodal communication corpora

    -provides a background for what can be simulated or made use of in technical systems

  • Source for design of computer

    supported applications

    -gives redundance, robustness, synergy

    -more natural and efficient HMI

    -more efficient and enjoyable Human-Human Communication



-New interactive search machines

(internet & local DB)

-VR applications (games, education, trade)

-Other educational support (Tutoring etc.)

-Public information

  • Commercial systems

  • Multimodality needs to be combined with multiculturality and multifunctionality

What is multimodal communication

What is multimodal communication?

Communication = def. Coactivation, sharing and coconstruction of information

Mode/modality =manner, way

Multimodal =

Several alternatives of interpretation

Main alternative

1. Modalities of perception and production

Visual- gestures

Auditory - sounds, speech

Tactile- touch

Olfactory- smell (molecules)

Gustatory - taste (molecules)

? Submodalities color, shape & movement

Other alternatives

Other alternatives

2. Medium =physical carrier

Light waves, sound wavemolecules,


waves etc.

3. Type of representation = index, icon,


4. Communication aids writing,

radio, TV etc.

Suggestion for definition:

Multimodal Communication

= coactivation, sharing and coconstruction of information simultaneously and sequentially through several modes of perception & production

Multimodal communication often becomes also

Multimodal communication often becomes also

  • Multimedial ( always)

  • Multirepesentational

  • Using several aids of communiction

Types of representation

Types of representation

3 ways of carrying, conveying and

sharing information

  • Index

  • Icon

  • Symbol

    All 3 are used simultaneously in

    ordinary human communication.

    E.g.Words (symbols)

    Gesture (icons)

    Voice quality (index)

    “I’m small”

    Variation in awareness and intentionality

    Bodily communication often more

    spontaneous, unaware, unintended (index,

    icon) than words (symbols).

Analysis of relation between modalities

Analysis of relation between Modalities

necessary both for

1. production, planning of different simultaneous expressions


2. perception, understanding of different simultaneous expressions

Theoretical issues

Theoretical Issues

  • How is (should) information (be) distributed (fission) in production across modalities, e g text, picture, voice, diagram, gesture

    2.How is information from different modalities integrated (fusion) in reception? What function do the modalities have in relation to each other?

    3. Which modes of production and perception are used in human-human communication (HHC)?

    4.Are some forms of representation modality neutral and others modality specific?

    spoken (& written) language vs


Theoretical issues continued

Theoretical issues continued

5. Is it possible to translate

between modalities?

  • vision <-> hearing,

  • Smell < -> touch

  • touch <-> vision

    Broad conception of translation and context to determine meaning

    6. Is it possible to translate

    between types of representation?

    Speech (symbol) <-> picture (icon)

    Speech (symbol) <-> freq diagram (index)

    7. What is synesthesia? - activated complementary?

    8.How can cognitive development in general/in different tasks best be supported by multimodal resources?

Theoretical issues continued1

Theoretical issues continued

9.How do we formally represent symbolic non-vocal and non-verbal types of representation?


Tradition of neglect of multimodal issues

11.How can the modes of production and perception be made use of, emulated, simulated in HMI (human-machine interaction) and computer supported HHC?

How should multimodal communication be studied

How should multimodal communication be studied?

1. Descriptive and explanatory (audio, video recorded) studies of human-human communication in naturalistic settings

  • establishment of multimodal corpora and techniques for establishing and analyzing these

    2. Experimental studies of human-human communication

    3. Simulation of human-human communication (VR and robot)

    4. Creation of multimodal application using Wizard of Oz techniques or “Dialog distilling”

Production means of expression

ProductionMeans of expression

Primary means

Primary means

Multimodal Face-to-face


Production & perception

Communicative body movements

Communicative Body Movements

1)Facial gestures (nose, eyebrows, cheek, forehead, chin etc)

2)Head movements

3)Gaze direction, mutual gaze

4)Pupil size

5)Lip movements

6)Hand and arm movements

7)Leg and foot movements

8)Body posture

9)Distance btw communicators

10) Spatial orientation

11) Clothing and bracelets

12) Touch

13) Smell

14) Taste

15) Non-linguistic sounds

Multimodal face to face communication

Multimodal Face-to-face Communication

Three levels of intentionality in communication

Three levels of intentionality in communication




All three occur simultaneously in normal human communication

All combinations are, in principle, possible,

but the three x:s are more common.

Multimodal communication

One can increase awareness and intentionality through practice.

indicate -> display -> signal

What has been practiced can become a habit

conventional signal -> automatic

Ways of processing information reception

Ways of processing information (reception)

1.Unaware reaction

  • Perception



    4.Other attitudes

    5.Behavioral response

    6.Other actions

Communication coactivation sharing and coconstruction between two or more agents

CommunicationCoactivation, sharing and coconstruction between two or more agents

Concluding remarks

Concluding remarks

  • Overview of multmodality in human communication

  • Login