Multimodal communication
This presentation is the property of its rightful owner.
Sponsored Links
1 / 22

Multimodal Communication PowerPoint PPT Presentation


  • 141 Views
  • Uploaded on
  • Presentation posted in: General

Multimodal Communication. Jens Allwood Swecog National Swedish graduate school in cognitive science. Multimodal Communication. 1. Why study multimodal communication? 2.What is multimodal communication? Multimodality in Human-Human Communication Production Types of Representation

Download Presentation

Multimodal Communication

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Multimodal communication

Multimodal Communication

Jens Allwood

Swecog

National Swedish graduate school in cognitive science


Multimodal communication1

Multimodal Communication

1. Why study multimodal communication?

2.What is multimodal

communication?

  • Multimodality in Human-Human Communication

    • Production

    • Types of Representation

    • Perception, Understanding

    • Contextual Dependence

      4.Some Theoretical issues

      5.How study multimodal

      communication?

      6.Multimodality and ICT (Information & Communication Technology)


Why study multimodal communication

Why study multimodal communication?

  • Interesting in itself.

  • What we are built for and are used to.

    -very much is still unknown

    -need for multimodal communication corpora

    -provides a background for what can be simulated or made use of in technical systems

  • Source for design of computer

    supported applications

    -gives redundance, robustness, synergy

    -more natural and efficient HMI

    -more efficient and enjoyable Human-Human Communication


Examples

Examples

-New interactive search machines

(internet & local DB)

-VR applications (games, education, trade)

-Other educational support (Tutoring etc.)

-Public information

  • Commercial systems

  • Multimodality needs to be combined with multiculturality and multifunctionality


What is multimodal communication

What is multimodal communication?

Communication = def. Coactivation, sharing and coconstruction of information

Mode/modality =manner, way

Multimodal =

Several alternatives of interpretation

Main alternative

1. Modalities of perception and production

Visual- gestures

Auditory - sounds, speech

Tactile- touch

Olfactory- smell (molecules)

Gustatory - taste (molecules)

? Submodalities color, shape & movement


Other alternatives

Other alternatives

2. Medium =physical carrier

Light waves, sound wavemolecules,

electromagnetic

waves etc.

3. Type of representation = index, icon,

symbol

4. Communication aids writing,

radio, TV etc.

Suggestion for definition:

Multimodal Communication

= coactivation, sharing and coconstruction of information simultaneously and sequentially through several modes of perception & production


Multimodal communication often becomes also

Multimodal communication often becomes also

  • Multimedial ( always)

  • Multirepesentational

  • Using several aids of communiction


Types of representation

Types of representation

3 ways of carrying, conveying and

sharing information

  • Index

  • Icon

  • Symbol

    All 3 are used simultaneously in

    ordinary human communication.

    E.g.Words (symbols)

    Gesture (icons)

    Voice quality (index)

    “I’m small”

    Variation in awareness and intentionality

    Bodily communication often more

    spontaneous, unaware, unintended (index,

    icon) than words (symbols).


Analysis of relation between modalities

Analysis of relation between Modalities

necessary both for

1. production, planning of different simultaneous expressions

and

2. perception, understanding of different simultaneous expressions


Theoretical issues

Theoretical Issues

  • How is (should) information (be) distributed (fission) in production across modalities, e g text, picture, voice, diagram, gesture

    2.How is information from different modalities integrated (fusion) in reception? What function do the modalities have in relation to each other?

    3. Which modes of production and perception are used in human-human communication (HHC)?

    4.Are some forms of representation modality neutral and others modality specific?

    spoken (& written) language vs

    pictures


Theoretical issues continued

Theoretical issues continued

5. Is it possible to translate

between modalities?

  • vision <-> hearing,

  • Smell < -> touch

  • touch <-> vision

    Broad conception of translation and context to determine meaning

    6. Is it possible to translate

    between types of representation?

    Speech (symbol) <-> picture (icon)

    Speech (symbol) <-> freq diagram (index)

    7. What is synesthesia? - activated complementary?

    8.How can cognitive development in general/in different tasks best be supported by multimodal resources?


Theoretical issues continued1

Theoretical issues continued

9.How do we formally represent symbolic non-vocal and non-verbal types of representation?

10.Problem

Tradition of neglect of multimodal issues

11.How can the modes of production and perception be made use of, emulated, simulated in HMI (human-machine interaction) and computer supported HHC?


How should multimodal communication be studied

How should multimodal communication be studied?

1. Descriptive and explanatory (audio, video recorded) studies of human-human communication in naturalistic settings

  • establishment of multimodal corpora and techniques for establishing and analyzing these

    2. Experimental studies of human-human communication

    3. Simulation of human-human communication (VR and robot)

    4. Creation of multimodal application using Wizard of Oz techniques or “Dialog distilling”


Production means of expression

ProductionMeans of expression


Primary means

Primary means

Multimodal Face-to-face

Communciation

Production & perception


Communicative body movements

Communicative Body Movements

1)Facial gestures (nose, eyebrows, cheek, forehead, chin etc)

2)Head movements

3)Gaze direction, mutual gaze

4)Pupil size

5)Lip movements

6)Hand and arm movements

7)Leg and foot movements

8)Body posture

9)Distance btw communicators

10) Spatial orientation

11) Clothing and bracelets

12) Touch

13) Smell

14) Taste

15) Non-linguistic sounds


Multimodal face to face communication

Multimodal Face-to-face Communication


Three levels of intentionality in communication

Three levels of intentionality in communication

Indicate

Display

Signal

All three occur simultaneously in normal human communication

All combinations are, in principle, possible,

but the three x:s are more common.


Multimodal communication

One can increase awareness and intentionality through practice.

indicate -> display -> signal

What has been practiced can become a habit

conventional signal -> automatic


Ways of processing information reception

Ways of processing information (reception)

1.Unaware reaction

  • Perception

    (apprehension)

    3.Understanding

    4.Other attitudes

    5.Behavioral response

    6.Other actions


Communication coactivation sharing and coconstruction between two or more agents

CommunicationCoactivation, sharing and coconstruction between two or more agents


Concluding remarks

Concluding remarks

  • Overview of multmodality in human communication


  • Login