Break out group d
1 / 19

Break-out Group # D - PowerPoint PPT Presentation

  • Uploaded on

Break-out Group # D. Research Issues in Multimodal Interaction. What are the different types. Speech Haptics Gesture Deictic Head and eye movement EEG Electrocephalograms physiological measurements. What has been done so far?. Semantic fusion of information Speech and Gesture

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Break-out Group # D' - ainsley-barton

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Break out group d

Break-out Group # D

Research Issues in Multimodal Interaction

What are the different types
What are the different types

  • Speech

  • Haptics

  • Gesture

  • Deictic

  • Head and eye movement

  • EEG Electrocephalograms

  • physiological measurements

What has been done so far
What has been done so far?

  • Semantic fusion of information

    • Speech and Gesture

  • Preliminary efforts as to what types of modalities to intergrate. This is application dependent.

  • Need standardization at the level of devices and types of information to be fused

Open research problems
Open Research Problems:

  • Should we stay with current paradigms or invent new methodologies?

  • There is no unifying framework for interaction in terms of devices/semantic integration. This is due to the lack of general purpose application.

  • We see specific applications eg, simulation, medical training

Open research questions
Open Research Questions

  • How to deal with specific tasks in terms of fusing channels. How should channels be fused.

  • How to do transitions between tasks, e.g., manipulation vs loccomotion

  • Need more experimentation and a theory as to where VR is needed?

Open research questions1
Open Research Questions

  • Formal study of tasks within applications(e.g., manipulation, selection, navigation, changing of attributes, numerical input)

  • Need more research on output. So far mostly visual and oral.

First breakout group
First breakout group

  • Taxonomy

  • Semantics

  • Cross-modal Representations (actions/perceptions)

Applications output group second meeting

Applications/Output Group Second Meeting

New Issues we Discussed in the afternoon

Dm third breakout group applications output
DM: Third Breakout Group: Applications/Output

  • Human Perception of the environment

  • Integration with Input

  • Relationships to basic principles

Human perceptive abilities
Human Perceptive abilities

  • Vision Technology: Limitation in terms or lighting or real time rendering

  • Limitations for other channels: Haptics, audio, olfaction, taste

  • The type/mix of output depends on the application

  • This is related to the internal representation


  • Issue of using many modalities to offset the limitations of each modality.

    • Right now we do not have enough research data to support that.

  • Do we or not need to represent exactly the environment?

    • Application dependent


  • Abstraction vs exact representation

    • Application dependent

  • Exact physical simulation vs fake physics. Ok or not to fool the user?

    • Probably application and technology dependent.

Other human perceptive modalities
Other Human Perceptive Modalities

  • Olfaction and taste: very little research

  • Some modalities are better understood than others (e.g., visual vs haptic or olfaction)

Continued summary

  • Big issues:

    • Sensory substitution

    • Level of detail (variable resolution)

    • Sampled vs synthetic generation

    • Online or offline computation

    • Preservation (or not) of individuality e.g two people with different sense of taste or heat etc

    • Higher-level emotional augmentation

Integration with input
Integration with Input

  • Haptics is the most widely used output sense that is also used for input

    • Head orientation, whole-body position, eye gaze also

  • Some output must be tightly coupled to input (it’s at the physical level)

    • Head motion to view changes, 3D audio

Integration with input cont
Integration with Input (cont.)

  • Eye gaze-based control requires some interpretation

  • Intentional vs unintentional movement

    • When is a gesture a gesture?

Relationship to basic principles
Relationship to Basic Principles

  • Mapping semantics to output

    • One or multiple representations for all modalities eg. Language and visual output where we have a common representation but gets translated differently for output

    • Spatio-temporal synchronization

    • Cross-modal representation (actions/perceptions)

  • Account for individual differences

Future paper topics
Future Paper Topics

  • All the previously mentioned open problems

  • Short Term

    • Update of the NRC report on modalities

  • Medium-Term

    • Modeling, Coupling and Output of modalities

    • In particular model smell and taste

Future paper topics1
Future Paper Topics

  • Long Term

    • Further modeling and coupling

    • Advanced display technology

    • Personalization of output