Break out group d
Sponsored Links
This presentation is the property of its rightful owner.
1 / 19

Break-out Group # D PowerPoint PPT Presentation


  • 99 Views
  • Uploaded on
  • Presentation posted in: General

Break-out Group # D. Research Issues in Multimodal Interaction. What are the different types. Speech Haptics Gesture Deictic Head and eye movement EEG Electrocephalograms physiological measurements. What has been done so far?. Semantic fusion of information Speech and Gesture

Download Presentation

Break-out Group # D

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Break-out Group # D

Research Issues in Multimodal Interaction


What are the different types

  • Speech

  • Haptics

  • Gesture

  • Deictic

  • Head and eye movement

  • EEG Electrocephalograms

  • physiological measurements


What has been done so far?

  • Semantic fusion of information

    • Speech and Gesture

  • Preliminary efforts as to what types of modalities to intergrate. This is application dependent.

  • Need standardization at the level of devices and types of information to be fused


Open Research Problems:

  • Should we stay with current paradigms or invent new methodologies?

  • There is no unifying framework for interaction in terms of devices/semantic integration. This is due to the lack of general purpose application.

  • We see specific applications eg, simulation, medical training


Open Research Questions

  • How to deal with specific tasks in terms of fusing channels. How should channels be fused.

  • How to do transitions between tasks, e.g., manipulation vs loccomotion

  • Need more experimentation and a theory as to where VR is needed?


Open Research Questions

  • Formal study of tasks within applications(e.g., manipulation, selection, navigation, changing of attributes, numerical input)

  • Need more research on output. So far mostly visual and oral.


First breakout group

  • Taxonomy

  • Semantics

  • Cross-modal Representations (actions/perceptions)


Applications/Output Group Second Meeting

New Issues we Discussed in the afternoon


DM: Third Breakout Group: Applications/Output

  • Human Perception of the environment

  • Integration with Input

  • Relationships to basic principles


Human Perceptive abilities

  • Vision Technology: Limitation in terms or lighting or real time rendering

  • Limitations for other channels: Haptics, audio, olfaction, taste

  • The type/mix of output depends on the application

  • This is related to the internal representation


Continued

  • Issue of using many modalities to offset the limitations of each modality.

    • Right now we do not have enough research data to support that.

  • Do we or not need to represent exactly the environment?

    • Application dependent


Continued

  • Abstraction vs exact representation

    • Application dependent

  • Exact physical simulation vs fake physics. Ok or not to fool the user?

    • Probably application and technology dependent.


Other Human Perceptive Modalities

  • Olfaction and taste: very little research

  • Some modalities are better understood than others (e.g., visual vs haptic or olfaction)


Continued-Summary

  • Big issues:

    • Sensory substitution

    • Level of detail (variable resolution)

    • Sampled vs synthetic generation

    • Online or offline computation

    • Preservation (or not) of individuality e.g two people with different sense of taste or heat etc

    • Higher-level emotional augmentation


Integration with Input

  • Haptics is the most widely used output sense that is also used for input

    • Head orientation, whole-body position, eye gaze also

  • Some output must be tightly coupled to input (it’s at the physical level)

    • Head motion to view changes, 3D audio


Integration with Input (cont.)

  • Eye gaze-based control requires some interpretation

  • Intentional vs unintentional movement

    • When is a gesture a gesture?


Relationship to Basic Principles

  • Mapping semantics to output

    • One or multiple representations for all modalities eg. Language and visual output where we have a common representation but gets translated differently for output

    • Spatio-temporal synchronization

    • Cross-modal representation (actions/perceptions)

  • Account for individual differences


Future Paper Topics

  • All the previously mentioned open problems

  • Short Term

    • Update of the NRC report on modalities

  • Medium-Term

    • Modeling, Coupling and Output of modalities

    • In particular model smell and taste


Future Paper Topics

  • Long Term

    • Further modeling and coupling

    • Advanced display technology

    • Personalization of output


  • Login