Signal Processing for
1 / 12

W3C Web Technology Day - PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Signal Processing for Multimodal Web Irek Defée Department of Signal Processing Tampere University of Technology. W3C Web Technology Day. Current status. Web is developed for traditional data and computer I/O: text, keyboard, mouse This is simple and effective but not a natural

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentationdownload

W3C Web Technology Day

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

W3c web technology day

Signal Processing for Multimodal WebIrekDefée Department of Signal Processing Tampere University of Technology

W3C Web Technology Day

Current status

Current status

  • Web is developed for traditional data and computer I/O: text, keyboard, mouse

  • This is simple and effective but not a natural

    way of human interaction with the world

  • Humans interact via

    perceptual system

Human perceptual system

Human Perceptual System

  • Human perceptual system has multiple senses: visual, acoustical, haptic

    (touch, body position, temperature)

    and actuators (vocal tract, muscles, motoric system)

  • The perceptual system is intrinsically MULTIMODAL: multiple senses and actuators operate in perfectly coordinated way

Perceptual information technology

Perceptual Information Technology

  • Information technology is evolving towards natural MULTIMODAL human interaction:

  • Touch gestures

    revolutionized mobile devices

  • Intelligent speech input

    is available

  • There is more to come:

    new sensors, cameras

    and intelligence

Signal processing role

Signal Processing Role

  • Perceptual Information Technology requires sophisticated signal processing

    and it is hard due to:

    - Complex input signals

    - Complex information encoding

    - Complex databases of knowledge

  • Highly sophisticated algorithms and

    huge processing power are required

Multimodal web

Multimodal Web

  • The trend towards perceptual information is noted at the W3C:

    Extending the Web to allow multiple modes of interaction: GUI, Speech, Vision, Pen, Gestures, Haptic interfaces, ...

  • Multimodal Interaction Activity:

    - Multimodal Architecture and Interfaces

    - EMMA

    - InkML

    - EmotionML

Multimodal architecture

Multimodal Architecture

W3c web technology day


  • Extensible Multimodal Markup Language for Annotations

    - containing and annotating the interpretation

    of user input

    - transcription into words of a raw signal, for

    instance derived from speech, pen

    - interpretation is to be generated by signal

    interpretation processes, such as speech and ink

    recognition, semantic interpreters

Ink markup language

Ink Markup Language

  • data format for representing ink

  • input and processing of handwriting,

    gestures, sketches, music using

    traces of pen



Emotion markup language

Emotion Markup Language

  • Annotation of material involving emotionality

  • Automatic recognition of emotions from sensors

  • Generation of emotion-related system responses: speech, music, colors, gestures, synthetic faces

  • Emotion vocabularies and representations:

    <emotion category-


    voc/xml#big6"> <category name="surprise"

    confidence="0.9 </emotion>

Department of signal processing

Department of Signal Processing

  • Signal processing has a key role as a front-end for the Multimodal Web

  • Department is on the forefront of research

    in the natural information processing:

    - Multimedia information analysis, retrieval and


    - Audio information analysis : speech and


    - Media information handling: representation and


W3c web technology day

for your attention!

  • Login