1 / 12

W3C Web Technology Day

Signal Processing for Multimodal Web Irek Defée Department of Signal Processing Tampere University of Technology. W3C Web Technology Day. Current status. Web is developed for traditional data and computer I/O: text, keyboard, mouse This is simple and effective but not a natural

kele
Download Presentation

W3C Web Technology Day

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Signal Processing for Multimodal WebIrekDefée Department of Signal Processing Tampere University of Technology W3C Web Technology Day

  2. Current status • Web is developed for traditional data and computer I/O: text, keyboard, mouse • This is simple and effective but not a natural way of human interaction with the world • Humans interact via perceptual system

  3. Human Perceptual System • Human perceptual system has multiple senses: visual, acoustical, haptic (touch, body position, temperature) and actuators (vocal tract, muscles, motoric system) • The perceptual system is intrinsically MULTIMODAL: multiple senses and actuators operate in perfectly coordinated way

  4. Perceptual Information Technology • Information technology is evolving towards natural MULTIMODAL human interaction: • Touch gestures revolutionized mobile devices • Intelligent speech input is available • There is more to come: new sensors, cameras and intelligence

  5. Signal Processing Role • Perceptual Information Technology requires sophisticated signal processing and it is hard due to: - Complex input signals - Complex information encoding - Complex databases of knowledge • Highly sophisticated algorithms and huge processing power are required

  6. Multimodal Web • The trend towards perceptual information is noted at the W3C: Extending the Web to allow multiple modes of interaction: GUI, Speech, Vision, Pen, Gestures, Haptic interfaces, ... • Multimodal Interaction Activity: - Multimodal Architecture and Interfaces - EMMA - InkML - EmotionML

  7. Multimodal Architecture

  8. EMMA • Extensible Multimodal Markup Language for Annotations - containing and annotating the interpretation of user input - transcription into words of a raw signal, for instance derived from speech, pen - interpretation is to be generated by signal interpretation processes, such as speech and ink recognition, semantic interpreters

  9. Ink Markup Language • data format for representing ink • input and processing of handwriting, gestures, sketches, music using traces of pen Trace attributes

  10. Emotion Markup Language • Annotation of material involving emotionality • Automatic recognition of emotions from sensors • Generation of emotion-related system responses: speech, music, colors, gestures, synthetic faces • Emotion vocabularies and representations: <emotion category- set="http://www.w3.org/TR/emotion- voc/xml#big6"> <category name="surprise" confidence="0.9 </emotion>

  11. Department of Signal Processing • Signal processing has a key role as a front-end for the Multimodal Web • Department is on the forefront of research in the natural information processing: - Multimedia information analysis, retrieval and databases - Audio information analysis : speech and music - Media information handling: representation and compression

  12. for your attention!

More Related