1 / 23

Multimodal research at UIAH

Multimodal research at UIAH. Kristiina Jokinen MediaLab University of Art and Design Helsinki Kristiina.Jokinen@uiah.fi. University of Art and Design. Education and research in the fields of design, new media, audiovisual communication, art education, arts Largest art school in Scandinavia

brad
Download Presentation

Multimodal research at UIAH

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multimodal research at UIAH Kristiina Jokinen MediaLab University of Art and Design Helsinki Kristiina.Jokinen@uiah.fi MUMIN Workshop

  2. University of Art and Design • Education and research in the fields of design, new media, audiovisual communication, art education, arts • Largest art school in Scandinavia • Nearly 1600 students, 15% from abroad • Lume, the Finnish centre for media research and development, was opened in 2000 • Departments of film and television, design for theatre, film and television, and new media MUMIN Workshop

  3. Media Lab • Formed in 1993 • Explore, discover and comprehend the new digital technology • Impact in society • Possibilities for communication, interaction and expression • Challenges to new media and information design • 2-year masters programme • MA in New Media (full time) • New Media program for professionals (flexible study method) • 20 full-time, 15 professionals, and approx. 30 minor subject students annually to the MA programmes MUMIN Workshop

  4. Adaptive Systems for Complex Interaction • Research on natural interaction between humans and computers • Interaction strategies, cooperative response planning • Various input modalities • Concepts, models • Apply machine-learning techniques to dialogue processing • Compare and test applicability of the techniques • Support Design-for-all principles in designing intelligent interfaces MUMIN Workshop

  5. Human-computer interaction • Computer as a tool • Passive and transparent • Supports the human goals, human control • Computer as an agent • Models of beliefs, desires, intentions (BDI) • Intelligent software mediating between the human and an application • Cooperation, negotiation • Complex interaction • Multimodal communication MUMIN Workshop

  6. Projects • USIX-Interact: Natural Interaction and Adaptive Methods http://www.mlab.uiah.fi/interact/ • DUMAS: Dynamic User Modelling for Adaptive Speech Interfaces http://www.sics.se/dumas/ • MUMMI: Multi-Modal Museum Interfaces (Study project together with Marjo Mäenpää and Antti Raike, Design for All, Virtual Art Exhibition) http://mlab.uiah.fi/mummi/ MUMIN Workshop

  7. Natural interaction • Language that suits to computers • Language that humans use to communicate => Language that humans and computers use when interacting with each other - Different ? How? - How it emerges from interaction? MUMIN Workshop

  8. Interact: key aspects for adaptivity • Conversational ability • Dialogue modelling and natural communication • Learning systems • Various methods and techniques • Various interface techniques (speech, text, map) • Language technology • Finnish and multilingual • Agent-based architecture • Jaspis development platform • http://www.mlab.uiah.fi/interact/ MUMIN Workshop

  9. University of Art and Design, Media Lab University of Helsinki, Language Technology University of Tampere, TAUCHI unit Helsinki University of Technology, Neural Networks Research Centre Fujitsu Invia oyj Tecnomen oyj Lingsoft oy The Arla Institute Finnish Association for the Deaf Finnish Technology Agency Interact Partners MUMIN Workshop

  10. DUMAS - Dynamic Universal Mobility for Adaptive Speech interfaces • EU 5th framework R&D project • Swedish Institute of Computer Science • UIAH, Media Lab • University of Tampere, TAUCHI-unit • UMIST, Manchester • ETEX, Frankfurt • Conexor oy, Helsinki • Timehouse oy, Helsinki • KTH, Stockholm • http://www.sics.se/dumas MUMIN Workshop

  11. Interactive email application Dynamic – various capabilities Universal – various situations and formats Mobility – various mobile applications for Adaptive – learning systems Speech – spoken and text input Interfaces – intelligent interaction DUMAS Objectives MUMIN Workshop

  12. Goals • Development of speech-based applications • multilingual: Finnish, Swedish, English • adapts to the user’s needs and habits • Main application: AthosMail • Experiments at the end of the project: • Athos-radiostation • Athos-text-TV • UIAH responsibility: user modelling components

  13. Challenges for User Modelling • Learn from the user-computer interaction those aspects that are important in making the use flexible and enjoyable: • cognitive load • speaking habits • dialogue strategies • User Model is involved in almost all decision making from speech recognition to dialogue management to speech synthesis • Classification and learning methods e.g. • neural networks, Bayes-nets, reinforcement learning

  14. Learning via Interaction • Situation: language is activity between rational agents (cf. Allwood) • Contact + perception + understanding + reaction • Task: achieve a communicative goal • maximise mutual comprehensibility • minimise ambiguity • Constraints: • language is possessed by a group of agents => cooperation • limited resources => adaptation to new situations • Reinforcement learning • agent takes an action a, finds itself in a state s, and receives a reward r • The task is to find a policy that maximizes the agent's reward in an environment MUMIN Workshop

  15. Adaptive multimodal interfaces • What and when to adapt? • User-centred parameters: • Habits and preferences • Attitudes and intentions • Environmental parameters: • E.g. speech recognizer accuracy MUMIN Workshop

  16. Action paths for an average user MUMIN Workshop

  17. MUMMI: Multimodality and Museum Interfaces • Study project together with Marjo Mäenpää, Antti Raike • Cooperation with the Finnish National Gallery: Marjatta Levanto, Riikka Haapalainen • New ways of relating the arts that are both visually interesting and accessible in terms of contents: • Virtual art exhibition, interactive guiding of the user through the exhibition • Text, speech, signing avatar • Design for all • Accessibility to the virtual visitors on museum web sites MUMIN Workshop

  18. Design for All • Places the user in the centre (user-centred design) • Cognitive factors (perception, memory, learning, problem-solving, etc.) as they come into play during interactions with things • Usefulness: what is relevant • do the functions, information, etc., match what the user actually needs? • Usability: ease-of-use • a simple concept, but not always easy or intuitive to implement • New ways to interact with computers? MUMIN Workshop

  19. Other Multimodal Projects at MediaLab • QuiQui’s Giant Bounce (Kukakumma Muumaassa) • Perttu Hämäläinen, Johanna Höysniemi • http://www.kukakumma.net/ • use your body to play • interaction with your body • child-centred design MUMIN Workshop

  20. Other Multimodal Projects at MediaLab • Cinemasense (Elokuvantaju) • Antti Raike • http://elokuvantaju.uiah.fi/ • web portal for film production (learning material) • organise cinematic concepts in the student’s mind • especially sign language MUMIN Workshop

  21. Other Multimodal Projects at MediaLab • Experimental, affective interfaces • Jukka Ylitalo, Heidi Tikka • http://mlab.uiah.fi/eia/ • interactive media, media and art MUMIN Workshop

  22. Burning Issues • Conversational interfaces • Dialogue processing: turn taking, feedback, repairs, non-verbal elements • Architectures • Learning in agent-based architectures • How to plug-&-play? • Processing techniques • Cognitive models of language understanding • Machine learning (supervised vs. unsupervised) • Design for all • Usability: for whom, why, what • Evaluation MUMIN Workshop

  23. References • Cinemasense http://elokuvantaju.uiah.fi/ • DUMAS http://www.sics.se/dumas • Experimental interfaces http://mlab.uiah.fi/eia/ • Interact http://www.mlab.uiah.fi/interact/ • MUMMI http://mlab.uiah.fi//mummi/ • QuiQui http://www.kukakumma.net/ • Jokinen et al. (2002). Adaptive Dialogue Systems – Interaction with Interact. Proceedings of the 3rd SIGDial Workshop, Philadelphia, US. • Jokinen, K., J. Rissanen, H. Keränen, and K. Kanto (2002). Learning interaction patterns for adaptive user interfaces. The 7th ERCIM UI4All Workshop, October, Paris, France. • Jokinen, K. and A. Raike (2002). Multimodality – the latest technology and visions and demands for the future. Multimodality IT-seminar, Castberggård, Denmark. MUMIN Workshop

More Related