EU research initiatives in multimodal interaction

1. EU research initiatives in multimodal interaction Mats Ljungqvist European Commission Information Society DG,Interfaces (Unit E1)

2. Outline of Presentation • Introduction • Who can participate? • “The story so far….” • FP5 activities • Overview FP6 call 1, 2 and 3 • Call 1: Multimodal Interfaces • Call 2: Cognitive Systems • Call 3: FET - Future and Emerging Technologies • The IST workprogramme for 2005-2006 • Calls 4 and 5 • Preparation of the 7th EU Framework Programme 2007-2011

3. Applied IST Research addressing Major Societal & Economic Challenges IST in FP6(~4 Billion Euro,2003-2006) Components & Micro-Systems Communication & Computing Infrastructures Knowledge & Interface Technologies

4. Who can participate? • All member states • States associated to FP6: same conditions as member states • Bulgaria, Romania, Turkey, Iceland, Israel, Liechtenstein, Norway, Switzerland • "INCO countries“:specific earmarked funding budget • Russia, New Independent States, Mediterranean Countries, Western Balkans, Developing countries • Countrieswith co-operation agreement participate without EU funding (exceptionally with funding) • Argentina, Australia, Brazil, Canada, China, Chile, India, Japan, Kazakhstan, Russia South-Africa, Ukraine, USA • Other countries participate without funding (exceptionally with funding).

5. Who can participate? • Another possibility • Join a project as unfunded “external member” sometimes possible according to modalities set up by the consortium. • Examples in some language resources projects (China, Canada, USA, …). • Examples of funded and unfunded participation (from 1st FP6 MMI call) • Russia, USA, Senegal, Romania, Turkey, ….

6. Cognition and Interfaces • Building on activities in earlier framework programmes (…,ESPRIT, Telematics, FP5) • Human Language Technologies • speech, multilinguality, translation • Multimodal, multisensorial dialogue • Signal processing and coding • Machine vision • Affective computing • Robotics • Disappearing computer • Neuroinformatics, presence research • Focus in FP6: • Multimodal Interfaces (MMI) • Cognitive Systems (CS) • Beyond robotics, life-like perception systems, Bio-inspired Intelligent Information Systems (in FET) Interface related work also in FP6 application areas

7. FP5 projects • Projects in FP5 are laying the foundations for FP6 initiatives in the area: • KERMIT Kernel Methods for Images and Text • M4 MultiModal Meeting Manager • Nespole! Negotiating through SPOken Language in E-commerce • FAME Facilitating Agent for Multicultural Exchange • PF_STAR Preparing Future Multisensorial Interaction Research • SIRIDUS Specification, Interaction and Reconfiguration In Dialogue Understanding Systems • ALERT Alert System for Selective Dissemination of Multimedia Information • CORETEX Improving Core Speech Recognition Technology • VISATEC Vision-based Integrated Systems Adaptive to Task and Environment with Cognitive abilities • ECVISION European Research Network for Cognitive AI-enabled Computer Vision System • LAVA Learning for Adaptable Visual Assistants • LC_STAR Lexica and Corpora for Speech-to-Speech Translation Technologies • SPEECON Speech Driven Interfaces for Consumer Applications • SPEECHDAT Speech Databases for Creation of Voice Driven Teleservices • ORIENTEL Multilingual access to interactive communication services for the Mediterranean and the Middle East • NECA A Net Environment for Embodied Emotional Conversational Agents • HUMODAN An automatic human model animation environment for augmented reality interaction • SMARTSKETCHES A multimodal approach to improve usability in the early stages of product design • FET initiatives • …and more at… www.cordis.lu/ist -->projects Learning Multimodality Speech/ language Vision Resources Emotions Haptics

8. Call 1 overview Multimodal Interfaces (MMI) Develop natural and adaptive multimodal interfaces, that respond intelligently to speech and language, vision, gesture, haptics and other senses • Interaction between and among human and the virtual and physical environment • Multilingual systems • Results • 14 proposals selected for negotiations • Most contracts signed December 2003 • Funding • 73 Meuro

9. MMI call results: NoE (Networks of Excellence) • SIMILAR - The European taskforce creating human-machine interfaces SIMILAR to human-human communication • multimodal interface research, multiple sensory modalities, fusion of modalities: visual interaction, interactive speech, interactive haptics, learning, and human human-computer interaction (HCI) • HUMAINE - Human-Machine Interaction Network on Emotion • study of systems that can register, model and/or influence human emotional and emotion emotion-related states - central to future interfaces. • ENACTIVE - Enactive Interfaces • interfaces and interaction paradigms based onEnactive Knowledge, i.e., knowledge stored in the form of motor responses and acquired by the act of "doing” - a form of cognition inherently tied to actions • PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning • pattern analysis, statistical modelling, and computational learning as core enabling technology for discovering, structuring and presenting complex information, or for processing of complex sensory data. Academic research in key enabling technologies

10. MMI call results: IP (Integrated Projects) • CHIL - Computers In the Human Interaction Loop • create environments in which computers serve humans who focus on interacting with other humans • AMI - Augmented Multi-party Interaction • new multimodal technologies to support human interaction, in the context of smart meeting rooms and remote meeting assistants. • TC-STAR - Technology and Corpora for Speech to Speech Translation • advanced research in all core technologies for Speech-to-Speech-Translation: speech recognition, translation, and synthesis. - supporting human-human interaction - perception in multiple modalities

11. MMI call results: STREP (Strategic Targeted Research Projects) • MATRIS -Markerless Real-time Tracking for Augmented Reality Image Synthesis • a system for markeless tracking, by analysis of the camera images and exploitation of unobtrusive inertial motion sensors. This will enable the system as a whole to determine its location and orientation mimicking the way a human orients himself. • DIVINES - Diagnostic and Intrinsic Variabilities in Natural Speech • improving speech recognition performance by research into feature extraction and modelling techniques that would have better capacities, particularly in handling speech intrinsic variabilities • TALK - Talk and Look, Tools for Ambient Linguistic Knowledge • to advance our understanding of generic technologies that will extend the Information State Update" (ISU) approach to adaptive multimodal and multi multi-lingual interaction for developing adaptive multimodal dialogue systems. • T’nD - Touch and Design • shape generation and modification based on novel haptic interaction and intelligent shape manipulation operators in order to exploit existing manual skill of designers - small focussed efforts - bootstrapping new research areas

12. MMI call results: STREP, SSA (Specific Support Action) • TAI-CHI - Tangible Acoustic Interfaces for Computer-Human Interaction • development of tangible interfaces that employ physical objects and space as media to bridge the gap between the virtual and physical worlds by exploring how physical objects, augmented surfaces and space can be transformed into tangible acoustic embodiments of natural seamless unrestricted interfaces. • HIWIRE - Human Input That Works In Real Environments • embedded robust multimodal dialogue systems with flexible speech input in mobile, open and noisy environments (e.g. aeroplanes, cars, street etc.). • MWEB - Multimodal Web Interaction • support for the development and adoption of industry standards (W3C Recommendations) enabling multimodal Web access using mobile devices.

13. Call 2 overview Cognitive Systems Construct physically instantiated or embodied systems that can perceive (‘see’, ‘hear’, ’smell’), understand the semantics of information conveyed through their perceptual input (‘recognise’, ‘categorise’) and interact with their environment (‘decide’, ‘act’), and evolve in order to achieve human-like performance in activities requiring context specific knowledge. • Artificial systems that combine perception, action reasoning, learning and communication • Provide enabling technologies for robotics, natural language understanding, man-machine interaction, complex real world systems • Results • Negotiations with 8 projects concluded • Most contracts to be signed in July-August 2004 • Funding • Around 25 Meuro

14. What is needed? • Cognitive Systems • Extensive foundational work of exploratory nature • “Cover ground” (as we don’t really know yet what is important) • The nature of cognition, knowledge, the role of language • Architectures • Perception, learning • Autonomy, notion of self, social interaction • Interdisciplinary partnerships • Multimodal Interfaces • Significant advances in key areas: • “Emotional interfaces” • Interfaces that are anticipatory, learning, evolving and emerging: the “Do what I mean” paradigm • Seamless control of IT enabled intelligent environment • Fusion/synchronisation of multiple modalities • Real-time, better than human interpretation of everyday language

15. IST in FP6: Work Programme 2005-2006 • Evolution of running Strategic Objectives • Calls in Dec 2004 or 2005 • Multimodal interfaces call in Dec 2004 or in 2005 ~50 m Euro? • Cognitive systems ~50 m Euro? • Applications and Services for the Mobile User & Worker • Open Web Consultation and Consultation Meetings in April 2004 • Reports will be made available www.cordis.lu/ist • IST-FET: New Research Initiatives • Bio-inspired Intelligent Information Systems • ~20 m Euro?, Call closes on 22 Sept • Presence and Interaction in Mixed Reality Environments • ~20 m Euro?, Call in Dec 2004? • Creating and Moulding Ambience • Call in 2005? • New IST Work programme: autumn 2004 www.cordis.lu/ist

16. 2004 2003 2005 2006 6th FP The Roadmap to FP7 2007 – 2011 ... 7th FP Feb ’04 Financial Perspectives(2007-2013) Early ’05 Proposalon FP Adoption FP 12 May ’04Preparing the future: reinforcing European Research Policies

17. FP7: 6+2 axes • Collaboration Collaborative Research • Private/public partnership Technological platforms • Individual research teams Basic Research Grants • Human resources Mobility and Life Long Learning • Research Infrastructures Access/Networking & support of new Infrastructures • Co-ordination of national & European Research Area regional research programmes and policies + • Space (ESA) • Security ICT research maps into all 6+2 axes

18. Next calls Call 3 now open (closes 22 September) Call 4 in late 2004, call 5 in 2005 Based upon 2005-06 Work Programme (Oct/Nov 2004) Contact www.cordis.lu/ist/so/interfaces mats.ljungqvist@cec.eu.int Contacts IST helpdesk Fax : +32 2 296 83 88 e-mail : ist@cec.eu.int