Interface technologies and cognitive systems
1 / 72

Interface technologies and cognitive systems - PowerPoint PPT Presentation

  • Uploaded on

Interface technologies and cognitive systems. Philippe Gelin European Commission - Information Society. Granada 16th December 2002. Content. PART I : European Research Area PART II : Interface Technologies PART III : Cognitive Systems PART IV : FP6 Instruments. PART I.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Interface technologies and cognitive systems' - thuyet

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Interface technologies and cognitive systems
Interface technologies and cognitive systems

Philippe Gelin

European Commission - Information Society

Granada 16th December 2002


  • PART I : European Research Area

  • PART II : Interface Technologies

  • PART III : Cognitive Systems

  • PART IV : FP6 Instruments

Part i


The European Research Area (ERA)


its Sixth Framework Programme (FP6)

Interface technologies and cognitive systems

  • The Treaty establishing the European Community provides for the creation of multiannual research and technological development

  • The Sixth Framework Programme (FP6) is the European Unions main instrument for the funding of research in Europe within 2002-2006.

  • The overall budget for FP6 is 17.5 billion euro.

Interface technologies and cognitive systems

  • In the past, collaborative actions have been initiated at European and Community level,

  • but now is the time to bring our endeavours together and to build a research and innovation equivalent of the "common market" for goods and services.

  • That structure is called the European Research Area (ERA) and is regrouping all Community supports for the better coordination of research activities and the convergence of research and innovation policies, at nationaland EU levels.

From era to ist
From ERA to IST

1. Focusing and integrating Community research 1.1. Priority thematic areas of research 1.1.1. Life sciences, genomics and biotechnology for health (i) Advanced genomics and its applications for health. (ii) Combating major diseases. 1.1.2. Information Society Technologies(3.625 billion Euro)1.1.3. Nano-technologies and nano-sciences, knowledge-based multifunctional materials, and new production processes and devices 1.1.4. Aeronautics and space 1.1.5. Food quality and safety 1.1.6. Sustainable development, global change and ecosystems (i) Sustainable Energy Systems (ii) Sustainable surface transport (iii) Global change and ecosystems 1.1.7. Citizens and governance in a knowledge-based society 1.2. Specific activities covering a wider field of research 1.2.1. Policy support and anticipating scientific and technological needs 1.2.2. Horizontal research activities involving SMEs 1.2.3. Specific measures in support of international cooperation 2. Strengthening the foundations of the European Research Area 2.1. Support for the coordination of activities 2.2. Support for the coherent development of policies

Ist programme 2002 2006
IST Programme 2002-2006

  • Instantiate the European challenges (ERA), Initiatives

  • Trans national partnerships

  • Critical mass of skills

    and adequate resources hence ...

  • Focus, Excellence, Impact and RoI

Ist vision
IST Vision

IST today …………………………..The IST in FP6 vision

  • PC based ………………………………. “Our surrounding” is the interface

  • “Writing and reading”…………….…. …Use all senses, intuitive

  • “Word” based information search……….Context-based knowledge handling

  • Low bandwidth, separate networks…. ….Infinite bandwidth, convergence, ..

  • Mobile telephony (voice)………….. ……Mobile/Wireless full multimedia

  • Micro scale………………………………Nano-scale

  • Silicon based……………………………..+ new materials

  • e-Services just emerging……………….. Wide adoption (eHealth, Learning,..)

  • < 10% of world population on-line….. …World-wide adoption

Interface technologies and cognitive systems

Knowledge andInterface Technologies

Communication ComputingSoftware

Future andEmerging Technologies

Technologies for major economicand societal challenges



Strategic objectives in 2002 2003 wp

Call 2

Advanced displays

Optical, opto-electronic, & photonic functional components

Open development platforms for software and services

Cognitive systems (25ME)

Embedded systems

Applications and services for the mobile user and worker

Cross-media content for leisure and entertainment (55ME)

GRID-based Systems for solving complex problems

Improving Risk management

Call 1

Pushing the limits of CMOS, preparing for post-CMOS

Micro and nano-systems

Broadband for all

Mobile and wireless systems beyond 3G

Towards a global dependability and security framework

Multimodal interfaces (65ME)

Semantic-based knowledge systems

Networked audiovisual systems and home platforms

Networked businesses and governments

eSafety of road and air transports

e Health

Technology-enhanced learning and access to cultural heritage (65ME)










Strategic Objectives in 2002-2003 WP

80% of budget is pre-allocated

Remainder flexible, to be allocated after evaluation

Allocation only for the first 2 years: to be revised for 2005-2006

Part ii interface technologies in fp 6 scope and focus in 2003

PART IIInterface Technologies in FP 6scope and focus in 2003


  • Why interface technologies?

    • IST Vision

  • What research?

    • Focus of first calls in FP6

  • Where are we today?

    • Ongoing activities


  • Building the knowledge society for all

    • Allow people to use the surrounding hi-tech easily and in a way that is consonant to humans rather than to computers

    • Facilitate interpersonal communication anywhere, anytime, beyond linguistic and cultural barriers

    • Assist people and augment their abilities when interacting and communicating


  • Bring people to the foreground – build technologies for the background

  • Our surrounding is the interface: everywhere, embedded in everyday objects (e.g: clothes, furniture, vehicles, smart materials, …)

  • Exploring interfaces beyond the PC, screens & keyboard. (brake the chains…)

  • Enabling multi-sensorial dialogues supported by ubiquitous computing & networking

  • Multi-linguistics, multi-cultural interactions

Known challenges
Known Challenges

  • Strongly multidisciplinary with many component technologies

  • Significant system level integration issues

  • Research resulting in autonomous (self-learning, self-organising), adaptive (time varying) systems that can work with partial and uncertain information

2003 2004 focus

Multimodal Interfaces (in call 1)

Objective: To develop natural and adaptive multimodal interfaces, that respond intelligently to speech and language, vision, gesture, haptics and other senses.

Focus 1 Multimodal interaction

Focus 2 Multilingual communication

2003-2004 FOCUS

Interface technologies and cognitive systems

Interaction between and among humans and the virtual and physical environment

intuitive multimodal interfaces that are autonomous and capable of learning and adapting to the user environment in dynamically changing contexts. They should recognise emotive user reaction and feature robust dialogue capability with unconstrained speech and language input.

Multimodal Interaction

  • Human-to-human: technology mediated communication

  • Human-to-things: virtual and physical

  • Human-to-self: health, well being

  • Human-to-content: information retrieval/browsing

  • Device-to-device: human mediated device communication

  • Human-to-embodied robots

  • Interface technologies and cognitive systems

    facilitating physical environmenttranslation for unrestricted inputs, especially for spontaneous (unrestricted) or ill-formed (speech) inputs, in task-oriented settings

    Unrestricted spontaneous speech-to-speech translation in task-oriented settings

    Statistical/mixed approaches to translation

    Adaptation to task/user, learning


    Multilingual Systems

    Basic research

    Basic research, component technologies physical environment


    accurate vision

    gesture tracking

    speech and audio processing

    language technologies

    affective computing

    machine learning

    autonomous systems

    fusion of multiple channels

    Basic Research

    System level integration

    Proof of concept in challenging application domains, including:

    wearable interfaces and smart clothes

    intelligent rooms and interfaces for collaborative working tools

    cross-cultural communications

    usability issues and evaluation

    System Level Integration

    Shared infrastructure

    Data including:: large amounts of multimodal data, synchronisation and IPR issues

    Metrology, technology evaluation, usability

    Infrastructural knowledge: machine learning, applied mathematics

    Best practice and standards

    Tools and platforms

    Socio-economic issues (e.g. human factors)

    Shared Infrastructure

    Current status

    Current Status including:

    Ongoing activities

    Collection, processing and browsing of multimodal meeting data

    systems that enable recording, structuring, browsing and querying of an archive of multi-modal recordings of meetings.

    Multimodal, multicultural, multilingual communication

    integration of multiple communications modes - vision, speech and object manipulation - by combining the physical and virtual worlds to provide support for multi-cultural communication and problem solving

    Speech-to-speech translation

    development of speech-to-speech translation and its integration in automatic in e-commerce and e-service environments

    data collection for speech-to-speech translation

    Ongoing Activities

    Ongoing activities1

    Preparing future multisensorial interaction research data

    providing technological baselines, comparative evaluations, and assessment of prospects of core technologies for speech-to-speech translation, the detection and expressions of emotional states, and technologies for children’s speech.

    Automatic animation of human models

    design and development of a virtual person animation system in controlled environments, which enables the modeling, analysis and simulation of human motion.

    Recognition of the user’s emotional state

    human computer interaction that can interpret its users' attitude or emotional state in terms of their speech and/or their facial gestures and expressions.

    See for more information

    Ongoing Activities

    Part iii

    PART III data

    Cognitive Systems in FP6scope and focus

    Outline data

    • why cognitive systems?

      IST vision

    • what research?

      focus of call

    • where are we starting from?

      current activities

    Vision data

    Ambient intelligence

    “… ability of computationally empowered devices to interconnect with each other and with us …”

    • sensors provide a window from world of interconnected computation into real physical world

    • using sensors, these devices will sense the world around us and respond by interacting with the world or by communicating with us

    • they need to ‘see’, ‘hear’, …. ‘understand’ and ‘act’

    • Ambient Intelligence requires perceptual systems capable of cognition

    Objective data

    To constructphysically instantiatedorembodiedsystemsthat can :

    • perceive,

    • understand(the semantics of information conveyed through their perceptual input)

    • learning (structured memory)

    • interactwith their environment, (CCI & CHI communication)

    • evolvein order to achieve human-like performance in activities requiring context-(situation and task) specific knowledge.

    Focus 2002 2003
    Focus 2002-2003 data

    • Research will aim at realizing:

      • completesystems with

      • real-timeperformance and/or bounded rationality,

      • have well developedmemory capacities(e.g. short term, long term, iconic, associative) with efficient representation,

      • and thatacquire representationsas needed to realise performance goals.

      • to performgoal-directedtasks

      • The emphasis is on closing the loop in realistic test cases.

    Known challenges1
    Known Challenges data

    • A main target of this research isinterdisciplinarity:

      • computer vision,

      • natural language understanding,

      • robotics,

      • artificial intelligence,

      • mathematics and

      • cognitive neuroscience

    • and its impact on overall system design.

    Outcomes data

    • Direct Applications:

    • in image recognition, in behavioural interpretation, in video annotation, speech recognition, in automatic categorisation and classification, in goal-specification and decision support,…

    • can serve to demonstrate and measure progress but applications NOT main target of research

    • Methods & approaches for constructing robust & adaptive systems capable of cognition

    Ongoing activities2
    Ongoing Activities data

    • cognitivevisionsystems 2000 – 2002

      • robust image recognition

      • from application-specific togeneralised solutions

      • from focus on low-level processing & robustness of individual components to systems approach where every component (incl. high-level cognitive functionalities) has a role to play in assuring robust behaviour of system

      • longer-term, largely academic, interdisciplinary effort

    • 8 RTD projects & 1 Network:

    Related topics
    Related Topics data

    • Multimodal Interfaces, call 1

      intuitive multimodal interfaces ... autonomous and capable of learning and adapting … in dynamically changing contexts

    • Beyond Robotics, call 1

      focus on open-ended lifelong learning systems

    • Disappearing Computer, call 1

      focus on open architectures allowing arbitrary combinations of ‘building blocks’ for ‘universal application’

    • Embedded Systems, call 2

      focus on networked systems, distributed control

    Part iv

    PART IV data


    Outline data

    • Schedule

    • New Instruments

    • Old Instruments

    • Applied to IFT & CS.

    • Expected results

    • Learning from EoI

    • “Do’s and don’ts”

    • Conclusions

    Fp6 roadmap
    FP6 ROADMAP data

    • 21 February 2001 Commission decides on first proposal for Framework Programme

    • March 2001 Informal presentation to the Ministers of Research and Education during the Uppsala Informal Council (1 - 3 March)

    • May 2001 Adoption by the Commission of proposals regarding the specific programmes and the implementation of the Framework programme

    • September 2001 Adoption by the Commission of proposals for decisions on the rules for participation and dissemination of results

    • 26 June 2001 Council discusses Commision proposal

    • July/Nov. 2001 First reading by the European Parliament

    • 12/2001- 01/2002 Common position of the Research Council

    • 11 March 2002 Council adopt final text

    • 15 May 2002 The European Parliament approves the Sixth Framework Programme

    • 3 June 2002 Council of Ministers formally adopted the Sixth Framework Programme

    • 12 June 2002 Parliament approves specific programmes

    • 19 June 2002 FP6 rules of participation compromise approved by ITRE

    • 3 July 2002 FP6 rules of participation compromise approved by EP

    • September 2002 Specific Programmes has been adopted (October)

    • End 09/2002 Rules for Participation to be formally adopted by Council and Parliament

    • November 2002 Conference marking the launch of the Sixth Framework Programme

    • 12/2002 First calls for proposals

    • end 2003 Second calls for proposals

    Call 1 timetable

    ! data









    Call 1 Timetable

    Publication: 2nd half December 2002

    Call deadline: 24 April 2003

    Evaluation: weeks of 12th and 19th May 2003

    IP/NoE hearings: week of 9th June 2003

    Negotiations: from July 2003

    Commission Decision: from October 2003

    Project start: January 2004

    Overview of fp6 instruments
    Overview of FP6 instruments data

    • Integrated Projects (IP)

    • Networks of Excellence (NoE)

    • Article 169 (joint implementation of national programmes)

    • Specific Targeted Research Projects (STReP’s)

    • Co-ordination actions (CA)

    • Specific Support Actions (SSA)



    Why new instruments
    Why new instruments ? data

    • Adapt to the size of the challenge

      • The philosophy of both instruments is to movefrom multiple projectfunding to the funding of coherent programmes of research activities,

    • Flexibility and responsibility

      • leaving the highest degree possible of autonomy and flexibility to European research consortia.

    Integrated projects
    Integrated Projects data

    • Expected to address the objectives within a holistic approach enabling,

    • where justified, competition within and across projects

    • A major target is to integrateactors and disciplines= build a community

    • An IP should be THE project in the target area

    • “critical mass”

    Integrated projects1
    Integrated Projects data

    • Core activities

      • genuine research work

      • “engineering” tasks (esp. methods & tools)

      • system integration & validation (“total system” approach)

    • along with

      • promotion & dissemination of results

      • training, awareness & best practice (researchers, integrators, launching users)

      • cooperation & exchanges with related nationaland international efforts (incl. standards bodies)

      • socio-economic impact & consequences

    Example of an ip

    Integrated project data

    Current projects

    Example of an IP

    Area: Multi-sensorial interfaces

    Knowledge & Data bases

    Virtual & Augmented Realities

    Intelligent Assistants & Virtual Humans

    Intelligent Virtual Worlds

    Physical Virtual World

    Interface Layer


    Gesture & Facial Recognition



    Body Sensors

    Haptic devices




    Interface technologies and cognitive systems

    Reporting data

    Accepted costs

    Detailed work plan


    Accepted costs

    Adjusted advance

    Accepted costs

    Detailed work plan


    Adjusted advance


    Adjusted advance

    Accepted costs

    Detailed work plan

    Initial advance

    Detailed work plan

    0 6 12 18 24 30 36 42 48 Months

    IP - payment & report schedule

    Example of 4 year contract

    Networks of excellence
    Networks of Excellence data

    • Spread excellence

    • Xfertilisation among disciplines

    • Bring together the best research in constituent disciplines, integrate resources, (e.g. image, video, multimodal,…)

    • Create a NEW research space (ERA)

    • Have a lasting nature

    • Define joint activities

    Networks of excellence1
    Networks of Excellence data

    • Joint programme of activities:

      • Integrating activities

        • coordinated programming of the partners’ activities

        • sharing of tools/facilities/ infrastructures

        • managing the knowledge portfolio

        • staff mobility and exchanges

      • Joint researchactivities

        • undertaking research on topics supporting the network’s goals

      • Spreading excellence

        • training of researchers and other key staff

        • dissemination and technology transfer to industry

    • “virtual” centres of excellence

    New or additionalto ongoing efforts

    Noe grant
    NoE - datagrant

    • Each call will fix the founding per researcher, based on the characteristics of the field of research concerned

      • Computer science less expensive than nuclear research

  • The number of researchers implied in the integration

    • PhD or more than 4 years of research

    • Existing head count at time of proposal.

    • Table of Found Per researcher issued at each call

  • @ max the cost incurred to implement the JPA

  • @ max 25% of the existing value proposed to be integrated

  • Specific targeted research projects
    Specific targeted research projects data

    • Similar to Old instrument, except for:

      • joint and several financial liability

      • costing / funding adapted to grant scheme

      • rates / activities: research (50%), demonstration (35%), combined research & demonstration

    • Activities

      • focussed RTD and / or demonstration; management

    • Partnership

      • more limited than for IPs; in terms of breadth, time, scope

    Coordination specific support actions
    Coordination & dataspecific support actions

    Coordination Actions

    • Instrument for ad-hoc co-operation between organisations (no durable integration !), similar to FP5 thematic networks

    • Activities include expert groups, seminars, studies, staff exchanges, project definition, etc

    • Can cover up to 100% of eligible costs, …

      Specific Support Actions

    • For conferences, working groups, fact finding, technology transfer, operational support and dissemination

    • Can cover up to 100% of eligible costs, and be carried out by one participant

    • Call for Grants: conferences, workshops or exhibitions

    Article 169
    Article 169 data

    • EC funding to support jointly executed member state programmes

      • most powerful tool for integration

    • However, long-winded implementation:

      • to generate a proposal requires a co-initiative by national programmes and the Commission

      • Council and Parliament would have to approve


    Expectation data

    Within Multimodal Interface


    Cognitive systems

    Expected outcome calls 1 and 2

    Multimodal Interfaces and Cognitive Systems data

    2/3rd of funds devoted new instruments

    6-10 new instruments

    8-12 old instruments










    Expected OutcomeCalls 1 and 2

    Interface technology tools
    Interface Technology Tools data

    • IPsare expected to address the objectives within a holistic approach enabling, where justified, competition within and across projects.

    • NoEsshould aim at lowering barriers between hitherto split communities and disciplines and advance knowledge in the field. They should help establish and reinforce shared infrastructures, including for training andevaluation, annotation standardsand appropriateusability metrics and benchmarks.

    • STREPsare expected to bootstrap research in identifiable or emerging sub-domains and to prepare associated communities.

    Cognitive system tools
    Cognitive System tools data

    • Integrated Projectsare expected to leverage these communities to integrate methods and insights towards the objective of realizing entire systems and topromote community building.

    • NoEswill provide a channel forfostering foundational research, for developing and maintaining common resources, specifically, ofopen systemsandtraining environmentsto study learning and evolving systems.

    Analysis of submitted eois 14 evenly populated clusters

    Infrastructures, resources and databasic technologies

    Interface and Artefact Design and Ergonomic, Human Factors and User Acceptance

    Cognitive Science for Ambient Intelligence

    Improvement of Human Senses and Perception

    Speech and Vision Based Intelligent Interfaces

    Interfaces Based on Recognition and Synthesis of Emotion, Facial Expression and Gesture

    Multilingual Access and Dialogue

    Intelligent Wearable Devices and Smart Cloths

    Interfaces for Mobile Devices and Services

    Intelligent Rooms with Support for Collaborative Creative Processes and Virtual Environments

    Multisensory interfacing with complex information spaces and virtual manipulation of information objects

    Virtual Reality and Environments

    Critical Interfaces for Safer Control of Vehicles and Devices, including Training for these Environments

    Physical Interfaces, Haptics, Mechatronics and Robotics

    Analysis of submitted EOIs14 ~evenly populated clusters

    Analysis of submitted eois

    150+ EoIs in Interface technologies (incl. Cognitive Systems)

    70% IPs, 30% NoEs

    60% of submitted IPs “defective”

    lack of required ambition, scope and integration

    some could be part of other IPs or a STRP

    30% of the submitted NoEs “defective”

    lack the main characteristics of this instrument

    e.g. joint activity programme, lasting integration of existing research capacity

    some more appropriate as coordination actions

    Analysis of Submitted EOIs

    Eoi per country
    EoI per Country Systems)

    Eoi shortcomings
    EoI Shortcomings Systems)

    • Content

      • Narrow scope, little or no EU and ERA dimension

      • Aims too general, lack of focus (e.g. programmes instead of projects)

      • Lack of innovation, not beyond current state of art

    • Planning

      • Links missing between objectives & workplan

      • Milestones missing or too general

      • Risk factors not addressed, no contingency plans

    • Management

      • Management structure vague or inadequate

      • Weak dissemination, exploitation plans

    Eoi partnership size
    EoI Partnership Size Systems)

    Nino Varile /

    How to partnership

    Highly competent Systems) and reliable partners

    Complementarity: cover all areas you need

    Duplication of competence:

    acceptable for IPs depending on project needs

    necessary for NoEs

    At least 3 of which 2 EU/NAS or associated states

    Ratio between Industry/SME/NAS/Academia participation depending solely on project needs

    How To: Partnership

    Do s and don ts

    Do Systems) carefully study the evaluation criteria

    relevance, excellence, ……

    Do respond to all the evaluation criteria, not just the scientific & technical ones

    Do remember that proposals are evaluated by peers = “no-nonsense”

    Don’t mirror the workprogramme: put your own ideas in the proposal

    Don’t participate in a multitude of competing proposals, select one and make it win

    Do’s and Don’ts

    Do s and don ts1

    Do Systems) pay extra attention to using the full range of activities allowable for the new instruments

    Don’tcreate artificial IPs (=inflated STREPs)

    Do pay due attention to co-ordination of large projects

    Do give realistic cost/resource estimates good return on investment

    Rememberthat50 MEUR ~ 1,200 person/years!

    Do’s and Don’ts

    Reminder Systems)

    • FP6 is about more ambitious endeavours, in a simpler procedural environment

    • IPs should aim at breakthroughs in the target area and yield new technologies, products and services

    • NoEs are to structure and shape research areas (interdisciplinary !)

    • STREP’s are more limited in scope but must go well beyond the state of the art

    Ist infopack
    IST Infopack Systems)

    • IST Workprogramme 2003-2004

    • Call for proposals text Call 1 (Official Journal)

      • Submission details (deadline, address...)

    • Brochure “The 6FP in Brief”

      • FP6 and Specific Programmes, instruments, proposal to contract (flowchart), contractual matters, etc

    • Guides for proposers Call 1

      • How to submit, references,..., proposal forms

      • Instrument-specific annexes

    • Evaluation Manual

      • “High level” description of ground rules for evaluation

    • Guidelines for Evaluators Call 1

      • Evaluation process, evaluation forms

    Proposal content
    Proposal content Systems)

    Part A

    • Form A1 - General information

    • Form A2 - Information on participants

    • Form A3 - Cost breakdown (cost and grant requested as per activity / participant)

      Part B

    • Aspects relating to evaluation criteria

    • Other aspects (ethics, safety, gender)

    • Overall workplan of project

      IPs, NOEs additionally provide

    • 18 month Implementation plan/JPA (detailed)

    No anonymity

    Mats Ljungqvist /


    ! Systems)










    • Process

      • Peer-review system (many experts needed !)

      • 2-step evaluation (not in Call 1)

      • Hearings of applicants (likely in Call 1)

    • Evaluation & selection criteria

    Higher thresholds on partnership and management.

    Higher weights on impact and excellence.

    Mats Ljungqvist /

    Stay in touch
    Stay in Touch! Systems)





    • email:




    Contact info
    Contact info Systems)

    Express Courier & visitors:

    European Commission

    EUFO 0170

    Euroforum, Building,

    10, rue Robert Stumper

    L-2557 Luxembourg

    Official address:

    European Commission,

    EUFO 0170

    Jean Monnet Building,

    Rue Alcide de Gasperi,

    L-2920 Luxembourg.

    Tel.: +352 - 4301 - 33423,

    Sec.: +352 - 4301 - 34178.

    Fax : +352 - 4301 - 34999


    Thank you

    Thank You Systems)