1 / 14

eNTERFACE’08 Multimodal Communication with Robots and Virtual Agents

eNTERFACE’08 Multimodal Communication with Robots and Virtual Agents. Overview. Context: Exploitation of multi-modal signals for the development of an active robot/agent listener Storytelling experience : Speakers told a story of an animated cartoon they had just seen.

calais
Download Presentation

eNTERFACE’08 Multimodal Communication with Robots and Virtual Agents

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. eNTERFACE’08Multimodal Communication with Robots and Virtual Agents

  2. Overview Context: • Exploitation of multi-modal signals for the development of an active robot/agent listener • Storytelling experience : • Speakers told a story of an animated cartoon they had just seen 2- Tell the story to a robot or an agent 1- See the cartoon

  3. Overview Active listening : • During natural interaction, speakers see if the statements have been correctly understood (or at least heard). • Robots/agents should also have active listening skills… • Characterization of multi-modal signals as inputs of the feedback model: • Speech analysis : prosody, keywords recognition, pauses • Partner analysis : face traking, smile detection • Robot/agent feedbacks (outputs): • Lexical non-verbal behaviors • Dialog management: • Feedback model: exploitation of both inputs and outputs signals • Evaluation: • Storytelling experiences are usually evaluated by annotation

  4. Organization: Workpackages: • WP1: Speech feature extraction and analysis • WP2: Partner analysis: face tracking and analysis • WP3: Robot and Agent Behavior Analysis • WP4: Dialog management for feedback behaviors • WP5: Evaluation and Annotation • WP6: Deliverables, reports.

  5. Speech Analysis Automatic detection of prominence during the interaction Computational attention algorithms:

  6. eNTERFACE’08Multimodal Communication with Robots and Virtual Agents Speech analysisfor prominencedetection

  7. Computational attention algorithms

  8. Computational attention algorithms

  9. Computational attention algorithms • Have more recently been tested for audio event detection M. MANCAS, L. COUVREUR, B. GOSSELIN, B. MACQ, 2007, "Computational Attention for Event Detection", Proceedings of ICVS Workshop on Computational Attention & Applications (WCAA-2007) , Bielefeld, Germany, Mar 2007. • In this project, we intend to test it for the automatic detection of salient speech events, for triggering avatar/robot feedback • Underlying hypothesis: listener is a child, with limited language knowledge  test the bottom-up approach, as opposed to the more language-driven top-down approach: A Top-down Auditory Attention Model For Learning Task Dependent Influences On Prominence Detection In Speech,Ozlem Kalinli and Shrikanth Narayanan, ICASSP’08, 3981-3984.

  10. Partner analysis Analysis of human behaviour (non-verbal interaction). Development of a component able to detect the face and key features of feedback analysis: shaking head, smiling… Methodology: Face detection: Viola & Jones face detection, Head shaking: frequency analysis of interest points Smile detection: Combining colorimetric and geometric approaches

  11. Robot and Agent Behavior Analysis Integratation of existing tools to produce an ECA/robot able to display expressive backchannels. The ECA architecture follows the SAIBA framwork. It is composed of several modules connected to each other via a Representation Language. The language FML (Functional Markup Language) connects the module 'intent planning' to 'behavior planning' and BML (Behavior) connects 'behavior planning to 'behavior realiser'. Modules are connected via psyclone, a white board architecture. Tasks: • define the capabilities the ECA/robot ought to have • create BML (Behavior Markup Language) entries for the lexicon • integrate modules that will endow ECA with such expressive capabilities. • work out carefully the synchronization scheme between modules, in particular between modules of Speaker and of Listener

  12. Dialog Management Development of a feedback model with the respect of the input signals (common) and the output capabilities (behavior) Methodology: • Representation of input data: • EMMA: Extensible MultiModal Annotation markup language • Definition of task-oriented representation • Dialog management: • State Chart XML (SCXML): State Machine Notation for Control Abstraction • Interpretation of the speaker’s conversation

  13. Evaluation and Annotation Investigate the impact of the feedback provided by the robot and the virtual agent on the user. A single model of feedback will be defined but implemented differently on the robot and the agent since they have different communication capabilities. The system will be partly simulated (WOZ). If time allows, a functional version of the system will be evaluated. Tasks: • Evaluation protocol: scenario, variables … • System implementation: WOZ • Data collection: recordings • Data analysis: coding schemes, analysis of annotation, computation of evaluation metrics

  14. Thank for your attention…

More Related