A multimodal dialogue-driven interface for accessing the content of recorded meetings
1 / 10

A multimodal dialogue-driven interface for accessing the content of recorded meetings - PowerPoint PPT Presentation

  • Uploaded on

A multimodal dialogue-driven interface for accessing the content of recorded meetings. Agnes Lisowska ISSCO/TIM/ETI University of Geneva IM2.MDM

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'A multimodal dialogue-driven interface for accessing the content of recorded meetings' - darrion-carry

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
A multimodal dialogue driven interface for accessing the content of recorded meetings

A multimodal dialogue-driven interface for accessing the content of recorded meetings

Agnes Lisowska

ISSCO/TIM/ETI University of Geneva


Work done in collaboration with: Martin Rajman, Mirek Melichar, Trung H. Bui (LIA & CGC at the EPFL) and Marcin Bogobowicz (graphic artist)

The problem

The data content of recorded meetings

Audio and video


printed documents


whiteboard activity

handwritten notes

Annotated transcriptions



argumentative structure

The users


tracking project progress

tracking employee performance


catch up on a missed meeting

check facts

learn about a project

The Problem

What type of interface?

User centered design ucd
User-Centered Design (UCD) content of recorded meetings

“The bottom line is that the tool should be designed around the needs of the user to allow for effective and efficient use of the application.”(Scanlon and Percival, IBM)

Who will the users be?

What tasks will users be doing?

Which modalities? Which combinations?

Which functionalities?

Type of dialogue strategy?

Degree of user control?

Data annotation

Types of data

Data organization

User requirements
User Requirements content of recorded meetings

  • Requirements gathering must be an iterative process

    • requirements can emerge or change over time and with experience

  • User requirements gathering within IM2.MDM

    • Questionnaire-based study (Lisowska 2003)

      • collected ~300 queries to the system

      • analyzed linguistically and thematically

      • results used to derive user requirements and drive preliminary design of Archivus interface

Hci and design principles in archivus
HCI and Design Principles in Archivus content of recorded meetings

  • Interaction metaphor

    • helps a user understand the various functionalities of a system

  • Multiple levels of interaction (user experience differs)

  • Visibility

    • important and frequently used elements always visible

  • Consistency

    • same elements in same location

    • same functionality performed in same way

  • Feedback to the user

    • user should see the results of their actions

Novice user

Average user

Expert user

System driven (dialogue system)

Mixed initiative (dialogue system + user control)

User driven

The archivus system

Designed based on user requirements content of recorded meetings

Interaction metaphor: using a library or archive

library is the database of meetings

each book in the library is a single meeting

Preliminary version allows for

many different modalities

full range of interaction styles



natural language

speech via speech recognition

text via keyboard or pen





The Archivus System

  • output

    • audio

      • system prompts

      • meeting recordings

    • video

      • meeting recordings

    • text

    • graphics

The archivus system1
The Archivus System content of recorded meetings

System control buttons

Constant view of database (search space)

System prompts


Natural language input feedback

Meeting ‘book’

Access to multimedia

Interactive search definition buttons

Interaction history

A multimodal dialogue driven interface for accessing the content of recorded meetings
Demo content of recorded meetings

Problem: Denis told you that there was a discussion about red sofas for the ISSCO lounge and you want to know more about it.

  • All you know is:

    • Denis was at the meeting

    • the meeting was about ISSCO lounge furniture

    • the meeting was in early April in Martigny

Testing and evaluation
Testing and Evaluation content of recorded meetings

  • Wizard of Oz methodology

    • allows testing without full implementation

    • needs to be iterative

    • reveals

      • how modalities are used

      • new user requirements

      • flaws in the design

References associated papers
References & Associated Papers content of recorded meetings

User-Centered Design

  • Scanlon, J & Percival, L. UCD for different project types (Parts 1 & 2). IBM developerWorks website.


    User Requirements Studies

  • Lisowska A., Popescu-Belis A. & Armstrong S. (2004) - User Query Analysis for the Specification and Evaluation of a Dialogue Processing and Retrieval System. LREC 2004 (Fourth International Conference on Language Resources and Evaluation), Lisbon, Portugal, vol.III, p.993-996.

  • Lisowska, A. (2003) - Multimodal Interface Design for the Multimodal

    Meeting Domain: Preliminary Indications from a Query Analysis Study. Project Report IM2.MDM-11, November 2003, 30 pages.

    The Archivus Interface

  • Lisowska A., Rajman M., & Bui T.H. (2004) - ARCHIVUS : A System for Accesssing the Content of Recorded Multimodal Meetings. Proceedings of the Joint AMI/PASCAL/IM2/M4 Workshop on Multimodal Interaction and

    Related Machine Learning Algorithms, Martigny, Switzerland, 8 p.

    Dialogue Management

  • T. H. Bui, M. Rajman and M. Melichar. Rapid Dialogue Prototyping Methodology. Proceedings of the 7th International Conference on Text, Speech Dialogue (TSD 2004), P. Sojka, I. Kopecek K. Pala (Eds.), Lecture Notes in Computer Science, Springer-Verlag, Berlin Heidelberg New York, Brno, Czech Republic, September 8-11, 2004

  • T. H. Bui and M. Rajman. Rapid Dialogue Prototyping Methodology. Technical Report No. 200401, Swiss Federal Institute of Technology (EPFL), Lausanne (Switzerland), January, 2004.