Skip this Video
Download Presentation
A multimodal dialogue-driven interface for accessing the content of recorded meetings

Loading in 2 Seconds...

play fullscreen
1 / 10

A multimodal dialogue-driven interface for accessing the content of recorded meetings - PowerPoint PPT Presentation

  • Uploaded on

A multimodal dialogue-driven interface for accessing the content of recorded meetings. Agnes Lisowska ISSCO/TIM/ETI University of Geneva IM2.MDM

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' A multimodal dialogue-driven interface for accessing the content of recorded meetings' - darrion-carry

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

A multimodal dialogue-driven interface for accessing the content of recorded meetings

Agnes Lisowska

ISSCO/TIM/ETI University of Geneva


Work done in collaboration with: Martin Rajman, Mirek Melichar, Trung H. Bui (LIA & CGC at the EPFL) and Marcin Bogobowicz (graphic artist)

the problem
The data

Audio and video


printed documents


whiteboard activity

handwritten notes

Annotated transcriptions



argumentative structure

The users


tracking project progress

tracking employee performance


catch up on a missed meeting

check facts

learn about a project

The Problem

What type of interface?

user centered design ucd
User-Centered Design (UCD)

“The bottom line is that the tool should be designed around the needs of the user to allow for effective and efficient use of the application.”(Scanlon and Percival, IBM)

Who will the users be?

What tasks will users be doing?

Which modalities? Which combinations?

Which functionalities?

Type of dialogue strategy?

Degree of user control?

Data annotation

Types of data

Data organization

user requirements
User Requirements
  • Requirements gathering must be an iterative process
    • requirements can emerge or change over time and with experience
  • User requirements gathering within IM2.MDM
    • Questionnaire-based study (Lisowska 2003)
      • collected ~300 queries to the system
      • analyzed linguistically and thematically
      • results used to derive user requirements and drive preliminary design of Archivus interface
hci and design principles in archivus
HCI and Design Principles in Archivus
  • Interaction metaphor
    • helps a user understand the various functionalities of a system
  • Multiple levels of interaction (user experience differs)
  • Visibility
    • important and frequently used elements always visible
  • Consistency
    • same elements in same location
    • same functionality performed in same way
  • Feedback to the user
    • user should see the results of their actions

Novice user

Average user

Expert user

System driven (dialogue system)

Mixed initiative (dialogue system + user control)

User driven

the archivus system
Designed based on user requirements

Interaction metaphor: using a library or archive

library is the database of meetings

each book in the library is a single meeting

Preliminary version allows for

many different modalities

full range of interaction styles



natural language

speech via speech recognition

text via keyboard or pen





The Archivus System
  • output
    • audio
      • system prompts
      • meeting recordings
    • video
      • meeting recordings
    • text
    • graphics
the archivus system1
The Archivus System

System control buttons

Constant view of database (search space)

System prompts


Natural language input feedback

Meeting ‘book’

Access to multimedia

Interactive search definition buttons

Interaction history


Problem: Denis told you that there was a discussion about red sofas for the ISSCO lounge and you want to know more about it.

  • All you know is:
    • Denis was at the meeting
    • the meeting was about ISSCO lounge furniture
    • the meeting was in early April in Martigny
testing and evaluation
Testing and Evaluation
  • Wizard of Oz methodology
    • allows testing without full implementation
    • needs to be iterative
    • reveals
      • how modalities are used
      • new user requirements
      • flaws in the design
references associated papers
References & Associated Papers

User-Centered Design

  • Scanlon, J & Percival, L. UCD for different project types (Parts 1 & 2). IBM developerWorks website.

User Requirements Studies

  • Lisowska A., Popescu-Belis A. & Armstrong S. (2004) - User Query Analysis for the Specification and Evaluation of a Dialogue Processing and Retrieval System. LREC 2004 (Fourth International Conference on Language Resources and Evaluation), Lisbon, Portugal, vol.III, p.993-996.
  • Lisowska, A. (2003) - Multimodal Interface Design for the Multimodal

Meeting Domain: Preliminary Indications from a Query Analysis Study. Project Report IM2.MDM-11, November 2003, 30 pages.

The Archivus Interface

  • Lisowska A., Rajman M., & Bui T.H. (2004) - ARCHIVUS : A System for Accesssing the Content of Recorded Multimodal Meetings. Proceedings of the Joint AMI/PASCAL/IM2/M4 Workshop on Multimodal Interaction and

Related Machine Learning Algorithms, Martigny, Switzerland, 8 p.

Dialogue Management

  • T. H. Bui, M. Rajman and M. Melichar. Rapid Dialogue Prototyping Methodology. Proceedings of the 7th International Conference on Text, Speech Dialogue (TSD 2004), P. Sojka, I. Kopecek K. Pala (Eds.), Lecture Notes in Computer Science, Springer-Verlag, Berlin Heidelberg New York, Brno, Czech Republic, September 8-11, 2004
  • T. H. Bui and M. Rajman. Rapid Dialogue Prototyping Methodology. Technical Report No. 200401, Swiss Federal Institute of Technology (EPFL), Lausanne (Switzerland), January, 2004.