The automatic generation of formal annotations in a multimedia indexing and searching environment
Download
1 / 20

The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment - PowerPoint PPT Presentation


  • 100 Views
  • Uploaded on

The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment. Thierry Declerck, Peter Wittenburg and Hamish Cunningham DFKI GmbH, Max-Planck-Institut für Psycholinguistik and University of Sheffield

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment' - kane-hebert


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
The automatic generation of formal annotations in a multimedia indexing and searching environment

The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment

Thierry Declerck, Peter Wittenburg and Hamish Cunningham

DFKI GmbH, Max-Planck-Institut für Psycholinguistik and University of Sheffield

ACL/EACL2001 Workshop on Human Language Technology and Knowledge Management


The mumis consortium
The MUMIS Consortium MultiMedia Indexing and Searching Environment

  • CTIT University of Twente, Enschede, NL NLP/IE

  • TSI University of Nijmegen, Nijmegen, NL ASR

  • DFKI Saarbrücken, D NLP/IE

  • MPI Nijmegen, NL Online SW

  • DCS University of Sheffield, UK NLP/IE

  • ESTEAM Gothenburg, SE (location Athens, GR) Translation Software

  • VDA Hilversum, NL Dissemination


Objectives
Objectives MultiMedia Indexing and Searching Environment

  • Technology development to automatically index (with formal annotations) lengthy multimedia recordings (off-line process)

    Find and annotate relevant events, together with the involved entities and relations

  • Technology development to exploit indexed multimedia archives (on-line process)

    Search for interesting scenes and play them via Internet

  • Test Domain: Soccer Games / UEFA Tournament 2000


Off line task
Off-line Task MultiMedia Indexing and Searching Environment

  • Automatic Speech Recognition (Radio/TV Broadcasts)

    Automatically transforms the speech signals into texts (for 3 languages — Dutch, English and German)

  • Natural Language Processing (Information Extraction)

    Analyse all available textual documents (newspapers, speech transcripts, tickers, formal texts ...), identify and extract interesting entities, relations and events

  • Merging all the annotations produced so far

  • Create a database with formal annotations


The generation of formal annotations

The Generation of Formal Annotations MultiMedia Indexing and Searching Environment

Metadata (type of game, teams, date, final score, players etc.), as they can be used a.o. for classifying and filtering videos in the MM digital archive

Events (particular actions with time codes, involved entities and related events), as they can be extracted from the video sequences

All Formal Annotations available in XMLStandard


The event table
The Event Table MultiMedia Indexing and Searching Environment

Related to domain ontology and multilingual terminology. Guiding the generation of formal annotations


Off line task1

Close caption MultiMedia Indexing and Searching Environment

3 Languages

multilingual IE

=> event tables

Event = goal

Type = Freekick

Player = Basler

Dist. = 25 m

Time = 17

Score: leading

Event = goal

Player= Basler

Team = Germany

Time = 18

Score = 1:0

Finalscore = 1:0

Merging of

Annotations

Event = goal

Player = Basler

Dist. = 25 m

Time = 18

Score = 1:0

Newspaper

Text

Newspaper

Text

Newspaper

Text

Newspaper

Text

Newspaper

Text

Newspaper

Text

Newspaper

Texts

3 Languages

Radio Commenting

3 Languages

  • Freekick

  • Goal

  • Pass

  • Defense

Radio Commenting

3 Languages

Radio Commenting

3 Languages

Audio Commenting (TV, Radio)

3 Languages

  • 17 min

  • 18 min

  • 24 min

  • 28min

  • 1:0

Events indexed in video recording

  • Foul

  • Freekick

  • Dribbling

  • Neville

  • Basler

  • Matthäus

  • Campbell

  • Basler

  • Scholl

  • 25 m

  • 25 m

  • 60 m

Off-line Task

Event = goal

Type = Freekick

Player = Basler

Team = Germany

Time = 18

Score = 1:0

Final score = 1:0

Distance = 25 m


The role of ie in mumis
The Role of IE in MUMIS MultiMedia Indexing and Searching Environment

  • Information Extraction (IE) is the task of identifying, collecting and normalizing relevant information for a specific application or user.

  • The relevant information is typically represented in form of predefined “templates”, which are filled by means of Natural Language (NL) analysis (Template = Event Table in MUMIS)

  • IE combines pattern matching mechanisms, (shallow) NLP and domain knowledge (terminology and ontology).


Extension of our ie system in mumis
Extension of our IE system in MUMIS MultiMedia Indexing and Searching Environment

  • Multilingual and multisource IE. Incremental information building

  • Cross-document co-reference resolution

  • Combine Metadata and event extraction => better organisation and dynamic updating of information (KM)

  • Multiple presentation of results: Template, Event table and Hyperlinks (Named Entities, rel. to KM)


Example of processing formal texts
Example of Processing Formal Texts MultiMedia Indexing and Searching Environment

  • Formal Text

  • The Formal Text annotated with domain-specific information


Example of processing semi formal texts
Example of Processing Semi-Formal Texts MultiMedia Indexing and Searching Environment

  • Semi-Formal Text

  • The Semi-Formal Text annotated with domain-specific information


Merging component
Merging Component MultiMedia Indexing and Searching Environment

  • Acting on the generated formal annotations (Metadata and Events), but also interleaving with the generation process of those

  • Checking consistency, eliminating redundancy (Template Merging), in accordance with domain ontology

  • Completing the information with domain knowledge, inference Machine


On line tasks
On-line Tasks MultiMedia Indexing and Searching Environment

  • Searching and Displaying

  • Search for interesting events with formal queries

    Give me all goals from Overmars shot with his head in 1. Half.

    Event=Goal; Player=Overmars; Time<=45; Previous-Event=Headball

  • Indicate hits by thumbnails & let user select scene

  • Play scene via the Internet & allow scrolling etc

    Of course: slow motion, fast play, start/stop, etc

  • User Guidance (Lexica and Ontology)


On line tasks1
On-line Tasks MultiMedia Indexing and Searching Environment

  • Freekick

  • Goal

  • Pass

  • Defense

Knowledge Guided

User Interface

&

Search Engine

  • 17 min

  • 18 min

  • 24 min

  • 28min

  • 1:0

Play

Movie Fragment

of that Game

  • Foul

  • Freekick

  • Dribbling

  • Neville

  • Basler

  • Matthäus

  • Campbell

  • Basler

  • Scholl

  • 25 m

  • 25 m

  • 60 m

München - Ajax

1998

München - Porto

1996

Deutschland - Brasilien

1998

Prototype Demo


More about mpeg moving picture coding experts group
More about MPEG MultiMedia Indexing and Searching Environment(Moving Picture Coding Experts Group)

  • MPEG-1: low-level media encoding and compression format (VHS quality, ~ 2-3 Mbps - good for streaming)

  • MPEG-2: improved media encoding and compression format (S-VHS quality, ~ 5-10 Mbps, digital TV and DVD standard

  • MPEG-4: Codes content as objects and enables those objects to be manipulated at the client, optimized compression


On line sw architecture

Ontology MultiMedia Indexing and Searching Environment

Client

Objects

Lexica

Client

Applet

JMF

Media Server

Objects

Query Engine

Objects

HTTP

RMI

RMI

(RTP,

RTSP)

WWW Server

Java Server

MPEG Movies

Keyframes

Annotations

Metadata

File

Server

Media

Server

MPEG1

DB

Server

rDBMS

Media

Server

MPEG1

Media

Server

MPEG1

Media

Server

MPEG1

JDBC

On-line SW Architecture

  • Client-Server structure:

    • fully distributed

    • JMF media presentation

    • RMI-based interaction

  • Query interface:

    • automatic pre-selection

    • guided by domain knowledge

    • interactive, visual feedback


On line hw architecture

Media Server MultiMedia Indexing and Searching Environment

RAID

1Gbps

Gb-Switch

Router

FC Switch

GB Switch

Tape

Library

Internet

Media Server

On-line HW Architecture

  • efficient & reliable storage management

  • (near-line capacity, media change, 2. Location)

  • high storage capacity (n TB, 1 h MPEG1 = 1 GB)

  • powerful media servers / powerful network


Ui annotation
UI / Annotation MultiMedia Indexing and Searching Environment

  • UI optimization

    • thumbnails not that informative

    • which thumbnail? (several around time mark)

    • automatic thumbnail adjustment?

    • seamless operation for user

    • lexicon/ontology guidance

    • user driven input

  • Manual annotation tools

    • MediaTagger

    • EUDICO


Gain user group
Gain - User Group MultiMedia Indexing and Searching Environment

  • What gets lost? Is it necessary?

  • Potential: direct Internet Service, less dependencies


Acknowledgements
Acknowledgements MultiMedia Indexing and Searching Environment

  • UEFA

  • DFB, FA, KNVB

  • EBU, WDR, NOS, SWR

Allez les Bleus!!


ad