The automatic generation of formal annotations in a multimedia indexing and searching environment
This presentation is the property of its rightful owner.
Sponsored Links
1 / 20

The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment PowerPoint PPT Presentation


  • 66 Views
  • Uploaded on
  • Presentation posted in: General

The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment. Thierry Declerck, Peter Wittenburg and Hamish Cunningham DFKI GmbH, Max-Planck-Institut für Psycholinguistik and University of Sheffield

Download Presentation

The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


The automatic generation of formal annotations in a multimedia indexing and searching environment

The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment

Thierry Declerck, Peter Wittenburg and Hamish Cunningham

DFKI GmbH, Max-Planck-Institut für Psycholinguistik and University of Sheffield

ACL/EACL2001 Workshop on Human Language Technology and Knowledge Management


The mumis consortium

The MUMIS Consortium

  • CTITUniversity of Twente, Enschede, NLNLP/IE

  • TSI University of Nijmegen, Nijmegen, NLASR

  • DFKISaarbrücken, DNLP/IE

  • MPI Nijmegen, NLOnline SW

  • DCSUniversity of Sheffield, UKNLP/IE

  • ESTEAMGothenburg, SE (location Athens, GR)Translation Software

  • VDAHilversum, NL Dissemination


Objectives

Objectives

  • Technology development to automatically index (with formal annotations) lengthy multimedia recordings (off-line process)

    Find and annotate relevant events, together with the involved entities and relations

  • Technology development to exploit indexed multimedia archives (on-line process)

    Search for interesting scenes and play them via Internet

  • Test Domain: Soccer Games / UEFA Tournament 2000


Off line task

Off-line Task

  • Automatic Speech Recognition (Radio/TV Broadcasts)

    Automatically transforms the speech signals into texts (for 3 languages — Dutch, English and German)

  • Natural Language Processing (Information Extraction)

    Analyse all available textual documents (newspapers, speech transcripts, tickers, formal texts ...), identify and extract interesting entities, relations and events

  • Merging all the annotations produced so far

  • Create a database with formal annotations


The generation of formal annotations

The Generation of Formal Annotations

Metadata (type of game, teams, date, final score, players etc.), as they can be used a.o. for classifying and filtering videos in the MM digital archive

Events (particular actions with time codes, involved entities and related events), as they can be extracted from the video sequences

All Formal Annotations available in XMLStandard


The event table

The Event Table

Related to domain ontology and multilingual terminology. Guiding the generation of formal annotations


Off line task1

Close caption

3 Languages

multilingual IE

=> event tables

Event = goal

Type = Freekick

Player = Basler

Dist. = 25 m

Time = 17

Score: leading

Event = goal

Player= Basler

Team = Germany

Time = 18

Score = 1:0

Finalscore = 1:0

Merging of

Annotations

Event = goal

Player = Basler

Dist. = 25 m

Time = 18

Score = 1:0

Newspaper

Text

Newspaper

Text

Newspaper

Text

Newspaper

Text

Newspaper

Text

Newspaper

Text

Newspaper

Texts

3 Languages

Radio Commenting

3 Languages

  • Freekick

  • Goal

  • Pass

  • Defense

Radio Commenting

3 Languages

Radio Commenting

3 Languages

Audio Commenting (TV, Radio)

3 Languages

  • 17 min

  • 18 min

  • 24 min

  • 28min

  • 1:0

Events indexed in video recording

  • Foul

  • Freekick

  • Dribbling

  • Neville

  • Basler

  • Matthäus

  • Campbell

  • Basler

  • Scholl

  • 25 m

  • 25 m

  • 60 m

Off-line Task

Event = goal

Type = Freekick

Player = Basler

Team = Germany

Time = 18

Score = 1:0

Final score = 1:0

Distance = 25 m


The role of ie in mumis

The Role of IE in MUMIS

  • Information Extraction (IE) is the task of identifying, collecting and normalizing relevant information for a specific application or user.

  • The relevant information is typically represented in form of predefined “templates”, which are filled by means of Natural Language (NL) analysis (Template = Event Table in MUMIS)

  • IE combines pattern matching mechanisms, (shallow) NLP and domain knowledge (terminology and ontology).


Extension of our ie system in mumis

Extension of our IE system in MUMIS

  • Multilingual and multisource IE. Incremental information building

  • Cross-document co-reference resolution

  • Combine Metadata and event extraction => better organisation and dynamic updating of information (KM)

  • Multiple presentation of results: Template, Event table and Hyperlinks (Named Entities, rel. to KM)


Example of processing formal texts

Example of Processing Formal Texts

  • Formal Text

  • The Formal Text annotated with domain-specific information


Example of processing semi formal texts

Example of Processing Semi-Formal Texts

  • Semi-Formal Text

  • The Semi-Formal Text annotated with domain-specific information


Merging component

Merging Component

  • Acting on the generated formal annotations (Metadata and Events), but also interleaving with the generation process of those

  • Checking consistency, eliminating redundancy (Template Merging), in accordance with domain ontology

  • Completing the information with domain knowledge, inference Machine


On line tasks

On-line Tasks

  • Searching and Displaying

  • Search for interesting events with formal queries

    Give me all goals from Overmars shot with his head in 1. Half.

    Event=Goal; Player=Overmars; Time<=45; Previous-Event=Headball

  • Indicate hits by thumbnails & let user select scene

  • Play scene via the Internet & allow scrolling etc

    Of course: slow motion, fast play, start/stop, etc

  • User Guidance (Lexica and Ontology)


On line tasks1

On-line Tasks

  • Freekick

  • Goal

  • Pass

  • Defense

Knowledge Guided

User Interface

&

Search Engine

  • 17 min

  • 18 min

  • 24 min

  • 28min

  • 1:0

Play

Movie Fragment

of that Game

  • Foul

  • Freekick

  • Dribbling

  • Neville

  • Basler

  • Matthäus

  • Campbell

  • Basler

  • Scholl

  • 25 m

  • 25 m

  • 60 m

München - Ajax

1998

München - Porto

1996

Deutschland - Brasilien

1998

Prototype Demo


More about mpeg moving picture coding experts group

More about MPEG (Moving Picture Coding Experts Group)

  • MPEG-1: low-level media encoding and compression format (VHS quality, ~ 2-3 Mbps - good for streaming)

  • MPEG-2: improved media encoding and compression format (S-VHS quality, ~ 5-10 Mbps, digital TV and DVD standard

  • MPEG-4: Codes content as objects and enables those objects to be manipulated at the client, optimized compression


On line sw architecture

Ontology

Client

Objects

Lexica

Client

Applet

JMF

Media Server

Objects

Query Engine

Objects

HTTP

RMI

RMI

(RTP,

RTSP)

WWW Server

Java Server

MPEG Movies

Keyframes

Annotations

Metadata

File

Server

Media

Server

MPEG1

DB

Server

rDBMS

Media

Server

MPEG1

Media

Server

MPEG1

Media

Server

MPEG1

JDBC

On-line SW Architecture

  • Client-Server structure:

    • fully distributed

    • JMF media presentation

    • RMI-based interaction

  • Query interface:

    • automatic pre-selection

    • guided by domain knowledge

    • interactive, visual feedback


On line hw architecture

Media Server

RAID

1Gbps

Gb-Switch

Router

FC Switch

GB Switch

Tape

Library

Internet

Media Server

On-line HW Architecture

  • efficient & reliable storage management

  • (near-line capacity, media change, 2. Location)

  • high storage capacity (n TB, 1 h MPEG1 = 1 GB)

  • powerful media servers / powerful network


Ui annotation

UI / Annotation

  • UI optimization

    • thumbnails not that informative

    • which thumbnail? (several around time mark)

    • automatic thumbnail adjustment?

    • seamless operation for user

    • lexicon/ontology guidance

    • user driven input

  • Manual annotation tools

    • MediaTagger

    • EUDICO


Gain user group

Gain - User Group

  • What gets lost? Is it necessary?

  • Potential: direct Internet Service, less dependencies


Acknowledgements

Acknowledgements

  • UEFA

  • DFB, FA, KNVB

  • EBU, WDR, NOS, SWR

Allez les Bleus!!


  • Login