the automatic generation of formal annotations in a multimedia indexing and searching environment
Download
Skip this Video
Download Presentation
The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment

Loading in 2 Seconds...

play fullscreen
1 / 20

The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment - PowerPoint PPT Presentation


  • 104 Views
  • Uploaded on

The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment. Thierry Declerck, Peter Wittenburg and Hamish Cunningham DFKI GmbH, Max-Planck-Institut für Psycholinguistik and University of Sheffield

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment' - kane-hebert


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
the automatic generation of formal annotations in a multimedia indexing and searching environment

The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment

Thierry Declerck, Peter Wittenburg and Hamish Cunningham

DFKI GmbH, Max-Planck-Institut für Psycholinguistik and University of Sheffield

ACL/EACL2001 Workshop on Human Language Technology and Knowledge Management

the mumis consortium
The MUMIS Consortium
  • CTIT University of Twente, Enschede, NL NLP/IE
  • TSI University of Nijmegen, Nijmegen, NL ASR
  • DFKI Saarbrücken, D NLP/IE
  • MPI Nijmegen, NL Online SW
  • DCS University of Sheffield, UK NLP/IE
  • ESTEAM Gothenburg, SE (location Athens, GR) Translation Software
  • VDA Hilversum, NL Dissemination
objectives
Objectives
  • Technology development to automatically index (with formal annotations) lengthy multimedia recordings (off-line process)

Find and annotate relevant events, together with the involved entities and relations

  • Technology development to exploit indexed multimedia archives (on-line process)

Search for interesting scenes and play them via Internet

  • Test Domain: Soccer Games / UEFA Tournament 2000
off line task
Off-line Task
  • Automatic Speech Recognition (Radio/TV Broadcasts)

Automatically transforms the speech signals into texts (for 3 languages — Dutch, English and German)

  • Natural Language Processing (Information Extraction)

Analyse all available textual documents (newspapers, speech transcripts, tickers, formal texts ...), identify and extract interesting entities, relations and events

  • Merging all the annotations produced so far
  • Create a database with formal annotations
the generation of formal annotations

The Generation of Formal Annotations

Metadata (type of game, teams, date, final score, players etc.), as they can be used a.o. for classifying and filtering videos in the MM digital archive

Events (particular actions with time codes, involved entities and related events), as they can be extracted from the video sequences

All Formal Annotations available in XMLStandard

the event table
The Event Table

Related to domain ontology and multilingual terminology. Guiding the generation of formal annotations

off line task1

Close caption

3 Languages

multilingual IE

=> event tables

Event = goal

Type = Freekick

Player = Basler

Dist. = 25 m

Time = 17

Score: leading

Event = goal

Player= Basler

Team = Germany

Time = 18

Score = 1:0

Finalscore = 1:0

Merging of

Annotations

Event = goal

Player = Basler

Dist. = 25 m

Time = 18

Score = 1:0

Newspaper

Text

Newspaper

Text

Newspaper

Text

Newspaper

Text

Newspaper

Text

Newspaper

Text

Newspaper

Texts

3 Languages

Radio Commenting

3 Languages

  • Freekick
  • Goal
  • Pass
  • Defense

Radio Commenting

3 Languages

Radio Commenting

3 Languages

Audio Commenting (TV, Radio)

3 Languages

  • 17 min
  • 18 min
  • 24 min
  • 28min
  • 1:0

Events indexed in video recording

  • Foul
  • Freekick
  • Dribbling
  • Neville
  • Basler
  • Matthäus
  • Campbell
  • Basler
  • Scholl
  • 25 m
  • 25 m
  • 60 m
Off-line Task

Event = goal

Type = Freekick

Player = Basler

Team = Germany

Time = 18

Score = 1:0

Final score = 1:0

Distance = 25 m

the role of ie in mumis
The Role of IE in MUMIS
  • Information Extraction (IE) is the task of identifying, collecting and normalizing relevant information for a specific application or user.
  • The relevant information is typically represented in form of predefined “templates”, which are filled by means of Natural Language (NL) analysis (Template = Event Table in MUMIS)
  • IE combines pattern matching mechanisms, (shallow) NLP and domain knowledge (terminology and ontology).
extension of our ie system in mumis
Extension of our IE system in MUMIS
  • Multilingual and multisource IE. Incremental information building
  • Cross-document co-reference resolution
  • Combine Metadata and event extraction => better organisation and dynamic updating of information (KM)
  • Multiple presentation of results: Template, Event table and Hyperlinks (Named Entities, rel. to KM)
example of processing formal texts
Example of Processing Formal Texts
  • Formal Text
  • The Formal Text annotated with domain-specific information
example of processing semi formal texts
Example of Processing Semi-Formal Texts
  • Semi-Formal Text
  • The Semi-Formal Text annotated with domain-specific information
merging component
Merging Component
  • Acting on the generated formal annotations (Metadata and Events), but also interleaving with the generation process of those
  • Checking consistency, eliminating redundancy (Template Merging), in accordance with domain ontology
  • Completing the information with domain knowledge, inference Machine
on line tasks
On-line Tasks
  • Searching and Displaying
  • Search for interesting events with formal queries

Give me all goals from Overmars shot with his head in 1. Half.

Event=Goal; Player=Overmars; Time<=45; Previous-Event=Headball

  • Indicate hits by thumbnails & let user select scene
  • Play scene via the Internet & allow scrolling etc

Of course: slow motion, fast play, start/stop, etc

  • User Guidance (Lexica and Ontology)
on line tasks1
On-line Tasks
  • Freekick
  • Goal
  • Pass
  • Defense

Knowledge Guided

User Interface

&

Search Engine

  • 17 min
  • 18 min
  • 24 min
  • 28min
  • 1:0

Play

Movie Fragment

of that Game

  • Foul
  • Freekick
  • Dribbling
  • Neville
  • Basler
  • Matthäus
  • Campbell
  • Basler
  • Scholl
  • 25 m
  • 25 m
  • 60 m

München - Ajax

1998

München - Porto

1996

Deutschland - Brasilien

1998

Prototype Demo

more about mpeg moving picture coding experts group
More about MPEG (Moving Picture Coding Experts Group)
  • MPEG-1: low-level media encoding and compression format (VHS quality, ~ 2-3 Mbps - good for streaming)
  • MPEG-2: improved media encoding and compression format (S-VHS quality, ~ 5-10 Mbps, digital TV and DVD standard
  • MPEG-4: Codes content as objects and enables those objects to be manipulated at the client, optimized compression
on line sw architecture

Ontology

Client

Objects

Lexica

Client

Applet

JMF

Media Server

Objects

Query Engine

Objects

HTTP

RMI

RMI

(RTP,

RTSP)

WWW Server

Java Server

MPEG Movies

Keyframes

Annotations

Metadata

File

Server

Media

Server

MPEG1

DB

Server

rDBMS

Media

Server

MPEG1

Media

Server

MPEG1

Media

Server

MPEG1

JDBC

On-line SW Architecture
  • Client-Server structure:
    • fully distributed
    • JMF media presentation
    • RMI-based interaction
  • Query interface:
    • automatic pre-selection
    • guided by domain knowledge
    • interactive, visual feedback
on line hw architecture

Media Server

RAID

1Gbps

Gb-Switch

Router

FC Switch

GB Switch

Tape

Library

Internet

Media Server

On-line HW Architecture
  • efficient & reliable storage management
  • (near-line capacity, media change, 2. Location)
  • high storage capacity (n TB, 1 h MPEG1 = 1 GB)
  • powerful media servers / powerful network
ui annotation
UI / Annotation
  • UI optimization
    • thumbnails not that informative
    • which thumbnail? (several around time mark)
    • automatic thumbnail adjustment?
    • seamless operation for user
    • lexicon/ontology guidance
    • user driven input
  • Manual annotation tools
    • MediaTagger
    • EUDICO
gain user group
Gain - User Group
  • What gets lost? Is it necessary?
  • Potential: direct Internet Service, less dependencies
acknowledgements
Acknowledgements
  • UEFA
  • DFB, FA, KNVB
  • EBU, WDR, NOS, SWR

Allez les Bleus!!

ad