Presented by archana r eddy jammula 800802639
This presentation is the property of its rightful owner.
Sponsored Links
1 / 48

Presented by: Archana R eddy Jammula 800802639 PowerPoint PPT Presentation


  • 46 Views
  • Uploaded on
  • Presentation posted in: General

Semantic Indexing of multimedia content using visual, audio and text cues Written By: . W . H. Adams . Giridharan Iyengar . Ching -Yung Lin . Milind Ramesh Naphade . Chalapathy Neti . Harriet J. Nock . John R. Smith. Presented by: Archana R eddy Jammula 800802639.

Download Presentation

Presented by: Archana R eddy Jammula 800802639

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Presented by archana r eddy jammula 800802639

Semantic Indexing of multimedia content using visual, audio and text cuesWritten By:.W. H. Adams.GiridharanIyengar. Ching-Yung Lin .MilindRamesh Naphade. ChalapathyNeti. Harriet J. Nock. John R. Smith

Presented by:

ArchanaReddy Jammula

800802639


Walking through

WALKING THROUGH……..

  • INTRODUCTION

  • SEMANTIC-CONTENT ANALYSIS SYSTEM

  • EXPERIMENTAL RESULTS

  • CONCLUSIONS


Introduction

INTRODUCTION

  • Large digital video libraries require tools for representing,

    searching, and retrieving content.

QUERY-BY-EXAMPLE (QBE)

SHIFT

QBE TO QUERY-BY-KEYWORD (QBK)


Overview

OVERVIEW

  • IBM project

  • Trainable QBK system for the labeling and retrieval of generic multimedia semantic concepts in video

  • Focus on detection of semantic-concepts using information cues from multiple modalities


Related work

RELATED WORK

  • A novel probabilistic framework for semantic video indexing by learning probabilistic multimedia representations of semantic events to represent keywords and key concepts

  • A library of examples approach

  • A rule-based system for indexing basketball videos by detecting semantics in audio.

  • A framework for detecting sources of sounds in audio using such cues as onset and offset.

  • A hidden-Markov-model (HMM) framework for generalized sound recognition.

  • Usage of tempo to characterize motion pictures.


Introduction of new approach

INTRODUCTION OF NEW APPROACH

Emphasis on the extraction of semantics from individual modalities, in some instances, using audio and visual modalities.

CHANGE: Approach that combines audio and visual content analysis with textual information retrieval for semantic modeling of multimedia

content.

Combines content analysis with information

retrieval in a unified setting for the semantic labeling

of multimedia content. Proposal of anovel

approach for representing semantic-concepts using a basis of

other semantic-concepts and propose a novel discriminant

framework to fuse the different modalities.


Approach

APPROACH

  • Visualize it as a machine learning problem.

  • Assumption:

    A priori definition of a set of atomic semantic-concepts (objects, scenes, and events) which is assumed to be broad enough to cover the semantic query space of interest high levelConcepts.

  • Steps:

    • The set of atomic concepts are annotated manually in audio, speech, and/or video within a set of “training” videos.

    • The annotated training data is then used to develop explicit statistical models of these atomic concepts; each such model can then be used to automatically label occurrences of the corresponding concept in new videos.ech, and/or video within a set of “training” videos.


Challenges

CHALLENGES

  • Low-level features appropriate for labeling atomic concepts must be identified and appropriate schemes for modeling these features are to be selected

  • High-level concepts must be linked to the presence /absence of other concepts and statistical models for combining these concept models into a high-level model must be chosen.

  • Cutting across these levels, information from multiple modalities must be integrated or fused


Semantic concept analysis system

SEMANTIC CONCEPT ANALYSIS SYSTEM:

Three components:

  • Tools for defining a lexicon of semantic-concepts and annotating examples of those concepts within a set of training videos;

  • Schemes for automatically learning the representations of semantic-concepts in the lexicon based on the labeled examples

  • Tools supporting data retrieval using the (defined) semantic-concepts.


Semantic concept analysis system1

SEMANTIC CONCEPT ANALYSIS SYSTEM:


Lexicon of semantic concepts

LEXICON OF SEMANTIC-CONCEPTS

  • Working set of intermediate- and high-level concepts, covering events, scenes, and objects.

  • Defined independently of the modality in which their cues occur: whilst some are naturally expressed in one modality over the other

  • Imposing hierarchy is difficult but if imposed then it is defined on terms of feature extraction or deriving from other concepts.


Presented by archana r eddy jammula 800802639

Semantic concept Parade is defined as:-collection of people-music-context in which this clip is interpreted as parade


Annotating a corpus

ANNOTATING A CORPUS

  • Annotation of visual data is performed at shot level;

  • Annotation of audio data is performed by specifying time spans over which each audio concept (such as speech) occurs

  • Multimodal annotation follows with synchronized playback of audio and video during the annotation process.

  • Media Streams present a lexicon of semantic concepts in terms of set of icons.


Learning semantic concepts from features

LEARNING SEMANTIC-CONCEPTS FROM FEATURES

  • Atomic concepts are modeled using features from a single modality and the integration of cues from multiple modalities occurs only within models of high-level concepts (a late integration approach);

  • Focus is on the joint analysis of audio, visual, and textual modalities for the semantic modeling of video.

  • Modeling approaches:

    • Probabilistic modeling of semantic-concepts and events using models such as GMMs, HMMs

    • Bayesian networks and discriminant approaches such as support vector machines (SVMs).


Probabilisticmodeling for semantic classification

PROBABILISTICMODELING FOR SEMANTICCLASSIFICATION

  • LOGIC: Model a semantic-concept as a class conditional probability density function over a feature space.

  • In the given set of semantic-concepts and a feature observation, choose the label as that class conditional density which yields the maximum likelihood of the observed feature.

  • As true class conditional densities are not available, assumptions are made and choices made generally are:

    • GMMs for independent observation vectors

    • HMMs for time series data.


Gmm probability density function

GMM - PROBABILITY DENSITY FUNCTION

  • GMM defines a probability density function of an n-dimensional observation vector x given a model M,


Hmm probability density function

HMM - PROBABILITY DENSITY FUNCTION

  • An HMM allows us to model a sequence of observations (x1, x2, . . . , xn) as having been generated by an unobserved state sequence s1, . . . , snwith a unique starting state s0, giving the probability of the model M generating the output sequence as


Discriminant techniques support vector machines

DISCRIMINANT TECHNIQUES: SUPPORT VECTOR MACHINES

  • Flaw of probabilistic modeling for semantic classification technique:

    The reliable estimation of class conditional parameters in the requires large amounts of training data for each class and the forms assumed for class conditional distributions may not be the most appropriate.

  • Benefit of using discriminant techniques: support vector machines technique:

    Use of a more discriminant learning approach requires fewer parameters and assumptions may yield better results


Svm technique

SVM technique:

  • Separates two classes using a linear hyper plane. The classes are represented as:


Learning audio concepts

LEARNING AUDIO CONCEPTS

  • Scheme for modeling audio-based atomic concepts starts with annotated audio training set .

  • For a set of HMMs, one for each audio concept, during testing, we use two distinct schemes to compute the confidences of the different hypotheses.


Representing concepts using speech

REPRESENTING CONCEPTS USING SPEECH

  • Speech cues may be derived from one of two sources:

    • Manual transcriptions

    • The results of automatic speech recognition (ASR) on the speech segments of the audio.

  • Procedure for labeling a particular semantic-concept using speech information alone assumes a priori definition of a set of query terms pertinent to that concept.


Presented by archana r eddy jammula 800802639

Cont..

SCHEME FOLLOWED:

  • Scheme for obtaining such a set of query terms automatically would be to use the most frequent words occurring within shots annotated by a particular concept ,the set might also be derived using human knowledge or word net.

  • Tagging, morphologically analyzing, and applying the stop list to this set of words yield a set of query terms Q for use in retrieving the concept of interest.

  • Retrieval of shots containing the concept then proceeds by ranking documents against Q according to their score, as in standard text retrieval, which provides a ranking of shots.


Learning multimodal concepts

LEARNING MULTIMODAL CONCEPTS

  • Information cues from one or more modalities are integrated.

  • we can build richer models that exploit the interrelationships between atomic concepts, which may not be possible if we model these high-level concepts in terms of their features.


Inference using graphical models

INFERENCE USING GRAPHICAL MODELS

  • Models used: Bayesian networks of various topologies and parameterizations.

  • Advantage : Bayesian networks allows us to graphically specify

    a particular form of the joint probability density function.

  • Joint probability function encoded by the Bayesian network, is

    P(E, A,V,T) = P(E)P(A/E)P(V/E)P(T/E)

(a) Bayesian Networks


Classifying concepts using svms

CLASSIFYING CONCEPTS USING SVMS

  • Scores from all the intermediate concept classifiers are concatenated into a vector, and this is used as the feature in the SVM.

(b) Support Vector Machines


Experimental results

EXPERIMENTAL RESULTS


The corpus

THE CORPUS

  • Dataset:

  • Subset of the NIST Video TREC 2001 corpus, which comprises production videos derived from sources such as NASA and Open Video Consortium.

  • 7 videos comprising 1248 video shots. The 7 videos describe NASA activities including its space program.

  • The most pertinent audio cues are:

    Music- 84% of manually labeled audio samples

    Rocket engine explosion- 60% of manually labeled audio samples


Preprocessing and feature extraction

PREPROCESSING AND FEATURE EXTRACTION

  • Visual shot detection and feature extraction

    • Color

    • Structure

    • Shape

  • Audio feature extraction


Lexicon

LEXICON

  • Lexicon in this experiment comprise more than 50 semantic conceptsfor describing events, sites, and objects with cues

    in audio, video, and/or speech.

  • A subset is described in Visual,Audio and Multimodal Concept experiments.


Evaluation metrics

EVALUATION METRICS

  • Retrieval performance is measured using precision-recall curves

  • Figure-of-merit (FOM) of retrieval effectiveness is used to

    summarize performance, defined as average precision . over the top 100 retrieved documents.


Retrieval using models for visual features

RETRIEVAL USING MODELS FOR VISUAL FEATURES

  • Results: GMM versus SVM classification

  • Depicts the overall retrieval effectiveness for a variety

  • of intermediate (visual) semantic-concepts with SVM

  • and GMM classifiers.


Presented by archana r eddy jammula 800802639

PRECISION-RECALL CURVES


Retrieval using models for audio features

RETRIEVAL USING MODELS FOR AUDIO FEATURES

  • Results: Minimum duration modeling

Fig: Effect of duration modeling


Implicit versus explicit fusion performace graph

IMPLICIT VERSUS EXPLICIT FUSION PERFORMACE GRAPH

Fig: Implicit versus explicit fusion


Results fusion of scores from multiple audio models

RESULTS: FUSION OF SCORES FROM MULTIPLE AUDIO MODELS

FOM results: audio retrieval, different intermediate concepts.


Results fusion of scores from multiple audio models1

RESULTS: FUSION OF SCORES FROM MULTIPLE AUDIO MODELS

FOM results: audio retrieval, GMM versus HMM performance

and implicit versus explicit fusion.


Retrieval using speech

RETRIEVAL USING SPEECH

Two sets of results:

  • The retrieval of the rocket launch concept using manually produced ground truth

    transcriptions

  • Retrievalusing transcriptions produced using ASR.


Presented by archana r eddy jammula 800802639

Cont..

FOM results: speech retrieval using human knowledge based query.


Bayesian network integration

BAYESIAN NETWORK INTEGRATION

  • All random variables are assumed to be binary valued

  • The scores emitted by the individual classifiers (rocket object and rocket engine explosion) are processed to fall into the 0–1 range by using the precision-recall curve as a guide

  • Mapacceptable operating points on the precision-recall curve to the 0.5 probability


Svm integration

SVM INTEGRATION

  • For fusion using SVMs:

    Procedure:

    Take the scores from all semantic models concatenating them into a 9-dimensional feature vector.


Presented by archana r eddy jammula 800802639

Fig: Fusion of audio, text, and visual models using the SVM

fusion model for rocket launch retrieval.


Presented by archana r eddy jammula 800802639

Fig: The top 20 video shots of rocket launch/take-off retrieved using multimodal detection based on the SVM model. Nineteen of the

top 20 are rocket launch shots.


Svm integration cont

SVM INTEGRATION cont..

FOM results for unimodal retrieval and the two multimodal fusion models


Conclusion

CONCLUSION

  • Feasibility of the framework was demonstrated for the

    semantic-concept rocket launch:

    -For concept classification using information in single modalities

    -For concept classification using information from multiple modalities.

  • Experimental results show that information from multiple modalities can be successfully integrated to improve semantic labeling performance over that achieved by any single modality.


Future work

FUTURE WORK

  • Schemes must be identified for automatically

    determining the low-level features which are most appropriate for labeling atomic concepts and for determining atomic concepts which are related

    to higher-level semantic-concepts.

  • The scalability of the scheme and its extension to much larger numbers of semantic-concepts must also be investigated.


Presented by archana r eddy jammula 800802639

THANK YOU


  • Login