Multimedia databases
Download
1 / 49

Multimedia Databases - PowerPoint PPT Presentation


  • 192 Views
  • Updated On :

Multimedia Databases. Prepared by Pradeep Konduri Instructor: Dr. Yingshu Li Georgia State University. Plan of Attack. Introduction Architecture Image Content Analysis Modeling Constructs Logical Implementation Real-World Applications Conclusion. Types of multimedia data.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Multimedia Databases' - ciel


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Multimedia databases l.jpg

Multimedia Databases

Prepared by Pradeep Konduri

Instructor: Dr. Yingshu LiGeorgia State University


Plan of attack l.jpg
Plan of Attack

  • Introduction

  • Architecture

  • Image Content Analysis

  • Modeling Constructs

  • Logical Implementation

  • Real-World Applications

  • Conclusion


Types of multimedia data l.jpg
Types of multimedia data

  • Text: using a standard language (SGML, HTML)

  • Graphics: encoded in CGM, postscript

  • Images: bitmap, JPEG, MPEG

  • Video: sequenced image data at specified rates

  • Audio: aural recordings in a string of bits in digitized form


Nature of multimedia applications l.jpg
Nature of Multimedia Applications

  • Repositories: central location for data maintained by DBMS, organized in storage levels

  • Presentations: delivery of audio and video data, temporarily stored.

  • Collaborative: complex design, analyzing data


Management issues l.jpg
Management Issues

  • Modeling: complex objects, wide range of types

  • Design: still in research

  • Storage: representation, compression, buffering during I/O, mapping

  • Queries: techniques need to be modified

  • Performance: physical limitations, parallel processing


Research problems l.jpg
Research Problems

  • Information Retrieval in Queries: Modeling the content of documents

  • Multimedia/Hypermedia Data Modeling and Retrieval: Hyperlinks, Used in WWW

  • Text Retrieval: Use of a thesaurus


Multimedia database applications l.jpg
Multimedia Database Applications

  • Documentation and keeping Records

  • Knowledge distribution

  • Education and Training

  • Marketing, Advertisement, Entertainment, Travel

  • Real-time Control, Monitoring


A generic architecture of mmdbms l.jpg
A Generic Architecture of MMDBMS

  • Media organization: organize the features for retrieval(i.e., indexing the features with effective structures)

  • Media query processing: accommodated with indexing structure, efficient search algorithm with similarity function should be designed


Multimedia database architecture l.jpg

<article>

.....

</article>

Multimedia Database Architecture

MM Data

Pre-

processor

<!ELEMENT ..>

.....

<!ATTLIST...>

Meta-Data

Recognized

components

Additional

Information

Query Interface

MM Data

MM Data

Instance

MM Data

Instance

Users

Multimedia DBMS

Multimedia Data

Preprocessing System

Database Processing


Document database architecture l.jpg

<article>

.....

</article>

Document Database Architecture

<!ELEMENT ..>

.....

<!ATTLIST...>

DTD

Manager

DTD files

DTD Parser

DTD

Type

Generator

Query Interface

<!ELEMENT ..>

.....

<!ATTLIST...>

Document

content

SGML/XML

Parser

DTD

XML or SGML

Document

Instance

SGML/XML

Documents

Parse

Tree

C++ Types

Users

Multimedia DBMS

Instance

Generator

C++ Objects

Document

Processing System

Database Processing


Image database architecture l.jpg

Image Analysis

and Pattern Recognition

<article>

.....

</article>

Image Database Architecture

Semantic

Objects

Syntactic

Objects

Image Content

Description

Meta-Data

Query Interface

Image

Annotation

Image

Users

Image

Multimedia DBMS

Image

Processing System

Database Processing


Image content analysis l.jpg
Image Content Analysis

  • Image content analysis can be categorized in 2 groups:

    • Low-level features: vectors in a multi-dimensional space

      • Color

      • Texture

      • Shape

    • Mid- to high-level features: Try to infer semantics

    • Semantic Gap


Image content analysis color l.jpg
Image Content Analysis: Color

  • Color space:

    • Multidimensional space

    • A dimension is a color component

    • Examples of color space: RGB, HSV

    • RGB space: A color is a linear combination of 3 primary colors (Red, Green and Blue)

  • Color Quantization

    • Used to reduce the color resolution of an image

  • Three widely used color features

    • Global color histogram

    • Local color histogram

    • Dominant color


Color histograms l.jpg
Color Histograms

  • Color histograms indicate color distribution without spatial information

    • Color histogram distance metrics


Image content analysis texture l.jpg
Image Content Analysis: Texture

  • Refers to visual patterns with properties of homogeneity that do not result from the presence of only a single color

  • Examples of texture: Tree barks, clouds, water, bricks and fabrics

  • Texture features: Contrast, uniformity, coarseness, roughness, frequency, density and directionality

  • Two types of texture descriptors

    • Statistical model-based

      • Explores the gray level spatial dependence of texture and extracts meaningful statistics as texture representation

    • Transform-based

      • DCT transform, Fourier-Mellin transform, Polar Fourier transform, Gabor and wavelet transform


Image content analysis shape l.jpg
Image Content Analysis: Shape

  • Object segmentation

    • Approaches:

      • Global threshold-based approach

      • Region growing,

      • Split and merge approach,

      • Edge detection app

    • Still a difficult problem in computer vision. Generally speaking it is difficult to achieve perfect segmentation


Salient objects vs salient points l.jpg

Segmented images with region boundaries

Extracted

salient points

Original images

Salient Objects vs. Salient Points

Generic low-level description of images into salient objects and salient points


Modeling images principles l.jpg
Modeling Images – Principles

  • Support for multiple representations of an image

  • Support for user-defined categorization of images

  • Well-defined set of operations on images

  • An image can have (semantic, functional, spatial) relationships with other images (or documents) which should be represented in the DBMS

  • An image is composed of salient objects (meaningful image components)


Salient object modeling l.jpg
Salient Object Modeling

  • Multiple representations of a salient object (grid, vector) are allowed

  • A salient object O is of a particular type which belongs to a user defined salient object types hierarchy

  • An image component may have some (semantic, functional, spatial) relationships with other salient objects


Semantic gap l.jpg
Semantic Gap”

semantics-intensive multimedia systems & applications

non-semantic

multimedia data models

Semantic Gap

require

model

raw data,primitive properties (size, format, etc)

semantic meaning of the data


Semantic modeling of multimedia why hard l.jpg
Semantic modeling of multimedia -- Why hard?

  • Context-dependency

    • Semantics is not a static and intrinsic property

    • The semantics of an object often depends on:

      • the application/user who manipulate the object

      • the role that the object plays

      • other objects in the same “context”

Example:

Van Gogh’s paintings

flower


Why hard cont l.jpg
Why hard? (cont.)

  • Modality-independency

    • Media objects of different modalities may suggest the similar/related semantic meanings.

    • Example:

Query:

Results:

Harry Potter has never been the star of a Quidditch team, scoring points while riding a broom far above the ground. He knows no spells, has never helped to hatch a dragon, and has never worn a cloak of invisibility.

image

video

text


Mediaview a semantic bridge l.jpg
MediaView– A “Semantic Bridge”

  • An object-oriented view mechanism that bridges the semantic gap between multimedia systems and databases

  • Core concept –media view (MV)

    • a customized context for semantic interpretation of media objects (text docs, images, video, etc)

    • collectively constitute the conceptual infrastructure of a multimedia system & application


Architecture l.jpg
Architecture

MediaView

Mechanism


Basic concepts l.jpg
Basic Concepts

An example…


Basic concepts26 l.jpg
Basic Concepts

Semantics-based data reorganization via media views


View operators l.jpg
View Operators

  • A set of operators that take media views and view instances as operands.

  • Focus on the operators that are indispensable in supporting queries and navigation over multimedia objects.


View operators28 l.jpg
View Operators

type-level

V-overlap

syntax<boolean>:= v-overlap (<media view1, mediaview2 >)

semanticstrue, if and only if ( o  O)(oextent(<media view1>) andoextent(<media view2>))

Cross

syntax{<object>}:= cross (<media view1, media view2 >)

semantics{<object>} := {o  O | o extent(<media view1>) andoextent(<media view2>)}

Sum

syntax{<object>}:= sum (<media view1, meida-view2 >)

semantics{<object>} := {o  O | o  extent(<media view1>) oroextent(<media view2>)}

Subtract

syntax{<object>}:= subtract (<media view1, media view2>)

semantics{<object>}:= {o  O | o extent(<media view1>) andoextent(<media view2>)}


View operators29 l.jpg
View Operators

instance-level

Class

syntax<base class> := class(<view instance>)

semantics<view instance> is a instance of <base class>

components

syntax{<object>} := components (<view instance>)

semantics {<object>} := { oO | o is a component (direct or indirect) of <view instance>}

i-overlap

syntax<boolean> := i-overlap (<view instnace1>, <view instance2>)

semanticstrue, if and only if ( o  O) (o  components (<view instance1>) and o  components(<view instance2>))


View algebra l.jpg
View Algebra

  • Functions

    -- derivation of new MVs from existing MVs

    Heuristic Enumeration

    • Blind enumeration

    • Content-based enumeration

    • Semantics-based enumeration


View algebra31 l.jpg
View Algebra

  • Algebra Operators

    • select from src-MV where <predicate>

    • project <property-list> from src-MV

    • intersect (src-MV1, src-MV2)

    • union (src-MV1, src-MV2)

    • difference (src-MV1, src-MV2)




Logical implementation l.jpg
Logical Implementation

  • MediaView Construction

  • MediaView Customization

  • MediaView Evolution


Mediaviews construction l.jpg
MediaViews Construction

  • Work with CBIR systems to acquire the knowledge from queries

    • Learn from previously performed queries

    • A multi-system approach to support multi-modality of media objects

  • Organize the semantics by following WordNet


Why wordnet l.jpg
Why WordNet?

  • Different queries may greatly vary with the liberty of choosing query keywords

  • We need an approach to organize those knowledge into a logic structure

    • A simple “context”: a concept in WordNet

    • Common media views: corresponds to simple contexts

    • We provide all common media views, based on which users can build complex ones.


Navigating the multimedia database l.jpg
Navigating the Multimedia Database

  • Navigating via semantic relationships of WordNet

    Semantic Relationship Examples

    Synonymy (similar) pipe, tube

    Antonymy (opposite) fast, slow

    Hyponymy (subordinate) tree, plant

    Meronymy (part) chimney, house

    Troponomy (manner) march, walk

    Entailment drive, ride




Mediaview customization l.jpg
MediaView Customization

  • Two level MediaView Framework


Mediaview customization41 l.jpg
MediaView Customization

  • Dynamically construct complex-context-based media views based on simple ones

    • An example complex context: “the Grand Hall in City University”

  • Several user-level operators are devised to support more complex/advanced contexts, besides the basic operators


User level operators l.jpg
User-level Operators

  • INHERIT_MV(N: mv-name, NS: set-of-mv-refs, VP: set-of-property-ref, MP: set-of-property-ref): mv-ref

  • UNION_MV(N: mv-name, NS: set-of-mv-refs): mv-ref

  • INTERSECTION_MV(N: mv-name, NS: set-of-mv-refs): mv-ref

  • DIFFERENCE_MV(N1: mv-ref, N2: mv-ref): mv-ref


Build a mediaview in run time l.jpg
Build a MediaView in Run-time

  • Example: find out info about "Van Gogh"

    • Who is "Van Gogh"?

    • What is his work?

    • Know more about his whole life.

    • Know more about his country.

    • See his famous painting "sunflower"


Build a mediaview in run time44 l.jpg
Build a MediaView in Run-time

  • Who is “Van Gogh”?

    • INHERIT_MV(“V. Gogh“, {<painter>},name=”Van Gogh” ,);

  • What is his work?

    • INTERSECTION_MV(“work”, {<painting>, vg});

  • Know more about his whole life.

    • INTERSECTION_MV(“life”, {<biography>, vg});

  • Know more about his country.

    • INTERSECTION_MV(“country”, {<country>, vg});

  • See his famous painting “sunflower”

    • Set sunflower = INTERSECTION_MV(“sunflower”, {<sunflower>, <painting>});Set vg_sunflower = INTERSECTION_MV(“vg_sunflower”, {vg_work, sunflower});


Authoring scenario l.jpg
Authoring Scenario

  • Creates a new media view named after the subject

    • All multimedia materials used in the document would be put into this MediaView for further reference.

  • To collect the most relevant materials for authoring, the user performs the MediaView building process.

    • Import suitable media objects by browsing media views

  • Reference the manner and style of authoring, to find other media views with similar topics.

    • Drag & Drop

    • “learning-from-references”


Summary l.jpg
Summary

  • Types of multimedia data: Text, Audio, Video, Images.

  • Management issues: Design, Storage, Modeling, Queries

  • Image Content Analysis: Color, Texture, Shape

  • MediaView – a semantic multimedia database modeling mechanism

    • to bridge the semantic gap between conventional database and semantics-intensive multimedia applications

  • A set of user-level operators to accommodate the specialization/generalization relationships among the media views


Summary contd l.jpg
Summary (contd..)

  • MediaView promises more effective access to the content of media databases

    • Users could get the right stuff and tailor it to the context of their application easily.

  • Providing the most relevant content from pre-learnt semantic links between media and context

    • high performance database browsing and multimedia authoring tools can enable more comprehensive applications to the user.

    • Users could customize specific media view according to their tasks, by using user-level operators


Further issues l.jpg
Further Issues

  • The development and transition of MediaView to a fully-fledged multimedia database system supporting “declarative” queries

  • Intensive and extensive performance studies

  • Advanced semantic relations (eg. temporal and spatial ones) can also be incorporated in combining individual media views



ad