Cs257 modelling multimedia information lecture 5
1 / 41

CS257 Modelling Multimedia Information LECTURE 5 - PowerPoint PPT Presentation

  • Uploaded on

CS257 Modelling Multimedia Information LECTURE 5. Introduction. Let’s review… What we’re dealing with (video data) What kinds of moving images might be stored in a digital video database What kinds of queries people might want to make to a video database

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'CS257 Modelling Multimedia Information LECTURE 5' - dory

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript


  • Let’s review…

    • What we’re dealing with (video data)

    • What kinds of moving images might be stored in a digital video database

    • What kinds of queries people might want to make to a video database

       Techniques for modelling and querying video content

Digital video data is
Digital Video Data is…

  • A sequence of frames, where each frame is an image (typically 25 frames per second)

  • May include a soundtrack: Speech (commentary / monologue / dialogue); Music; Sound effects

  • May include text, i.e. subtitles / closed captions

  • Video data has temporal aspects as well as spatial aspects: the temporal organisation of the moving images conveys meaning in its own right – change the order of scenes  change the meaning of a film

  • Cinematic techniques (pan, zoom, etc.) and editing effects can alsoconvey information

Moving images include
Moving Images include…

  • Television

  • Cinema

  • Meteorological images

  • Surveillance cameras

  • Medical images

  • Biomechanical images

  • Dance

     Diverse range of video database applications, users and information needs

What is video content
What is Video Content?

  • Generally speaking video content can be said to comprise:

    • Objects (including people) with properties


    • Activities (actions, events) involving 0 or more objects

    • Organised in space and time

  • (This is an attempt at a generic approach, i.e. one that is applicable to all kinds of moving images)

Example queries to film archives
Example queries to film archives




“Ferries departing at night”

“Animal rights protesters outside Hillingdon cat farm, wearing masks”

From Enser and Sandom (2002).

Queries to video databases
Queries to Video Databases

  • User may not want to see the whole of a video file (e.g. a whole film), may just want certain parts or intervals (e.g. shots or scenes)

     Need to attach metadata to different parts of the video [More on this in Part 1 of this Lecture]

Queries to video databases1
Queries to Video Databases

  • Users may want to query for a particular event involving particular people, e.g. “find me video with Bill hitting Tom” – why not use a list of keywords [hit, Bill, Tom] for query and to represent film content?

     Need more structured descriptions of what’s happening (both for queries and for video metadata), i.e. who is doing what to whom with what and why. [More on this in Lecture 6]

Queries to video databases2
Queries to Video Databases

  • User may want to specify a temporal sequence of events, e.g. “find me video where this happens then this happens while that happens”

    [More on this in Lecture 6]

Overview of lecture 5
Overview of LECTURE 5

  • PART 1:Need to be able to index (attach metadata to) and retrieve parts of video data files:  Video Data Models

  • PART 2: How developments in the MPEG standards have enabled new ways to access and create digital video

  • LAB – Finish previous exercises; bring coursework questions

Part 1 video data models
PART 1: Video Data Models

  • Must decide how to model (i.e. structure) video data so that metadata can be attached appropriately – must consider potential user information needs

  • Video data can be modelled as:

    • BLOb’s (Binary Large Objects)

    • Frames

    • Intervals (discrete, hierarchical, overlapping);

    • Object-based schemes (cf. MPEG-4)

  • NOTE, this does not entail chopping up the video data file

Modelling video data as a blob
Modelling Video Data as a BLOb

  • Metadata may be associated with the video data as a whole

  • The kinds of metadata for visual information discussed in Lecture 4 apply equally well to moving images; but note – ideally only metadata that is true for the whole video data file should be associated with a BLOb

Modelling video data as a blob1
Modelling Video Data as a BLOb






Modelling video data as frames
Modelling Video Data as Frames

  • An exhaustive metadata description of a video data file would include details for each and every frame (remember each frame is a still image)

  • However, with 25-30 fps, the cost of this is usually prohibitive and there are few applications where it would be beneficial

Modelling video data as frames1
Modelling Video Data as Frames

Frame: 1

Attribute A:

Attribute B:

Frame: 2

Attribute A:

Attribute B:

Frame: 3

Attribute A:

Attribute B:


Modelling video data as intervals
Modelling Video Data as Intervals

  • It is more usual to model video data as ‘meaningful’ intervals – where ‘meaningful’ depends on the particular domain and application

  • The intervals may be discrete or overlapping

  • The intervals may be arranged in a hierarchy so that metadata descriptions can be inherited

Discrete intervals
Discrete Intervals

Interval: 1

Start Time / Frame: 1

End Time / Frame: 103

Attribute A:

Attribute B:

Attribute C:

Interval: 2

Start Time / Frame: 104

End Time / Frame: 155

Attribute A:

Attribute B:

Attribute C:

Interval 1

Interval 2

Interval 3

Interval 4


Overlapping intervals

Interval 1

Interval 2

Interval 3

Overlapping Intervals

Interval: 2

Start Time / Frame: 008

End Time / Frame: 035

Attribute A:

Attribute B:

Attribute C:

Interval: 3

Start Time / Frame: 045

End Time / Frame: 086

Attribute A:

Attribute B:

Attribute C:

Interval 4


Hierarchical intervals
Hierarchical Intervals

Interval: 10

Start Time / Frame: 000

End Time / Frame: 210

Attribute A:

Attribute B:

Attribute C:

Interval 10

Interval 7

Interval 8

Interval 9

Interval 1

Interval 2

Interval 3

Interval 4

Interval 5

Int 6


Hierarchical intervals1
Hierarchical Intervals

  • One example of video data that can modelled as hierarchical intervals are news broadcasts, made up of separate parts (main news, sports news, weather) and each part has separate stories, and each story is made up of shots and scenes

  • Then attach metadata to each shot and scene

Cs257 modelling multimedia information lecture 5

From: Corridoni et al (1996), ‘Multi-perspective Navigation of Movies’, Journal of Visual Languages and Computing 7, 445-466. [Available through Library eJournals].

Exercise 5 1
Exercise 5-1

  • Metadata about both audio and video data streams can be organised by modelling the data as a Binary Large Object (BLOb) or in terms of intervals. Intervals may be modelled as single- or multi-levelled, they may be discrete or overlapping, and they may be organised in a hierarchy. Give an example of a video stream and an example of an audio stream which would be best modelled as:

  • (i)A Binary Large Objects (BLOb)

  • (ii)Overlapping multi-level intervals

  • (iii)A hierarchy of discrete intervals

  • For each of (i), (ii) and (iii) you should discuss the information needs of potential users and give examples of the metadata you would attach to the data streams

Part 2 developments in mpeg standards for digital video
Part 2: Developments in MPEG standards for digital video

Video Data: In the beginning….

  • Before the mid-1980s, digitising, storing and displaying still images, never mind moving ones, required dedicated high power and high cost hardware

  • Now video data is commonplace on home PCs; and throughout the TV and film industries

  • Moreover, the ways in which video data can be accessed have become increasingly intuitive and flexible. In a large part these developments have been driven by successive MPEG standards

     The MPEG standards: some hows and whys….

Moving pictures expert group
Moving Pictures Expert Group

  • A committee comprising 100s of international experts

  • MISSION: “Development of international standards for compression, decompression, processing and coded representation of moving pictures, audio, and their combination, in order to satisfy a wide variety of applications”

    Each standard developed with a specific aim

Mpeg 1 why
MPEG-1: why?

Compression: video data at 1.5 Mbits/s

  • e.g. for storage on a CD-ROM and playback on a low-spec PC

  • then for distribution of video data over the Internet (not necessarily in real time) – progressive rendering

    Started 1988; finished 1992

    (MPEG-1 Audio Layer III  MP3)

Mpeg 1 how
MPEG-1: how?

Combines visual, audio and timing information into one data stream suitable for digital transmission

Removes redundancy

  • Inter-frame redundancy (cf. still images)

  • Intra-frame redundancy – i.e. regions that do not change (significantly) -> store some frames and interpolate others

    Quality issues: fidelity of colour map; pixel resolution; frame resolution

Mpeg 2 why
MPEG-2: why?

Television requirements

  • higher quality

  • “efficient transmission over error-prone delivery systems”

  • Set top boxes and DVDs

  • TV and film production

    Started 1990; finished 1996

Mpeg 2 how
MPEG-2: how?

Digital Storage Media Command and Control: protocols for managing bitstreams


  • Higher bandwidth

  • Dedicated hardware for coding (far from real time); professional systems still cost many £K


Mpeg 4 why
MPEG-4: why?

Next generation mobile communication

  • Further compression

  • PLUS… Reuse / Personlisation / Customisation / Interactivity / Creativity


    • Composition of real and synthetic media

    • Multiplex and synchronise objects over networks

    • Interact with audiovisual scene that is generated at the receiver’s end

      (Started 1993; parts finished in 2000/1)

Mpeg 4 how
MPEG-4: how?


Coding of audio-visual objects - real or synthetic

  • Still images (e.g. a fixed background)

  • Video objects (e.g. talking person); need not be rectangular frames

  • Audio objects (e.g. person’s speech)

  • Text and graphics – inc. VR

  • Synthetic heads, speech and sound

Mpeg 4 how1
MPEG-4: how?

Standard scene description (cf. VRML):

  • Place media objects in coordinate system

  • Apply transforms to change geometry / acoustics

  • Form compound media objects

  • Apply streamed data to objects to modify attributes

  • Interactively change user’s viewing and listening points

Mpeg 7

Indexing and retrieval

“Multimedia Content Description Interface”

“different from previous MPEG standards… what is represented is not the information itself but information about the information”

i.e. it is a metadata standard.

Mpeg 71

Description Definition Language

  • for low-level visual features;

  • for low-level audio features;

  • for ‘semantic content’, e.g. descriptions of the narrative world depicted by a piece of media

  • Started 1997; official standard in 2001

  • MPEG-7 cameras developed to capture some features at time of recording

  • Exercise 5 2
    EXERCISE 5-2

    • Consider the following people who are each developing a different multimedia system to handle digital video data. Which of the MPEG standards would be of interest to each person and why?

      • Annabel: she has collected a digital library of historical documentaries which she would like people to be able to access via a high-speed intranet by making a range of queries about semantic content

      • Brian: he is distributing a video browser with his home-made cartoons on a CD-ROM

      • Claire: she is setting up a pay-per-view digital television service

      • Dave: he is an experimental filmmaker who is developing a system to help him edit video material in new and interesting ways

    • It can be argued that the format in which multimedia data is coded has an impact on the potential functionality of systems that will store, retrieve and manipulate that data. Discuss, with explanations, the impact on potential system functionality that you think could arise from the following choices of coding format:

      • Coding a moving image in MPEG-1 or coding it in MPEG-4

      • Coding a piece of music in ‘.wav’ or coding it in ‘.midi’

    Lecture 5 learning outcomes

    • After the lecture, you should be able to:

      • Select an appropriate video data model given a specific kind of video data and some user information needs, and explain / justify your decision

      • Describe and explain how each MPEG standard enabled new ways to access and create digital video

    Optional reading

    • For information about all the MPEG standards, see: www.chiariglione.org/mpeg/index.htm

    • For an overview of MPEG-7:


    Optional reading1

    • Benitez et al. (2002), “Description of a Single Multimedia Document”, in: Manjunath, Salembier and Sikora (eds.), Introduction to MPEG-7: multimedia content description interface.

      There are 4 copies of this article in the Library Article Collection.

    • Note, focus on pages 113-4 and 124-138, and particularly on Figs. 8.9, 8.11, 8.14 and 8.16.

    Optional reading2

    Multimedia Standards in Television / Media Production

    • Issues of multimedia standards are discussed by a number of other professional groups, including those interested in mxf (multimedia exchange format) and aaf (advanced authoring format).

      For a short article about mxf and aaf, see:



      mxf: www.pro-mpeg.org/

      aaf: www.aafassociation.org/

    Optional reading3

    • The following articles are available for download via the University Library’s e-journals section. Many of the authors are from the MPEG-7 committee but the papers are more concise than the official MPEG technical documents.

      Martinez, J.M., Koenen, R., and Pereira, F. (2002), “MPEG-7: the generic multimedia content description standard, part 1.” IEEE Multimedia 9 (2), April-June 2002, 78–87

      Martinez, J.M. (2002), “Standards - MPEG-7 overview of MPEG-7 description tools, part 2. ” IEEE Multimedia 9 (3), July-Sept. 2002, 83 -93.

      Chang, S.-F., Sikora, T. and Purl (2001), “An Overview of the MPEG-7 Standard”, IEEE Transactions on Circuits and Systems for Video Technology, 11 (6), 688-695.

      Nack, F. and Lindsay, A.T. (1999), “Everything you wanted to know about MPEG-7. Part 1.” IEEE Multimedia 6 (3), July-Sept. 1999, 65 –77.

      Nack, F. and Lindsay, A.T. (1999), “Everything you wanted to know about MPEG-7. Part 2.” IEEE Multimedia 6 (4), Oct.-Dec. 1999 , 64 –73

    Today s lab
    Today’s Lab

    • Continue with image retrieval