1 / 41

CS257 Modelling Multimedia Information LECTURE 5

CS257 Modelling Multimedia Information LECTURE 5. Introduction. Let’s review… What we’re dealing with (video data) What kinds of moving images might be stored in a digital video database What kinds of queries people might want to make to a video database

dory
Download Presentation

CS257 Modelling Multimedia Information LECTURE 5

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CS257 Modelling Multimedia InformationLECTURE 5

  2. Introduction • Let’s review… • What we’re dealing with (video data) • What kinds of moving images might be stored in a digital video database • What kinds of queries people might want to make to a video database  Techniques for modelling and querying video content

  3. Digital Video Data is… • A sequence of frames, where each frame is an image (typically 25 frames per second) • May include a soundtrack: Speech (commentary / monologue / dialogue); Music; Sound effects • May include text, i.e. subtitles / closed captions • Video data has temporal aspects as well as spatial aspects: the temporal organisation of the moving images conveys meaning in its own right – change the order of scenes  change the meaning of a film • Cinematic techniques (pan, zoom, etc.) and editing effects can alsoconvey information

  4. Moving Images include… • Television • Cinema • Meteorological images • Surveillance cameras • Medical images • Biomechanical images • Dance  Diverse range of video database applications, users and information needs

  5. What is Video Content? • Generally speaking video content can be said to comprise: • Objects (including people) with properties AND • Activities (actions, events) involving 0 or more objects • Organised in space and time • (This is an attempt at a generic approach, i.e. one that is applicable to all kinds of moving images)

  6. Example queries to film archives “Transport” “Meercats” “Portillo” “Ferries departing at night” “Animal rights protesters outside Hillingdon cat farm, wearing masks” From Enser and Sandom (2002).

  7. Queries to Video Databases • User may not want to see the whole of a video file (e.g. a whole film), may just want certain parts or intervals (e.g. shots or scenes)  Need to attach metadata to different parts of the video [More on this in Part 1 of this Lecture]

  8. Queries to Video Databases • Users may want to query for a particular event involving particular people, e.g. “find me video with Bill hitting Tom” – why not use a list of keywords [hit, Bill, Tom] for query and to represent film content?  Need more structured descriptions of what’s happening (both for queries and for video metadata), i.e. who is doing what to whom with what and why. [More on this in Lecture 6]

  9. Queries to Video Databases • User may want to specify a temporal sequence of events, e.g. “find me video where this happens then this happens while that happens” [More on this in Lecture 6]

  10. Overview of LECTURE 5 • PART 1:Need to be able to index (attach metadata to) and retrieve parts of video data files:  Video Data Models • PART 2: How developments in the MPEG standards have enabled new ways to access and create digital video • LAB – Finish previous exercises; bring coursework questions

  11. PART 1: Video Data Models • Must decide how to model (i.e. structure) video data so that metadata can be attached appropriately – must consider potential user information needs • Video data can be modelled as: • BLOb’s (Binary Large Objects) • Frames • Intervals (discrete, hierarchical, overlapping); • Object-based schemes (cf. MPEG-4) • NOTE, this does not entail chopping up the video data file

  12. Modelling Video Data as a BLOb • Metadata may be associated with the video data as a whole • The kinds of metadata for visual information discussed in Lecture 4 apply equally well to moving images; but note – ideally only metadata that is true for the whole video data file should be associated with a BLOb

  13. Modelling Video Data as a BLOb Attribute1: Attribute2: Attribute3: AttributeX… Time

  14. Modelling Video Data as Frames • An exhaustive metadata description of a video data file would include details for each and every frame (remember each frame is a still image) • However, with 25-30 fps, the cost of this is usually prohibitive and there are few applications where it would be beneficial

  15. Modelling Video Data as Frames Frame: 1 Attribute A: Attribute B: Frame: 2 Attribute A: Attribute B: Frame: 3 Attribute A: Attribute B: Time

  16. Modelling Video Data as Intervals • It is more usual to model video data as ‘meaningful’ intervals – where ‘meaningful’ depends on the particular domain and application • The intervals may be discrete or overlapping • The intervals may be arranged in a hierarchy so that metadata descriptions can be inherited

  17. Discrete Intervals Interval: 1 Start Time / Frame: 1 End Time / Frame: 103 Attribute A: Attribute B: Attribute C: Interval: 2 Start Time / Frame: 104 End Time / Frame: 155 Attribute A: Attribute B: Attribute C: Interval 1 Interval 2 Interval 3 Interval 4 Time

  18. Interval 1 Interval 2 Interval 3 Overlapping Intervals Interval: 2 Start Time / Frame: 008 End Time / Frame: 035 Attribute A: Attribute B: Attribute C: Interval: 3 Start Time / Frame: 045 End Time / Frame: 086 Attribute A: Attribute B: Attribute C: Interval 4 Time

  19. Hierarchical Intervals Interval: 10 Start Time / Frame: 000 End Time / Frame: 210 Attribute A: Attribute B: Attribute C: Interval 10 Interval 7 Interval 8 Interval 9 Interval 1 Interval 2 Interval 3 Interval 4 Interval 5 Int 6 Time

  20. Hierarchical Intervals • One example of video data that can modelled as hierarchical intervals are news broadcasts, made up of separate parts (main news, sports news, weather) and each part has separate stories, and each story is made up of shots and scenes • Then attach metadata to each shot and scene

  21. From: Corridoni et al (1996), ‘Multi-perspective Navigation of Movies’, Journal of Visual Languages and Computing 7, 445-466. [Available through Library eJournals].

  22. Exercise 5-1 • Metadata about both audio and video data streams can be organised by modelling the data as a Binary Large Object (BLOb) or in terms of intervals. Intervals may be modelled as single- or multi-levelled, they may be discrete or overlapping, and they may be organised in a hierarchy. Give an example of a video stream and an example of an audio stream which would be best modelled as: • (i)A Binary Large Objects (BLOb) • (ii)Overlapping multi-level intervals • (iii)A hierarchy of discrete intervals • For each of (i), (ii) and (iii) you should discuss the information needs of potential users and give examples of the metadata you would attach to the data streams

  23. Part 2: Developments in MPEG standards for digital video Video Data: In the beginning…. • Before the mid-1980s, digitising, storing and displaying still images, never mind moving ones, required dedicated high power and high cost hardware • Now video data is commonplace on home PCs; and throughout the TV and film industries • Moreover, the ways in which video data can be accessed have become increasingly intuitive and flexible. In a large part these developments have been driven by successive MPEG standards  The MPEG standards: some hows and whys….

  24. Moving Pictures Expert Group • A committee comprising 100s of international experts • MISSION: “Development of international standards for compression, decompression, processing and coded representation of moving pictures, audio, and their combination, in order to satisfy a wide variety of applications” Each standard developed with a specific aim

  25. MPEG-1: why? Compression: video data at 1.5 Mbits/s • e.g. for storage on a CD-ROM and playback on a low-spec PC • then for distribution of video data over the Internet (not necessarily in real time) – progressive rendering Started 1988; finished 1992 (MPEG-1 Audio Layer III  MP3)

  26. MPEG-1: how? Combines visual, audio and timing information into one data stream suitable for digital transmission Removes redundancy • Inter-frame redundancy (cf. still images) • Intra-frame redundancy – i.e. regions that do not change (significantly) -> store some frames and interpolate others Quality issues: fidelity of colour map; pixel resolution; frame resolution

  27. MPEG-2: why? Television requirements • higher quality • “efficient transmission over error-prone delivery systems” • Set top boxes and DVDs • TV and film production Started 1990; finished 1996

  28. MPEG-2: how? Digital Storage Media Command and Control: protocols for managing bitstreams Requirements • Higher bandwidth • Dedicated hardware for coding (far from real time); professional systems still cost many £K (?MPEG-3?)

  29. MPEG-4: why? Next generation mobile communication • Further compression • PLUS… Reuse / Personlisation / Customisation / Interactivity / Creativity Requirements • Composition of real and synthetic media • Multiplex and synchronise objects over networks • Interact with audiovisual scene that is generated at the receiver’s end (Started 1993; parts finished in 2000/1)

  30. MPEG-4: how? **FROM FRAMES TO AUDIO-VISUAL OBJECTS** Coding of audio-visual objects - real or synthetic • Still images (e.g. a fixed background) • Video objects (e.g. talking person); need not be rectangular frames • Audio objects (e.g. person’s speech) • Text and graphics – inc. VR • Synthetic heads, speech and sound

  31. MPEG-4: how? Standard scene description (cf. VRML): • Place media objects in coordinate system • Apply transforms to change geometry / acoustics • Form compound media objects • Apply streamed data to objects to modify attributes • Interactively change user’s viewing and listening points

  32. MPEG-7 Indexing and retrieval “Multimedia Content Description Interface” “different from previous MPEG standards… what is represented is not the information itself but information about the information” i.e. it is a metadata standard.

  33. MPEG-7 Description Definition Language • for low-level visual features; • for low-level audio features; • for ‘semantic content’, e.g. descriptions of the narrative world depicted by a piece of media • Started 1997; official standard in 2001 • MPEG-7 cameras developed to capture some features at time of recording

  34. EXERCISE 5-2 • Consider the following people who are each developing a different multimedia system to handle digital video data. Which of the MPEG standards would be of interest to each person and why? • Annabel: she has collected a digital library of historical documentaries which she would like people to be able to access via a high-speed intranet by making a range of queries about semantic content • Brian: he is distributing a video browser with his home-made cartoons on a CD-ROM • Claire: she is setting up a pay-per-view digital television service • Dave: he is an experimental filmmaker who is developing a system to help him edit video material in new and interesting ways • It can be argued that the format in which multimedia data is coded has an impact on the potential functionality of systems that will store, retrieve and manipulate that data. Discuss, with explanations, the impact on potential system functionality that you think could arise from the following choices of coding format: • Coding a moving image in MPEG-1 or coding it in MPEG-4 • Coding a piece of music in ‘.wav’ or coding it in ‘.midi’

  35. LECTURE 5:LEARNING OUTCOMES • After the lecture, you should be able to: • Select an appropriate video data model given a specific kind of video data and some user information needs, and explain / justify your decision • Describe and explain how each MPEG standard enabled new ways to access and create digital video

  36. OPTIONAL READING • For information about all the MPEG standards, see: www.chiariglione.org/mpeg/index.htm • For an overview of MPEG-7: www.chiariglione.org/mpeg/standards/mpeg-7/mpeg-7.htm

  37. OPTIONAL READING • Benitez et al. (2002), “Description of a Single Multimedia Document”, in: Manjunath, Salembier and Sikora (eds.), Introduction to MPEG-7: multimedia content description interface. There are 4 copies of this article in the Library Article Collection. • Note, focus on pages 113-4 and 124-138, and particularly on Figs. 8.9, 8.11, 8.14 and 8.16.

  38. OPTIONAL READING Multimedia Standards in Television / Media Production • Issues of multimedia standards are discussed by a number of other professional groups, including those interested in mxf (multimedia exchange format) and aaf (advanced authoring format). For a short article about mxf and aaf, see: www.snellwilcox.com/knowledgecenter/mxf_aaf.html Homepages; mxf: www.pro-mpeg.org/ aaf: www.aafassociation.org/

  39. OPTIONAL READING • The following articles are available for download via the University Library’s e-journals section. Many of the authors are from the MPEG-7 committee but the papers are more concise than the official MPEG technical documents. Martinez, J.M., Koenen, R., and Pereira, F. (2002), “MPEG-7: the generic multimedia content description standard, part 1.” IEEE Multimedia 9 (2), April-June 2002, 78–87 Martinez, J.M. (2002), “Standards - MPEG-7 overview of MPEG-7 description tools, part 2. ” IEEE Multimedia 9 (3), July-Sept. 2002, 83 -93. Chang, S.-F., Sikora, T. and Purl (2001), “An Overview of the MPEG-7 Standard”, IEEE Transactions on Circuits and Systems for Video Technology, 11 (6), 688-695. Nack, F. and Lindsay, A.T. (1999), “Everything you wanted to know about MPEG-7. Part 1.” IEEE Multimedia 6 (3), July-Sept. 1999, 65 –77. Nack, F. and Lindsay, A.T. (1999), “Everything you wanted to know about MPEG-7. Part 2.” IEEE Multimedia 6 (4), Oct.-Dec. 1999 , 64 –73

  40. Today’s Lab • Continue with image retrieval

More Related