Technical University of Crete Department of Electronic and Computer Engineering

1. Technical University of CreteDepartment of Electronic and Computer Engineering Multimedia Data Management Euripides G.M. Petrakis http://www.intelligence.tuc.gr/~petrakis http://courses.ced.tuc.gr Chania 2010 Introduction

2. Definition • Multimedia: composite entities combining text, audio, images, video (bit-stream objects), graphics • Multimedia Information Systems: database systems that support all multimedia data types and handle very large volumes of information Introduction

3. TEXT • The most common type of information • The least space intensive data type • The form in which text is stored varies (plain ascii, word files, spreadsheets, annotations, database fields etc.) • Text fonts are becoming complex allowing special effects (color, shade, fill etc.) Introduction

4. AUDIO • Space intensive (one minute can take up Mbytes), presented as analog, digital or MIDI • Analog waveform: electrical signal, • amplitude specifies the loudness of the sound • in microphones, tapes, records, amplifiers, speakers • Digital waveform audio: digital, • less sensitivity to noise and distortion • involves larger processing and storage capacities • Digital Audio Tape (DAT), Compact discs (CD) • WAV (Microsoft’s wave file format) Introduction

5. MIDI (Musical Instrument Digital Interface) Furht et.al.96 • Commands that describe how the music should be played are stored (instead of sound) • A music synthesizer generates sound • Provides high data compression, • Widely accepted Introduction

6. IMAGES • Digital images: sequences of pixels • Pixels: numbers interpreted to display intensity, color, contrast etc • Binary (0-1 values), gray-scale (8 bits/pixel), colour (3x8 values for RGB) • Space overhead depends on image type, resolution, compression scheme • Image formats: tiff, bmb, jpeg etc. Introduction

7. Image Concepts and Structures • Binary images: 1 bit/pixel • black & white photos, facsimile images • Computer Graphics: 4 bits/pixel • Grayscale images: 8 bits/sample • Color images: 16, 24 bits/pixel Introduction

8. RGB Representation • A color is produced by adding • red, green and blue • The straight line R=G=B specifies gray values ranging from black to white Introduction

9. YUV Representation • YUV describes the luminance and chrominance components of an image • 1 luminance: gray scale version of an image • Y = 0.299R + 0.587G + 0.114B • 2 chrominance components: • U = 0.564(B - Y) • V = 0.713(R - Y) Introduction

10. Conversions • Conversion between RGB and YUV requires multiplication operations • an approximation: • Y = R/4 + G/2 +B/2, U=(B-Y)/2, V=(R-Y)/2 • R = Y + 2V, G = Y – (U + V), B = Y + 2U • YCbCr is another color format • for compression • Cb = U/2 + 0.5, Cr = V/1.6 + 0.5 Introduction

11. VIDEO • The most space intensive data type • A sequence of frames • Realistic video playback, transmission, compression/decompression require transfer rates about 30frames/sec • Microsoft’s AVI and Apple’s Quicktime file formats integrate video and audio in the same presentation Introduction

12. Audio-video Modes of Operation Furht et.al.96 • Can be either stored or used / transmitted live in real-time • Can be used interactively or non-interactively Introduction

13. GRAPHICS • Objects described through their basic elements (e.g., 2D, 3D shapes) • these elements can have different sizes, position, orientation, surface, fill etc. • compact representations • generated and can be manipulated by design tools (e.g., CAD tools) • Their descriptions are stored in files Introduction

14. 2D and 3D graphics objects Khoshafian Baker 96 Introduction

15. MULTIMEDIA objects • Text, audio, images, video, graphics are elements of complex multimedia objects • Various tools or applications integrate, process and combine multimedia • Applications: multimedia authoring applications that output documents and databases and end-user applications (e.g., video on demand) • Tools: for viewing, updating, querying (presentation viewers, browsers etc.) Introduction

16. Multimedia Databases (MDB) • Means stored information or database management systems (dbms) • Multimedia dbms (mdbms) integrate conventional database capabilities together with different technologies such as Hierarchical storage management (HSM) and Information retrieval (IR) Introduction

17. Multimedia Technologies • Technologies integrated within a mdbms • HSM support • IR support (exact and approximate) • Spatial data types and queries • Interactive querying, relevance feedback, refining • Automatic feature extraction • Automatic content retrieval and indexing • Query optimization Introduction

18. Database capabilities • Persistence: object persist through invocations • Transactions: content is inserted, deleted, updated • Concurrency control: transactions run concurrently • Recovery: failed transactions are not propagated to the db • Querying: content can be retrieved • Versioning: access previous states of objects • Integrity: transactions guarantee consistency of content • Security: constraints for accessing/updating objects • Performance: optimal data structures and programs Introduction

19. Hierarchical Storage Management (HSM) • Support storage of multimedia objects • On-line: on RAM, magnetic disk • Near-on-line: on optical storage • Off-line: on tapes, shelves • Each level has different • Performance: decreases from top to bottom • Capacity: increases from top to bottom • Cost: decreases from top to bottom Introduction

20. Information Retrieval (IR) Capabilities • Retrieval is the most common operation • Deletions and updates are less common • Exact match: search based on exact information • Inexact: search based on inexact information e.g.,partial, neighborhood search, can be fuzzy or probabilistic • The results are ranked by order of relevance to the query • Query refinement • Iterate over query results • Adjust weights of query terms or features • And finally resubmit queries Introduction

21. MDBMS architecture Khoshafian Baker 96 Introduction

22. MDBMS Implementation • Relies on 3rd party vendors for each component • Relational dbms for typical records • separate optical storage module for text/audio/graphics/images/video • Text retrieval system (e.g. Lucene) • Audio/image/video retrieval system • Feature extraction system • Multimedia object interface system Introduction

23. Object-Oriented Multimedia Databases • Better design, better suited for multimedia applications • Uniform handling of data and operations • Data types are objects with internal structures and operations that capture the behavior of objects (e.g., audio playback, video browsing) • OO dbms does not satisfy all MM requirements • Provides primitives for object handling • Multimedia components need to be implemented or integrated Introduction

24. Multimedia Applications • Multimedia Systems suggest a variety of applications • Multimedia conferencing • Multimedia on demand (interactive TV, news on demand) • See next page for more … Introduction

25. Multimedia Applications Furht et.al.96 Introduction

26. Multimedia Conferencing (MC) • Multimedia conferencing enable a number of participants to exchange multimedia information • Each participant has a workstation linked to other workstations over high-speed networks • Each participant can send or receive mm data and perform certain collaborative activities Introduction

27. A video conference system Furht et.al.96 • The biggest performance challenge occurs when the participants transmit voice and video • These are mixed together to form a composite stream consisting of video and voice streams Introduction

28. Software Architecture Furht et.al.96 Introduction

29. Architectures • Fully distributed: direct connections between the participants • Processing and mixing of media at every location • Shortest delay • The connections increase rapidly • Centralized (star) network: a central is connected to every participant • Processing and mixing at central node • The central node waits until all media is received before mixing and broadcasting Introduction

30. Architectures (cont.) • Double star network: a central node from one star network is connected to another central node of another star network • Hierarchical network: intermediate nodes, root and leaves (participants) connected hierarchically • intermediate nodes perform mixing and processing • the completely mixed data is sent to root who broadcasts directly to the leaves • reduces network traffic significantly Introduction

31. Multimedia conferencing network architectures Furht et.al.96 Introduction

32. Video on Demand (VoD) • Fast networks coupled with powerful computers and compression techniques will be capable of delivering stream data in real-time • On-demand multimedia services • interactive entertainment • video news distribution • video rental services • digital multimedia libraries Introduction

33. Interactive Television (ITV) • An ITV system must be capable of providing • basic TV • subscription TV • pay per view • video on demand • shopping • education • electronic newspaper • financial transactions • single-user and multi-user games Introduction

34. This Course • Emphasis on • Text, images, video • Information retrieval & systems • Data organization • Web information systems • Semantic Web • Video & MPEG standards • No emphasis on • Architectures • Specific applications (VoD, ITV,MC) • Services Introduction