5. Multimedia Data

1. 5. Multimedia Data

2. Multimedia Data Representation • Digital Audio • Sampling/Digitisation • Compression (Details of Compression algorithms – following chapters) • Graphics/Image Formats • Digital Video (Basics of Video, Didgiatl Video, closely entwined with Compression)

3. Digital Audio Application of Digital Audio - Selected Examples • Music Production • Hard Disk Recording • Sound Synthesis • Samplers • Effects Processing • Video - Audio Important Element: Music and Effects • Web - Many uses on Web • Spice up Web Pages • Listen to Cds • Listen to Web Radio • Many More Uses - try and think of some?

4. Digitization of Sound • Sound is a continuous wave that travels through the air • The wave is made up of pressure differences. Sound is detected by measuring the pressure level at a particular location. • Sound waves have normal wave properties (reflection, refraction, diffraction, etc.).

5. Source • Generates Sound • Air Pressure changes • Electrical - Loud Speaker • Acoustic - Direct Pressure Variations • Destination • Receives Sound • Electrical - Microphone produces electric signal • Ears - Responds to pressure hear sound

6. Sound is required input into a computer: it needs to sampled or digitised: • Microphones, video cameras produce analogue signals (continuous-valued voltages). • Continuous Analogue Waveform • To get audio or video into a computer, we have to digitize it (convert it into a stream of numbers). Need to convert Analogue-to-Digital - Specialised Hardware • So, we have to understand discrete sampling (both time and voltage) • Sampling - divide the horizontal axis (the time dimension) into discrete pieces. Uniform sampling is ubiquitous. • Quantization - divide the vertical axis (signal strength) into pieces. Sometimes, a non-linear function is applied. • 8 bit quantization divides the vertical axis into 256 levels; 16 bit gives you 65536 levels.

7. Digitizing Audio • That is the basic idea of digitizing a sound unfortunately things are (practically speaking) not so simple. • Questions for producing digital audio (Analogue-to-Digital Conversion): 1. How often do you need to sample the signal? 2. How good is the signal? 3. How is audio data formatted?

8. Computer Manipulation of Sound • Once Digitised processing the digital sound is essentially straightforward although it depends on the processing you wish to do (e.g. volume is easier to code than accurate reverb) Sample Rates and Bit Size How do we store each sample value (Quantisation)? • 8 Bit Value • (0-255) • 16 Bit Value • (Integer) (0-65535) How many Samples to take? • 11.025 KHz Speech (Telephone 8KHz) • 22.05 KHz Low Grade Audio (WWW Audio, AM Radio) • 44.1 KHz CD Quality

9. Implications of Sample Rate and Bit Size Affects Quality of Audio • Ears do not respond to sound in a linear fashion • Decibel (dB) a logarithmic measurement of sound • 16-Bit has a signal-to-noise ratio of 98 dB - virtually inaudible • 8-bit has a signal-to-noise ratio of 50 dB • Therefore, 8-bit is roughly 8 times as noisy • 6 dB increment is twice as loud

10. Typical Audio Formats • Popular audio file formats include .au (Unix workstations), .aiff (MAC, SGI), .wav (PC, DEC workstations) • A simple and widely used audio compression method is Adaptive Delta Pulse Code Modulation (ADPCM). Based on past samples, it predicts the next sample and encodes the difference between the actual value and the predicted value.

11. Delivering Audio over a Network • Trade off between desired fidelity and file size Bandwidth Considerations for Web and other media. Compress Files: Could affect live transmission on Web

12. Streaming Audio Buffered Data: • Trick get data to destination before it's needed • Temporarily store in memory (Buffer) • Server keeps feeding the buffer • Client Application reads buffer Needs Reliable Connection, moderately fast too. Specialised client, Steaming Audio Protocol (PNM for real audio).

13. Introduction to MIDI (Musical Instrument Digital Interface) Definition of MIDI: a protocol that enables computer, synthesizers, keyboards, and other musical device to communicate with each other Components of a MIDI System: Synthesizer (a sound generator) Sequencer (a music editor) Track (to organize the recordings) Channel (to separate information in a MIDI system) Timbre (the quality of the sound) Pitch (musical note that the instrument plays) Voice (the portion of the synthesizer that produces sound) Patch (the control settings that define a particular timbre)

14. Digital Audio and MIDI • There are many application of Digital Audio and MIDI being used together: • Modern Recording Studio - Hard Disk Recording and MIDI • Analogue Sounds (Live Vocals, Guitar, Sax etc) - DISK • Keyboards, Drums, Samples, Loops Effects - MIDI • Sound Generators: use a mix of • Synthesis • Samples • Samplers - Digitise (Sample) Sound then • Playback • Loop (beats) • Simulate Musical Instruments

15. Digital Audio, Synthesis, Midi and Compression - MPEG 4 Structured Audio • We have seen the need for compression already in Digital Audio - Large Data Files • Basic Ideas of compression (see following Chapters) used as integral part of audio format - MP3, real audio etc. • Mpeg-4 audio - actually combines compression synthesis and MIDI to have a massive impact on compression. • MIDI, Synthesis encode what note to play and how to play it with a small number of parameters - Much greater reduction than simply having some encoded bits of audio. • Responsibility to create audio delegated to generation side.

16. MPEG 4 Structured Audio MPEG-4 covers the whole range of digital audio: • from very low bit rate speech • to full bandwidth high quality audio • built in anti-piracy measures • Structured Audio Structured Audio Tools MPEG-4 comprises of 6 Structured Audio tools