Multimedia

1. Multimedia Week 9 Sound

2. Lecture Media dictionary Bitmaps Your interface Supported session Using imported artwork Look at navigation Last week Multimedia

3. Lecture Your interface and Flash navigation Sound Chapter 9 Digital Multimedia, 2nd edition The nature of sound Digitization Compression File formats Working with sound Supported session Working with sound Look at navigation This week Multimedia

4. Interactivity and Flash • An event triggers every action • In Flash no action occurs unless something triggers it • There are two types of events in Flash • Movie events • User events • All actions in Flash are a form of ActionScript Multimedia

5. Flash and interaction • Behaviors • Ready made scripts • You can use behaviors to control movie clips, video, and sound files. • ActionScript • A script can consist of a single command, such as instructing a SWF file to stop playing, • Or a series of commands and statements, such as first evaluating a condition and then performing an action Multimedia

6. Flash and navigation • Frame to frame navigation • Behaviors on (release) { //Movieclip GotoAndStop Behavior this.gotoAndStop("50"); //End Behavior • Load another movie • ActionScript Multimedia

7. Events, handlers and actions • Whenever a user clicks the mouse or presses a key, an event is generated. • In a SWF file, frames, buttons, movie clips, and text fields all generate events to which your user can respond • Actions • this.gotoAndStop("50"); • this.gotoAndPlay("50"); Multimedia

8. Sound in multimedia • Consider why you would use sound, will it add to your multimedia production? • List how it could be used in your multimedia production • What controls would you use? Why? Multimedia

9. 275–276 The Nature of Sound • Conversion of energy into vibrations in the air (or some other elastic medium) • Most sound sources vibrate in complex ways leading to sounds with components at several different frequencies • Frequency spectrum – relative amplitudes of the frequency components • Range of human hearing: roughly 20Hz–20kHz, falling off with age Chapman N and Chapman J (2004).

10. 276–280 Waveforms • Sounds change over time • e.g. musical note has attack and decay, speech changes constantly • Frequency spectrum alters as sound changes • Waveform is a plot of amplitude against time • Provides a graphical view of characteristics of a changing sound • Can identify syllables of speech, rhythm of music, quiet and loud passages, etc Chapman N and Chapman J (2004).

11. Amplitude • Amplitude is perceived as loudness High amplitude Lower amplitude Click on the amplitudes to hear the difference. Multimedia

12. Frequency • Frequency is perceived as pitch. High frequency Low frequency Click on the frequencies to hear the difference. Multimedia

13. 36 Sampling and Quantization • Why is the sampling rate important? • Why is the quantisation level important? Sampling Quantisation Chapman N and Chapman J (2004). Multimedia

14. 40–41 Sampling Theorem • If the highest frequency component of a signal is at fh the signal can be properly reconstructed if it has been sampled at a frequency > 2fh • Nyquist rate • Undersamping leads to aliasing • Sound distortion, image 'jaggies' or Moiré patterns, jerky or retrograde motion Chapman N and Chapman J (2004). Multimedia

15. 281–282 Digitization – Sampling • Captures amplitude values at regular intervals • Sampling rate is expressed in kHz (1000 per second) • Sampling Theorem implies minimum rate of 40kHz to reproduce sound up to limit of hearing • CD: 44.1kHz • Sub-multiples often used for low bandwidth – e.g. 22.05kHz for Internet audio • DAT: 48kHz • (Hence mixing sounds from CD and DAT will require some resampling, best avoided) Chapman N and Chapman J (2004).

16. Sampling standards • Telephone Quality • 11.025 kHz, 8-bit depth, Mono (10kb/sec) • Radio Quality • 22.050, 8-bit depth, Mono (21 kb/sec) • CD Quality • 44.100 kHz, 16-bit depth, Stereo (172 kb/sec) Multimedia

17. 283–285 Digitization – Quantization • 16 bits, 65536 quantization levels, CD quality • 8 bits: audible quantization noise, can only use if some distortion is acceptable, e.g. voice communication • Dithering – introduce small amount of random noise before sampling • Noise causes samples to alternate rapidly between quantization levels, effectively smoothing sharp transitions Chapman N and Chapman J (2004).

18. 283–284 Undersampling & Dithering Chapman N and Chapman J (2004).

19. Stereo • Stereo sound is composed of TWO very similar sound waves. • Subtle differences in location and movement • Must have a separate sound wave per channel (twice the size file) Multimedia

20. Mono • Uses only one channel • Good way to keep file sizes small • As a general rule, use mono where possible. Multimedia

21. 287 Data Size • Sampling rate r is the number of samples per second • Sample size s bits • Each second of digitized audio requires rs/8 bytes • CD quality: r = 44100, s = 16, hence each second requires just over 86 kbytes (k=1024), each minute roughly 5Mbytes (mono) Chapman N and Chapman J (2004).

22. 287–288 Clipping • If recording level is set too high, signal amplitude will exceed maximum that can be recorded, leading to unpleasant distortion • But if level is set too low, dynamic range will be restricted Chapman N and Chapman J (2004).

23. 289 Sound Editing • Timeline divided into tracks • Sound on each track displayed as a waveform • 'Scrub' over part of a track e.g. to find pauses • Cut and paste, drag and drop • May combine many tracks from different recordings (mix-down) Chapman N and Chapman J (2004).

24. 290–295 Effects and Filters • Noise gate • Low pass and high pass filters • Notch filter • De-esser • Click repairer • Reverb • Graphic equalizer • Envelope Shaping • Pitch alteration and time stretching • etc Chapman N and Chapman J (2004).

25. 295 Compression • In general, lossy methods required because of complex and unpredictable nature of audio data • CD quality, stereo, 3-minute song requires over 25 Mbytes • Data rate exceeds bandwidth of dial-up Internet connection • Difference in the way we perceive sound and image means different approach from image compression is needed Chapman N and Chapman J (2004).

26. 298–299 Perceptually-Based Compression • Identify and discard data that doesn't affect the perception of the signal • Needs a psycho-acoustical model, since ear and brain do not respond to sound waves in a simple way • Threshold of hearing – sounds too quiet to hear • Masking – sound obscured by some other sound Chapman N and Chapman J (2004).

27. 300–301 MP3 • MPEG Audio, Layer 3 • Three layers of audio compression in MPEG-1 (MPEG-2 essentially identical) • Layer 1...Layer 3, encoding proces increases in complexity, data rate for same quality decreases • e.g. Same quality 192kbps at Layer 1, 128kbps at Layer 2, 64kbps at Layer 3 • 10:1 compression ratio at high quality • Variable bit rate coding (VBR) Chapman N and Chapman J (2004).

28. 301 AAC • Advanced Audio Coding • Defined in MPEG-2 standard, extended and incorporated into MPEG-4 • Not backward compatible with earlier standards • Higher compression ratios and lower bit rates than MP3 • Subjectively better quality than MP3 at the same bit rate Chapman N and Chapman J (2004).

29. 302 Audio Formats • Platform-specific file formats • AIFF, WAV, AU • Multimedia formats used as 'container formats' for sound compressed with different codecs • QuickTime, Windows Media, RealAudio • MP3 has its own file format, but MP3 data can be included as audio tracks in QuickTime movies and SWFs Chapman N and Chapman J (2004).

30. 303–304 MIDI • Musical Instruments Digital Interface • Instructions about how to produce music, which can be interpreted by suitable hardware and/or software • cf. vector graphics as drawing instructions • Standard protocol for communicating between electronic instruments (synthesizers, samplers, drum machines) • Allows instruments to be controlled by hardware or software sequencers Chapman N and Chapman J (2004).

31. 304 MIDI and Computers • MIDI interface allows computer to send MIDI data to instruments • Store MIDI sequences in files, exchange them between computers, incorporate into multimedia • Computer can synthesize sounds on a sound card, or play back samples from disk in response to MIDI instructions • Computer becomes primitive musical instrument (quality of sound inferior to dedicated instruments) Chapman N and Chapman J (2004).

32. Editing sound • Use a piece of Sound Editing Software • Cool Edit 2000 • Sound manipulation program • open, create or record new sound files • Edit existing sounds • Mix audio or musical parts, touch them up, and add effects • Adobe Audition Multimedia

33. Sound and Flash • Sounds that continuously, independent of the Timeline • Or you can synchronize animation to a sound track. • You can add sounds to buttons to make them more interactive Macromedia, Flash MX 2004 Multimedia

34. Event sounds and Stream sounds • An Event sound must download completely before it begins playing, and it continues playing until explicitly stopped • Stream sounds begin playing as soon as enough data for the first few frames has been downloaded; stream sounds are synchronized to the Timeline for playing on a website. Macromedia, Flash MX 2004 Multimedia

35. Importing sound • You can import the following sound file formats into Flash: • WAV (Windows only) • AIFF (Macintosh only) • MP3 (Windows or Macintosh) • Select File > Import > Import to Library • To add a sound to a document from the library, you assign the sound to a layer and set options in the Sound controls in the Property inspector • It is recommended that you place each sound on a separate layer. • To define the starting point of a sound or to control the volume of the sound as it plays, you use the sound-editing controls in the Property inspector Macromedia, Flash MX 2004 Multimedia

36. Controlling sound • You can control sound playback using sound Behaviors or ActionScript • The most common sound-related task in Flash is starting and stopping sounds at keyframes in synchronization with animation • You can use Flash to compress sound when it is published Macromedia, Flash MX 2004 Multimedia

37. Sync options • Event • The entire sound must download first • Can be triggered more than once • Will play all the way through • Start • Cannot have multiple instances • Stop • Stops Event or Start sound • Stream • Breaks the sound up into frame-sized pieces which means it does not all need to be downloaded before the Movie can play Multimedia

38. Sound strategy • Re-use sound from the library • It is only stored once • Loop shorter sounds in place of importing longer sounds • Compress as much as possible without loss of quality • Use separate sound-editing program to simplify and remove extraneous noise Kay M. (2003) Multimedia

39. A question • Scenario: You have been asked by Hope library to add a sound track to a flash movie on using the library • Discuss and critically compare two methods of incorporating sound within your Flash movie • The MP3 sound track that the library have given you is poor quality, discuss the reasons that may have caused this during the digitisation process Multimedia

40. Activity • Play a CD • Copy a portion of a track • Experiment with sound effects • Add fades • Export 30 secs as an mp3 • Make a media dictionary entry • Add to a Flash movie • Insert controls using Behaviours Multimedia

41. For Next Week • Accessibility • Publishing and distribution • Independent study • Attend the supported session to get to know Flash • Create accessible Flash • Content Publishing Multimedia

42. References • Chapman N and Chapman J (2004). Digital Multimedia, Second Edition. London. Wiley • Flash Help and How Do I • Kay M. (2003) The Web Wizard's Guide to Flash, Pearson Education, USA Multimedia