Technical Issues:Sound • While graphics and animation can bring a game world to life, it has been said that sound and music give the game its soul. • A game world without audio would seem rather hollow and empty in comparison. • Audio can make its appearance in games in a variety of different forms: • Sound effects. • Voice and speech effects. • Musical score.
Technical Issues:Sound Screen shot for Mad Maestro! It is one of many new games where thefocus of the game is on sound and music itself.
Sound Requirements for Video Games • Sound and music are used to provide the audio component of representation in video games. • They are used to give an indication of what the game world and its inhabitants sound like, and usually much more. • Unfortunately, this is something many game developers often neglect. • Most new developers ignore sound and music entirely in their first game or two.
Sound Requirements:Set the Mood • Perhaps even more so than visual effects, sound and music can set the tone and mood for a video game. • It can be anything from light and cheerful to dark and foreboding. • The tempo of music can also influence or reflect the pace of action in the game. • If done properly, audio effects can support storytelling and help immerse the player at the same time.
Sound Requirements:Set the Mood Screen shot of the Legend of Zelda: Ocarina of Time. The music throughout the game helps to establish the tone and mood.
Sound Requirements:Provide Feedback • Just like visual effects, sound effects should provide some kind of audible feedback for every user action. • Such feedback should indicate the success or failure of an action. • A sound effect for a successful action should appear natural for such a completion. • A sound effect for a failure can either be natural, or some kind of beep or buzzer. • For example, shooting a gun should make a bang if you have ammunition, or a click if you do not.
Sound Requirements: Provide Feedback Screen shot from Oni. Oni provides good audio feedback to the user. Everyaction has a audible result. In this case, for example, a successful punch hasa “smack” sound effect, and a missed punch has a “whiff” sound effect.
Sound Requirements:Provide Cues to Players • Sound effects should be used to provide cues of future events to the player. • For example, hearing footsteps coming down a hall as a prelude to someone entering a room. • Music can do the same, by changing its tone to reflect the mood of events. • For example, the pace of music can pick upor turn foreboding when conflict is about to occur.
Sound Requirements:Provide Cues to Players Screen shot from Splinter Cell. Sound cues are given throughout the gamethrough ambient sounds, whispers of overheard conversations, andchanges in the background music of the game.
Sound Requirements:Be Realistic • Most game audio effects should sound as realistic and fitting as possible. • This is interesting, because graphics, animation, and game physics can range from realistic to outlandish and still be fitting. • The audio effects in place, however, must match and fit the visual effects or else the game seems inconsistent. • If you cannot generate a suitable effect, then sample one from the real world!
Sound Requirements:Be Realistic Screen shot from Doom II. Even though you are fighting out-of-this-worldcreatures with out-of-this-world weaponry, the sounds seem tofit the situation in a realistic fashion.
Sound Requirements:Pick the Right Talent • Whether you are picking actors for voice overs or musicians to provide the soundtrack for a game, make sure you select the right people for the job. • They have the right sound for the game. • They can do it on time. • They fit within budget. • They are reliable enough to stick through the entire job from start to finish. • They will provide you the necessary rights for the game when the job is done.
Sound Requirements:Pick the Right Talent Screen shot from Prisoner of War. This allegedly has some of the worstvoice acting ever heard in a game. Apparently, those German accents were really, really bad!
Sound Requirements:Create a Living World • It is important to put ambient sound effects into a game to help bring life to it. • This can include many things: • Conversations between non player characters. • Animal life in the world (even if it is not seen). • Environment effects such as wind and rain. • Terrain effects such as water. • Having these audio effects makes the game seem considerably richer. • The player feels they are in a living world.
Sound Requirements:Create a Living World Screen shot from the Legend of Zelda: The Ocarina of Time. The world is full of life and sound. Whether it is sound of chickens, a babbling brook, or creatures at night, it has considerable ambience.
Sound Requirements:Get Timing Right • The human ear is an incredibly sensitive piece of equipment. • Most people can pick up on problems with audio faster than they can with graphics or video. • Consequently, it is very important to make sure that audio is timed correctly. • Voice must be in sync with characters. • Sound effects must be in sync with the corresponding actions. • There should no hiccups, pauses, popping, or glitches in sound as it is delivered!
Sound Requirements:Allow Customization • You should allow users as much customization as possible with game audio effects. • Audio volume (split into master, effects, voice, and music volumes if possible). • Audio quality (sample rate). • Number of channels (mono, stereo, or 3D). • Music tracks to play (either choosing from those provided with the game, or user supplied tracks). Note that it does not always make sense to allow this last one.
Sound Requirements:Allow Customization Screen shot from Grand Theft Auto. Not only could you tune audio volume, but you could listen from a variety of game provided radio stationson CD, or insert a CD of your own favourite music instead.
Audio Techniques • There are a variety of techniques for providing audio effects in a video game. • Typically, the technique used depends on the type of audio effect: • Sound effects. • Voice and speech effects. • Musical score. • In most cases, there are two choices: generating the effect, or playing back a sampled and stored effect.
Sound Effects • All sounds arrive at our ears in the form of some kind of wave. • Thus, for a sound effect in a game, we have essentially two choices: • Generate a sound with the appropriate waveform that gives the effect that we are looking for. • Pick a sound from the real world close to what is wanted, sample its waveform, and store the result in game playback. • Both approaches can be quite useful, and have their advantages and disadvantages.
Sound Effects:Generation • Generating a sound effect involves selecting from basic waveforms, combining them together, applying an envelope, and sending the result to the sound device. • There are many waveforms to choose from: • Sine, cosine, tangent, square, triangle, sawtooth, various noises, and so on. • They can be combined in many ways: • Additively, subtractively, consecutively, piecewise, and so on.
Sound Effects:Generation Samples of the basic sound waves. (Top row, left to right: sine, cosine,tangent, and white noise; bottom row, left to right: triangle, squareand sawtooth.)
Sound Effects:Generation • An envelope is a simple technique used to alter the pitch or volume of a sound wave form: • Attack: The time from the start of a sound to reach its maximum intensity. • Hold: The time the sound is at maximum intensity. • Decay: The time the sound drops to a lower intensity. • Sustain: The time at this lower intensity until release. • Release: When the sound fades to nothing. • Envelopes can introduce a variety of interesting effects to a generated sound.
Sound Effects:Generation Attack Hold Decay Sustain Release A sample envelope.
Sound Effects:Generation • Advantages: • No storage space required for sound samples. • Since effects are generated, it is easy to have variation in the effects produced. • Conceptually easy to do. • Disadvantages: • Can take time to find the desired sound. • Can be hard to produce realistic noises (though the original Nintendo could technically only create sine, square, and noise waves, and it didn’t suffer for it). • Computation power required to generate the effects on the fly; this is a problem if resources are scarce.
Sound Effects:Generation Screen shot from WaveGen. A nice and simple waveform generatorto demonstrate different sound generation issues.
Sound Effects:Sampling and Storing • The basic premise behind this approach is that a sound waveform is read at periodic intervals; this is known as sampling. • If a sufficient number of samples is taken, the collection of readings will closely approximate the original waveform. • So, if these samples are stored in a file of some kind, they can be played back in the future to reproduce the original waveform and the original sound as well.
Sound Effects:Sampling and Storing A sample waveform sampled at regular intervals.
Sound Effects:Sampling and Storing • Various factors affect the quality of the sampling process: • Sampling frequency. The more frequent the sampling, the better the approximation you have to the original waveform. Values tend to range from 8000 Hz (phone grade) to 44100 Hz (CD quality). • Sampling bits. The more bits used to store a sample, the higher the granularity of individual samples. Values tend to be either 8 or 16 bits. • Number of channels. This is typically either 1 for a mono recording or 2 for a stereo recording. (More channels can be used for environmental or surround audio, but this process can also be done differently.)
Sound Effects:Sampling and Storing • A variety of formats can be used for storing sound effects: • WAV: Microsoft’s standard sound format. • AU: Sun Microsystems sound format. • AIFF: Apple/SGI sound format. • SND: NeXT sound format. • A variety of various raw formats. • Sometimes, sound effects are stored in MP3, Ogg Vorbis, or CD format as well. • In many cases, these formats use some kind of compression to reduce the storage requirements of recorded samples.
Car horn Cat meow Door slam Elephant Explosion Glass breaking Machine gun Modem Phone ring Pig squeal Pour a drink Splash Thunder Traffic jam Typewriter Vacuum cleaner Sound Effects:Sampling and Storing
Voice Effects • Voice effects, like all sounds, can be represented as waveforms. • Like sound effects, voice effects can be either generated or sampled and stored. • Sampling and storing voice effects can be done in the same way as other sounds. • Generating realistic sounding voice effects, however, is still an incredibly difficult task to do properly.
Voice Effects:Sampling and Storing • Stored voice effects can be treated essentially the same as stored sounds. • The difference, in this case, is that the sound is being created by an actor reading a script, and not through some other means. • For clarity, it is typically a good idea to use the best sampling techniques for all game speech. • Examples: • Humans from Warcraft I & II. • Orcs from Warcraft I & II.
Voice Effects:Generation • With sampled voice effects, a game is ultimately limited in its speech options. • There is only a finite amount of storage space, so only a limited number of voice effects can be supported. • If more flexibility is required, with speech chosen at run-time, some kind of generation technique is required. • This can be computationally expensive, and difficult to obtain realistic results.
Voice Effects:Generation • To generate a voice effect, the game must do the following basic steps: • Decide upon the text to be spoken. • Select a voice in which the text will be spoken, and various voice characteristics. • Phonetically break the text into its basic phonemes, emphases, pauses, and other components. • Piece together recorded samples for these phonetic components and smooth over any seams between adjacent samples. • Output the results.
Voice Effects:Generation • Modern speech generation has shown many improvements. • Microsoft TTS kits in many languages. • AT&T and Bell Labs research with naturalsounding voices in many languages. • Many, many more … this is active research! • Some examples from over the years: AT&T Natural Voices C64 SAM 1980’s Present
Musical Score • Music, like all sound, can also be represented as a waveform. • Like sound effects, music can be either generated or sampled and stored. • Sampling and storing music can be done in the same way as other sounds. • Generating music can be done without too much difficulty with some realistic effects.
Musical Score:Sampling and Storing • Music for video games can be sampled and stored much in the same way as sound effects. • Typically, one of the following formats is used for doing so: • CD audio (an audio track on the game CD) • MP3 (MPEG audio) • Ogg Vorbis (an open, free alternative) • WAV (Microsoft audio format) • An example (from Grand Theft Auto):
Musical Score:Generation • There are two well supported approaches to sequencing collections of sounds to produce music in video games. • MIDI (Musical Instrument Digital Interface) • Allows multiple channels of computer generated sounds to be composed and played sequentially. Sounds tend to be tones from musical instruments. • MODules • Allows sequencing a series of sounds to assemble them into a tune, with a variety of special effects applied along the way. Sounds can be anything.
A Final Word:Environmental Audio • Original games only had mono output to a single speaker. • The next advancement in games was stereo sound to two speakers. • Audio is separated into a left and right channel to provide some sense of direction as to where the sounds originated from. • For modern gaming, significant work has been put into environmental audio, also known as “surround sound”.
A Final Word:Environmental Audio • There are two main techniques to environmental audio: • Use a variety of techniques to provide depth in sound using only two speakers. • Use four or more speakers and directing audio appropriately. • For best effects, you must take level design and game physics into consideration. • Echoes, reverberations, dampening, and so on. • Blocking of sounds caused by obstacles or terrain. • And so on … you must now “render” your audio!
A Final Word:Environmental Audio Speaker placement for a Dolby 5.1 surround sound system. You get a frontcentre channel, front left and right channels, rear left and right channels, and a subwoofer (for non-directional bass). Newer technologies are emerging …