Audio for Games

1. Audio for Games Scott Selfon Xbox Advanced Technology Group Microsoft Confidential

2. Overview • The roles of audio in games • Audio challenges on a game console • Score, Sound Effects, and Dialog • Challenges • Typical Solutions • Content creation / implementation process • Audio integration and end game • Case studies on Xbox • Leading edge / future of game audio • Q & A Microsoft Confidential

3. Game Audio Has Come a Long Way… • Fidelity • Tinny mono speaker  5.1 home theater systems • 1 channel/4 bits  256 channels, 24 bits • FM oscillators  48 kHz wavetable synthesis • Audio processing power • Off-line effects processing  real-time DSP effects • 4-channel mixing  Realtime Dolby Digital encode • Realtime 3D positioning • Production pipeline • Low level chip programming  Separate sound designer/composer and audio programmer Microsoft Confidential

4. The Roles of Audio in Games • Audio is 50% of the experience • Support (improve?) the visuals • Provide information about what we don’t see • Provide emotional support for gameplay • Rewards/encouragement • The best audio implementations are rarely noticed by the gamer • 1995 MIT study • Game reviews and audio Microsoft Confidential

5. Audio Challenges on a Game Console: Aesthetic • Competing with other ‘living room’ experiences • But much less linear • Events may be skipped, repeat, play in unpredictable order • Players expect same kind of realism as they get from non-interactive media Microsoft Confidential

6. Audio Challenges on a Game Console: Technical • Finite Resources • Storage space (not as much of an issue currently) • Memory (actual playing sound data, audio library size, etc.) • Processing power (realtime effects, mixing, etc.) • Interactivity • Avoiding repetition is perhaps most important • Secondary goal: compelling non-linear audio • Need solutions that don’t require a lot of development time Microsoft Confidential

7. Audio Challenges on a Game Console: Organizational • “Post-Production” mentality towards audio – scheduled late • Often appropriate in film, television • Often viewed in budgeting as expense rather than investment • Tradition & expectations for game audio - hasn’t been always been of highest quality (see previous slide) • No dedicated audio programming resource • Often up to composer/sound designer to understand and determine what’s possible and feasible • Cross-platform audio solutions • Simplest-to-implement programmatic solutions used Microsoft Confidential

8. Film/TV/Radio Sound Designer Composer/arranger(s) Orchestrator(s) Recording Engineer(s) Mixing/Mastering Engineer(s) Music Supervisor Music Editor SFX Editor ADR/Dialog Editor Foley Artist(s) Casting (for voices) Games Sound Designer/Composer (maybe 2 separate people) Audio programmer (part time) Audio Role Comparison Microsoft Confidential

9. Hiring a Content Creator • Generally ‘buy-out’ (work for hire) • Composer paid straight fee (per platform?) • Developer/publisher generally owns all rights to music • May or may not include cost of recording, musicians, mixing, etc. • May negotiate for publishing • Per copy sold royalty (rare) • Paid for TV/radio spots • Royalty for thematic use in sequels, spinoffs • Opportunity for game soundtrack CD Microsoft Confidential

10. Budgeting for Audio • Musical underscore • Live, synth, or mix? And how much? • How does it start and stop? (linear, looping, branching) • Licensing existing music • Sound Effects / Ambience • Live recordings or sample libraries? • Creation of new modeled sounds • Sub-contracting • Often essential for large projects Microsoft Confidential

11. Laying out an Attack Plan • Look at all ways system can make noise • Work with technical document to determine what’s preferable/do-able • E.g. MIDI or streamed (CD/DVD/HD) music • Real-time effects (reverb, filtering, etc.) • What other game resources are being used • Maximize audio capabilities while minimizing impact on rest of game • Often different for different parts of game Microsoft Confidential

12. Tools, Tools, Tools… • Evaluating for technical and aesthetic criteria • Make authoring and programming pipelines relatively independent • Maintain creative flexibility • Linear audio • Authoring: Same as for other media • Programming: Cross-platform and platform specific options • Non-linear audio = generally proprietary • PC: DirectMusic or ‘in-house’ engines • Consoles: Platform-specific or ‘in-house’ solutions Microsoft Confidential

13. Underscore Challenges • Film • ~90 minutes, often <60 minutes of score • Linear, unchanging storyline • Game • 10+ hours of gameplay • Gameplay events may be quite variable (in order, duration, etc.) • Don’t need “music” everywhere (but at least ambience) • Score: Often linear tracks several minutes long (may loop) Microsoft Confidential

14. Typical Underscore Solutions for Consoles • Create several long linear tracks • May loop • May be played ‘soundtrack’ style (no interaction with gameplay) • Save purely musical score for key one-time game points • Use ambience instead to set mood • Avoids repetition of recognizable thematic material • Advanced: Non-linear audio • Alternate versions with looping and branching • Various intensity levels of music that crossfade based on game state Microsoft Confidential

15. Licensed Music • Not appropriate for all games • Often difficult to make interactive/dynamic • Disconnects score from moment-to-moment gameplay • Can be effective in some situations • Name recognition/cross-marketing potential • Can set titles up for user soundtrack support • Typical uses • Existing brands/licenses (esp. movies and TV) • Heavily used by racing games Microsoft Confidential

16. Composing a Game Score • Spotting session • Where can music support action? • Compose cues for linear sections of game (for instance, cutscenes) • Need final cut for accurate timings (hah hah) • Record and mix • Pre-rendered: Synth, live players, mixture • Real-time rendered: MIDI, individual pads, etc. • Implement playback in game • Triggers to start, stop, transition, etc. • Edit to match ‘final’ animations Microsoft Confidential

17. Sound Effects Challenges • No pre-existing soundtrack to work with • Same sound event may be heard hundreds of times • Variation • Environmental interaction • Tying FX to game physics and AI • Sheer number of sound elements to create • Combinations and permutations • Everything that can make a noise should Microsoft Confidential

18. Typical Sound Effects Solutions • Variation • Implement basic pitch randomization • Several versions of each wave • Break a sound into components and record versions of each of those • Tying effects to game physics / AI • Implementation varies per-title • Often vast majority of dev time spent here • Advanced: Dynamic processed sounds • Environmental effects (reverb) • Filtering (force of impacts, radioizing, etc.) Microsoft Confidential

19. Creating Ambience and Sound Effects • Spotting session (if game playable), or look at artwork (environments, objects) • List everything that can make noise, prioritize • Object interaction matrix • Often start with organic sounds • Sound effects library ($$$, can be generic) • Create own samples (free, unique) • Sound processing (pre-rendered or dynamic) • Reverb, filtering, chorus, effects Microsoft Confidential

20. Dialog Challenges • Casting – producer may want involvement • Directing / Coaching • Stitching (first | and | 10 | from the | 25) • Dialog repetition is easily recognized • Localization • Re-recording voice talent • Engine integration • Grammatical differences • Recreating pre-rendered processing Microsoft Confidential

21. Typical Dialog Solutions • Triggers for dialog (just like music) • Variation for common lines (often too few) • Stitching – coach the talent • Localization • Solutions vary per-title • Very careful asset management • Sometimes source assets can just be swapped • Advanced: Sophisticated dialog engine • Variation tracking to avoid repetition • People don’t always have to say anything Microsoft Confidential

22. Prepping for Dialog • Content creator works with producer, game designer, and/or writer to: • List and describe characters • Write each character’s scripts • Cast person for each character • Real voice talent makes a difference • Many actors can do multiple roles Microsoft Confidential

23. Dialog Production • Record all dialog • Encourage improvisation • Coaching for creating convincing stitchable dialog • Cut up dialog into separate files • Clean up for stitching – sound designer for sports title can spend majority of time here • Post-recording processing • Document everything for localization Microsoft Confidential

24. Squeezing it all into 5 bytes • Best to determine resources up front • Memory and CPU available • Disk space and bandwidth available • Inevitably, not enough room • Asset swapping (programming time) • Compression • Use of streaming where possible • Cut assets Microsoft Confidential

25. And the Handoff… • Source assets given to game • Typically high fidelity + compressed versions delivered • Freelance composer/sound designer: often end of work • Programmer integrates audio assets into title • Often tasked with mixing (either with or without content creator feedback) • Implement runtime sound processing (reverb, effects, filtering, etc.) unless engine natively supports • Testers confirm sounds are in place • Asset/trigger lists, scripts, and/or manual triggers • Variation testing? Microsoft Confidential

26. After the Fact: More Challenges • Archive, archive, archive • Create a recording of typical gameplay for reference • Disaster always strikes at the 11th hour • Keep an organized collection of any sounds created (spreadsheet, database) • Localization • Voice talent recasting, re-recording • Recreate processing (unless real-time used) • Releasing a CD Microsoft Confidential

27. Audio Case Study:Project Gotham Racing • Stereo ambience, with some 3D positioned foreground ambient elements • Player’s car • Stereo engine ambience • Engine sound varies with RPM, velocity, physics, etc. • Tire sound varies with surface (sidewalk, stairs, street, etc.) • 3D positioning on other cars • Engines (varies with RPM, velocity, physics) • Car horns • Cornering, tire squeals, collisions, etc. (all with some variation) • Dynamic sounds • Tire squeals (randomized between several versions and pitches) • Collisions (depend on speed, material involved, etc.) Microsoft Confidential

28. Project Gotham Racing: Music • Game disc-supplied licensed songs (WMA) • User Soundtrack support • Allows for personalized driving music mixes • Music uniquely integrated into game • Use of ‘car radio’ or ‘CD’ playback • ‘Radio’: Location-specific DJ interstitials Microsoft Confidential

29. Audio Case Study: Splinter Cell • Detailed ambiences, foley, sound effects • Audio as a gameplay element (stealth) • Stealth: Noise tied to game AI • Footsteps, interaction with objects, weapons • Audio as an informational tool • Interactive musical score • Tells player when enemies alerted • Use of musically appropriate boundaries for transitions Microsoft Confidential

30. Audio Case Study: Halo • 4,311 triggerable events (soundtags) • 11,629 source waves (soundfiles) • 2.5 GB uncompressed audio • 22 kHz mono and 44 kHz stereo • 80 minutes of source music • 20+ hours of gameplay • 12 voice talents, 7,735 lines of dialog • …all in 2-MB memory footprint Microsoft Confidential

31. Halo: Audio Tools Used • Pre-rendered linear music/FX = traditional standard audio post production tools • Pro Tools, Studio Vision, Peak, Digital Performer • Dynamic audio = custom tool/engine, randomization/permutation for each event • Pick one or more waves to play • Random variation in pitch • Weighting for how often a sound will be used Microsoft Confidential

32. Halo Audio Engine: Dialog • Simulation of actual speech patterns • Close connection to game AI • Large number of events a character can respond to, multiple phrases per action • Characters don’t always speak • Tracking speech • Multiple voice talents for multiple characters • Character only says one thing at a time • Positioning (3D versus locked-to-speaker) • Repetition avoidance Microsoft Confidential

33. Halo Audio Engine: Sound Effects • Variations in all sound effects • Weighting (frequency for multiple versions of sound) • Randomized pitch control • Variations in ambience • Various loops of different lengths • Layered 3D-positioned detail sounds (birds, crickets, machinery hum, etc.) • “Cascading” sound effects • If many instances of an effect are playing, play a single more complex sound • Response to game environment • Specific conditions/events/game physics • Environment interaction (reverb) • 3D spatial positioning versus static Microsoft Confidential

34. Halo Audio Engine: Music • Typically linear or looping • If looped, alternate versions with varying lengths • Dynamic execution (“spotted” with programmer) • Varying transition types based on situation • Branching • Crossfades • Game event tags • Appropriate absence of music • Silence is golden - not scored wall to wall • Timeouts Microsoft Confidential

35. Halo 5.1 Usage • Sound effects typically 3D-positioned (4 channels) • Some ambience 3D-positioned as well • “Quad” music and ambience • Doubled stereo (fronts 100% / rears 50%) • Crossfades based on location • Center channel • Cortana (“first person” dialog) • Radio communication • LFE channel • Used to emphasize more massive sounds • Heartbeat (low health) • Explosions Microsoft Confidential

36. Halo Environmental Reverb Usage • Reverb settings changed based on environment • Indoors versus outside • Confined spaces versus large spaces • Caves versus hallways • Sound interactions with reverb • Especially 3D-positioned sound effects • Some ambient elements also Microsoft Confidential

37. Halo Implementation (1) • 2-MB in-memory footprint • 22-kHz mono ADPCM for 3D sounds • 44-kHz stereo ADPCM for music + ambience • ADPCM = ~3.5:1 compression • Use of Xbox hard disk • Copy all sound data to drive at start of level (faster to read than DVD) • Pull small chunks into memory rather than entire banks (pseudo-”streaming”) • Drive persistence = faster load next time Microsoft Confidential

38. Halo Implementation (2) • Sound during load screens • Small looped sample • Multiple instances played with pitch shifting and filtering as load progresses • “Dynamic” – speed and actual sound vary with read time Microsoft Confidential

39. Leading Edge Audio Challenges • Voice integration (esp. for online titles) • Chat and gameplay implications • Manipulation (masking, radioizing, etc.) • Voice recognition • Command and control • Further multichannel audio usage • Fuller 5.1 implementations • Dynamic audio mixing • Physical modeling • Still too expensive (CPU, programming effort) for most games Microsoft Confidential

40. Wrap Up:The State of Game Audio • Lots of independent composers/sound designers • Many coming from outside game industry • G.A.N.G. (www.audiogang.com) • Still typical for developers to reinvent audio engine with each title (newer tools try to solve) • Xbox efforts • Xbox Registered Developer and Registered Content Creator Programs • Tools and dedicated support for audio content creators and programmers • Game Designer/Producer education Microsoft Confidential

41. Q & A • scottsel@xbox.com • Resources • G.A.N.G. (www.audiogang.com) • Game Audio Pro e-mail list (Yahoo) • DirectMusic (www.microsoft.com/directx) • 2003 Game Developer Conference lectures Microsoft Confidential