Chris Grigg Beatnik Inc. Chairman, MMA Tech Board chrisg @ beatnik Tom Savell

1. New MIDI-Based Game Audio Techniques Chris Grigg Beatnik Inc. Chairman, MMA Tech Board chrisg@beatnik.com Tom Savell Creative Labs – Advanced Technology Center MMA Tech Board tcs@atc.creative.com

2. Session Overview What’s MIDI? • The Technology • The Standards Orgs The New MIDI Standards • iXMF: Interactive XMF • 3D MIDI Controllers • High Definition Protocolfor MIDI Devices Questions

3. What’s MIDI? What’s MIDI?

4. MIDI: The Technology • MIDI 1.0 over 5-pin DIN • The Underlying Models • The Recommended PracticesGM, Show Control, MTC, MMC, GM2, SP-MIDI, etc. • The File FormatsSMF, Extended RMID, XMF, Mobile XMF • Support in Computer OS’sMac OS, Windows, BeOS, Linux etc. • Support in Other Media StandardsMPEG-4 etc. • Support in Audio Plug-In API’sVST, DX, AU etc. • Support in Mobile Phones3GPP etc.

5. MIDI: The Standards Orgs • MMA: MIDI Manufacturers Association (world except Japan) • Tech Board • Working Groups: Mobile WG XMF WG DLS WG Transport Layer WG High-Def WG • IASIG: Interactive Audio SIG • AMEI: Assoc. of Musical Electronic Industry (Japan only)

6. The New MIDI Standards The New MIDIStandards

7. iXMF: Interactive XMF 1iXMF:InteractiveXMF

8. iXMF: Overview • First open, non-proprietary standard for a file format for interactive audio content • Uses MMA’s XMF container technology • Allows game developers to more easily create cross-platform titles • Makes it easier for audio professionals to develop interactive content • Developed by the iXMF-WG inside IASIG • V1.0 spec about to be published • Prominent early adopter

9. iXMF: Motivation (1) There IsNoStandard What’s theIndustryStandard? BuildProprietaryTool Do ItRight NextTime LimitedTime & $for Tool Break the‘Depressing Circle’! LimitedToolQuality :-( LimitedContentQuality Harderto MakeContent

10. iXMF: Motivation (1) There IsNoStandard What’s theIndustryStandard? BuildProprietaryTool Do ItRight NextTime LimitedTime & $ LimitedToolQuality :-( LimitedContentQuality Harderto MakeContent

11. iXMF: Motivation (2) • Many companies continually re-creating the wheel – in-house / proprietary • Much of what they do is the same – only the format is different • This is redundant, parallel work –> no synergies • User base (+budget) is very small, so tool quality improves slowly (at best) • Tool quality tends to limit IA content quality • IA work done on one platform doesn’t port to any other platform--> Re-do for every platform--> IA tends to be seen as cost-prohibitive--> Less IA gets used

12. iXMF: Process (Standardization) • A Project BBQ idea • An IASIG Working Group • Linda Law is Chair • Chris Grigg is Technical Lead • Long Requirements phase • Draft Specification • Public Review period ended in May 2008 • Final edits to v1.0 spec under way now • Release of v1.0 spec expected soon (October?)

13. iXMF: Technical Design (1) The iXMF File • Self-Contained IA Content Module • Includes both Media and Adaptive Behaviors

14. iXMF: Technical Design (2) Runtime Architecture • iXMF File feeds IA Engine Game Middleware Platform Provided

15. iXMF: Technical Design (3) Cues– soundtrack element objects • iXMF file contains zero or more Cues • Cue = one soundtrack element –a piece of music, a sfx, or a change to music/sfx • Game requests sound/music element by starting a Cue: • by name: startCue( “mainTitlesMusic” ); • by number: startCue( kMainTitlesMusic ); • In iXMF file, Cue contains (or references): • zero or more media Chunks • some Scripts

16. iXMF: Technical Design (4) Scripts– methods / events of Cues • Stored in iXMF file in portable binary format • Variables –iXMF Global / Cue class / Cue instance • Expressions –all the C operators • User Subroutines –call another user Script • Built-In Statements • Generic programming stuff • Specialized IA primitives, to minimize the need for programming • Extension Statements • No user-defined Statements! –Keep it simple • Tools can hide Scripts behind GUI

17. iXMF: Technical Design (5) Automatic Behaviors(“Engine Model”) • Script hooks: • IxmfFileLoad, IxmfFileUnload • CueStart, CueStop, CueCancel • CommitPoint, ChangeOver • Markers can call Scripts • NextChunk chaining • Mix Transitions – crossfade, sync points/rules • SyncGroups • MixGroups • Media LRU tracking • Scripts can invoke game Callbacks • Game can read/write iXMF variables

18. iXMF: Technical Design (6) Custom Extensibility: • Add Script commands • “Opcode Spaces” • Add custom media formats: • Platform-specific audio formats • Generative music • Even video, graphics • Extension Area in every iXMF data structure • Community Extensibility: • Future Profiles of the v1.0 standard • Future versions of the iXMF standard

19. iXMF: Technical Design (7) Portability • Cues & Scripts will work properly on any iXMF player IF: • The media formats used are supported AND • ( No extensions are used by content OR Any extensions used are supported OR Any extensions used aren’t critical ) • Profiles can manage portability expectations • Content is often mastered for specific platforms – portability not an issue there • A high-res iXMF “source” file can give authoring-time portability • Export to platform-specific runtime formats

20. iXMF: The Ecosystem • iXMF is non-proprietary • iXMF is royalty-free to use or implement • So: Expect multiple runtime implementations, platforms, authoring tools --> Ecosystem • Commercial --> Competition --> Improvement • Expect interoperability for base features / profiles • Don’t expect custom extensions to be portable • Look for integration with game dev / media mgmt databases • Hope that the IA community can start speaking a common language • Expect vendors & IASIG to be responsive • Expect IA to become the norm, not exotic

21. iXMF: The Future V1.0 Spec releasing soon. Then: • New Versions • New Features • New Media Types • Profiles for specialization as needed – e.g.: • Advanced Games • Simple/Casual Games • Mobile Phones/Games • Musical Instruments • DJ Applications • Location-Based Entertainment • Little-Endian Join the iXMF Working Groupto participate!

22. 3D MIDI Controllers 2 3D MIDIControllers

23. 3D MIDI: Overview • Provides set of controllers for independent positioning on each MIDI channel • Uses spherical coordinate system: azimuth angle, elevation angle, and distance • Seamless integration with legacy pan control • Egocentric approach describes head-related 3D sources • Well-defined standardized behavior • Extensible through MMA standardization process • V1.0 to be published soon.

24. 3D MIDI: Position Controllers • Registered Parameter Number (RPN) Bank 0x3D • Azimuth and Elevation Angular Resolution of 0.02 degrees from -180 to +179.98 degrees • Distance represented as ratio of distance to a maximum distance: e.g. 0.000061 is 6.1 centimeters if max. distance is 1km.

25. 3D MIDI: Pan Spread and Roll Angle • Pan Spread defines the 3D angular distance of the MIDI stereo field represented by the legacy pan controller • Roll Angle defines the angle relative to horizontal of the MIDI stereo field represented by the legacy pan controller

26. 3D MIDI: Position Model

27. 3D MIDI: Gain Parameters • Gain parameters provide content author control • Gain • Independent of CC#7/11 • Gain at max distance • Gain when distance ratio is 1.0 • Reference distance • No additional gain below this distance ratio

28. attenuation 0 dB max_attenuation distance reference_distance max_distance 3D MIDI: Distance Model

29. 3D MIDI: Ex. Distance of a Fly • The buzz of a fly would typically not be heard beyond 10 meters away • maximum_distance = 10 • Sounds significantly louder a few centimeters away from your ear • reference_distance_ratio = 0.01 -- between 0 and 10 cm, the fly sound is not attenuated

30. 3D MIDI: Future Enhancements • Reverberation Controls • Source / Reverb mix dependency on distance. • Environmental effects (e.g. occlusion) • Could be made compatible with IA-SIG I3DL2 guideline • Subject to MMA Standardization Process

31. High Definition Protocol for MIDI Devices 3 HD ProtocolforMIDI Devices

32. HD-PMD: Introduction • Whole new generation of the basic MIDI message protocol • Address shortcomingsof the MIDI 1.0 protocol (1983) • Grow the worldwide MIDI marketsby opening up new functionality,new musical expressiveness • HD work started in 2006 by MMA Tech Board • Standardization work still under way • Overview Only! All Technical Details Subject to Change!

33. HD-PMD: Motivation MIDI 1.0 Technology Needs an Update: • MIDI 1.0 first released in 1983 • HW transport slow / Timing not tight • Complicated msg structure / Parsing: • Many multi-message functions • Message size varies • Byte aligned • Hardware orientation --> Hard to write SW for • Advanced features difficult to access • Expansion room exhausted long ago Desired New Functionality Includes: • Better support for non-keyboard instrument models (strings, voices, horns, winds etc.) • Better support for non-western music • Better support for instruments with continuous pitch / very fine pitch nuance

34. HD-PMD: Standards Process • MMA Working Group in US/Europe • AMEI Working Group in Japan • Joint & Parallel technical work • Weekly WG teleconferences • WG email reflector • MMA/AMEI coordination via email • MMA/AMEI Joint Face-to-Face meetingsin LA / SF / Tokyo 2-3x per year

35. HD-PMD: Technical Design • Transports: An ongoing discussion – • Traditional wired transports • Software-only transports: • Network/comm protocols • Plug-In APIs • OS MIDI services APIs • Messages • Significantly reconfigured • Simplicity • Future extensibility • New musical functionality

36. HD-PMD: Messages • Uniform Packet 64/128 bits (8/16 bytes)Simplifies transports - Memory efficient(MIDI 1.0 has 1, 2, 3, or N-byte messages) • Message = 1 or More Packets • Single-Packet Messages (SPM) • Multi-Packet Messages (MPM) • Fixed # of Packets • Variable # of Packets (VLM)

37. HD-PMD:Opcode SpacesCompared • MIDI 1.0 • Totally full. • Every “Status Nybble”either in use, or else officially deprecated • Extensions very cumbersome • HD is shown at right • Lots of expansion room • Especially with sub-commands

38. HD-PMD: New Functionality(Proposed – Under Discussion) Notes: • More Velocity resolution (16-bit – 512x MIDI 1.0 depth) • Precise Pitch option(1/256 semitone resolution) • Trigger Index option(like MIDI 1.0 Note Number) Controllers: • More Value resolution (16/32-bit – 2-4x MIDI 1.0 depth) • More Controller Numbers available (32x MIDI 1.0 space) • No more MSB / LSB / Data Entry! • CC is a unified message, including Bank number • RPN now just another controller number • NRPN now a unified message–all MIDI 1.0 numbers available • Per-Channel and Per-Note Controllers, RPNs, and NRPNs Addressing: • More Channels (7 bits - 8x MIDI 1.0 space) • Groups concept - Add’l dimension of addressing(5 bits - effectively 256x MIDI 1.0 channel space)

39. HD-PMD: What’s Next? • More joint MMA/AMEI Meetings • Draft v1.0 Specification (English) is in progress • Experimental implementations of candidate transports ongoing • Decide on v1.0 Spec(s) release date(s) • Decide on possible additional Recommended Practices for HD After v1.0 Spec Release: • Continuing process to standardize new features…forever…and ever…and ever…

40. Questions? Questions?

41. Answers! • YES you can join the iXMF Working Group… • YES you can get a copy of the v1.0 iXMF Spec soon… …at www.iasig.org • YES you can get a copy of the 3D MIDI Spec soon… • YESif you are commercially involved in designing or producing MIDI HW or SW you can join the MMA and the MIDI High Def Working Group… …at www.midi.org

42. (You just sat through)New MIDI-Based Game Audio Techniques Chris Grigg Beatnik Inc. Chairman, MMA Tech Board chrisg@beatnik.com Tom Savell Creative Labs – Advanced Technology Center MMA Tech Board tcs@atc.creative.com