1 / 45

Practical techniques for managing Code and Assets in multiple cross platform titles.

Learn effective strategies for managing code and assets in video game development, including asset pipelines, version control, converters, and more.

jswan
Download Presentation

Practical techniques for managing Code and Assets in multiple cross platform titles.

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Practical techniques for managing Code and Assets in multiple cross platform titles. Julien Koenen Senior Programmer at keen games j.koenen@keengames.com

  2. Keen games • Video game developer based in Frankfurt (Germany) • Creating video games for over 17 years • Successfully developed more than 25 titles on 8 platforms

  3. Contents • Asset Pipeline • Exporters and Converters • Overview of our asset pipeline • Different Versions of your product • Code Build Infrastructure • Motivation • Requirements • Our Solution • Q&A

  4. Asset Pipeline - Motivation • Get the Data into your game • Version control / Work flow • Convert the Data into optimized target format • Optimize for the specific platform • Validation / error checking • Check for errors / problems with an asset • Bone count, Texture aspect ratio, Polygon count, File size, … • Manage multiple versions of the same Project

  5. Project Versions • There are multiple dimensions in our project version space: • Target platform (psp,nds,wii,xbox360,ps3,...) • Language/Region (eu,us,jp,...) • Distribution Channel (download,disc) • Target Audience (e3_demo, gdc_demo, release) • The version of a product is defined by a vector in this multi-dimensional space • Version = (psp,eu,disc,release)

  6. Getting the Data into the Game • The first stage is the DCC Tool and getting your data out of there. • Basically two options: • Use the storage format directly • Examples: Photoshop, OpenOffice, Sound files, Videos, ... • Export the data into an intermediate transport format • Most 3D Data (COLLADA is not usable as a storage format yet), Animation Data

  7. Try to get rid of the transfer format! • It’s always a problem to keep the storage and the transfer data in sync • One option is to embed the exported data into the storage data • For example embed your exported 3D data into a 3D Studio MAX file chunk. • Or the other way around: • Embed the .bip (character studio animation) in your exported animation file

  8. If you need an exporter • Don't ever put any options in the exporter UI! • The user of the exporter is (typically) an artist… • Most options are platform specific and can't/shouldn't be given at export time • The options are typically not saved • Last but not least: It's a nightmare to add or change an important option late in production (“Could you please re-export ALL the animations now?”)

  9. If you need an exporter • Don't export directly into your runtime format • Export everything that you might need on ANY platform • Keep the format backwards and forwards compatible • Don't ever break the transport file format! • Have shared code to read/write the transport format and encapsulate the file format specifics

  10. Use Converters ! • Convert everything! • The Converter reacts on the components of the version vector • Converter + Input Data + Version → Runtime Data • Examples: • Endianness • Optimize rendering data • Check platform capabilities / restrictions • Check distribution restrictions • Select input data (language specific)

  11. Converters – Advantages • No problems with lost parameter values • No artists asking you what they should put in those option fields – (ok, not really ;) • Easier to add/change options • Easier to add platform and project specific options / behavior • Easy to test different optimization approaches (easier to change the runtime format)

  12. Converters - Disadvantages • Potentially huge transport format • Buy larger hard drives(Could be a burden to some of the other parts of the pipeline though) • You have to actually program the converters • You have to have an infrastructure that allows for fast and easy creation of converters • Converters tend to be really re-useable across projects

  13. Overview of our Asset Pipeline • Each project has a set of jobs (one for each source asset) • All job settings and sources are stored in a central place (we use perforce for the assets) • Custom UI • Ties in with our workflow tools (different rights/roles) • Artists/Asset-creators work locally and commit their changes when they are finished

  14. Overview of our Asset Pipeline • We have converter executables and job types that reference these converters • Example: Different JobTypes for Textures (UI,Enironment,Character,…) • JobTypes are project specific • Job Types can be used to define restrictions / constraints • The job type definition forces/presets some of the parameters of the converter to common settings

  15. Overview of our Asset Pipeline • Each project has it's own copy of the converter executables in it's directory structure • Custom project-specific converters • Control the version of the converter used for a project • This is necessary to be able to archive a project in the end • Most projects have special converters that are specific to this project • map/level data converters

  16. Overview of our Asset Pipeline • Database contains list of pending (to be built) jobs • Each client pulls pending jobs, builds them locally and pushes the results on a central server • Converter executable+Job settings+Source Assets have to create the same result ALL the time on ALL machines (context-independent) • Each client copies the latest build from the server

  17. Overview of our Asset Pipeline • The converted resources are placed into the game build folder (flat file folder with all files) • There is one game build folder for each version of the project • For the disc builds we reorder and pack these files • Most platforms load directly from this directory • On others you need disc emulation/rom building first • Fast reload for some assets to support fast iteration times

  18. Job Settings • We tried to keep the settings as platform independent as possible • Specify the intent instead of the implementation Example: Texture format: We specify • color_quality (low,medium,high,very_high) and • alpha_setting (none,stencil,smooth) • and let the converter choose the correct format on each platform • You still need to have the option to overwrite the setting on a per platform basis

  19. Job Settings • There has to be a way to change settings of multiple jobs at the same time • We use the JobTypes to identify groups of jobs • A general tagging system is probably a better solution • The Settings have to be stored securely and have to be under version control • Right now we just use a SQL database for that (no versioning) • Changes in the next iteration of our tools...

  20. Assets: Summary • Don't specify options in your exporter • Use converters • Not just for optimizing! • Don't do work in the game that you could do beforehand • Use an asset build system • It's very easy to get a basic version running (And you still get most of the advantages) • Don't over-engineer!

  21. Code - Motivation • Lots of duplicated/copied code • Configuration of project settings is really annoying • Version-hell (compiler versions, sdk versions) • Setup of new workstations not simple enough • Archiving of projects is a lot of work • Bad dependency visibility

  22. Code – Platform Issues • PS3/PSP: • /usr/local/{cell,psp}/*.* → Not nice and we had some trouble getting this cleanly integrated • No separation between platform sdk and compiler • We had to separate it ourselves • Xbox 360: • Environment Variables / Visual Studio Include paths... • In general: • Please stop using Environment Variables !!!

  23. Code - Requirements • Building the code has to be 100% reproducible • Be able to define reusable modules of code • Multiple Projects shouldn't interfere with each other • The Project should be able to select versions of each module • Definition language has to be flexible enough to work in all cases • Has to be simple and natural to use

  24. Code – Reproducible Builds • Reproducible Builds are THE most important point • Without that you don’t have a build system… • The build depends on the compiler/tools, the environment, the configuration settings and the sources • Remove dependencies on the environment • Have a single version number for your code+tools • Building that version HAS to produce the same result each time (even in a year from now!)

  25. Code – Re-usable modules • Sharing code between projects • Sharing the code for reading/parsing assets • Sharing code between different tools (ui widgets) • Material Editor and Particle Editor both need Texture Preview • Systems/“Engine“ code (Collision, Sound, Rendering, Materials,...) • Share between a test project for the module and a game

  26. Code – Multiple projects • Working on multiple projects at the same time • All in different stages of the production process • With three projects at the same time you basically have a major milestone every month… • Don’t break other projects • Projects might need to use different versions of a certain third party module / platform sdk

  27. Code – Simple to use • Very important point to get the system accepted and used • Needs to be less work than the alternatives (At least not more) • Needs to be free of bugs (You really don't want to have bugs in your build system) • Needs to support the tools that are used by the developers (IDEs)

  28. Code – Our solution • There are some tools that do solve parts of our requirements but we found none that was satisfying • So we implemented a system that builds the software from a platform/compiler independent definition • Implemented in ruby, because it's a simple and available language • Yes, it's fast enough ;)

  29. Code – Our solution • Projects are defined in a project file • Project defines global settings / compiler versions • Imports modules needed in that project • Each module defines • It's own dependencies • It's configuration • No need to set include paths in your project anymore • The source code it's made off • The module can run ruby code (flexibility)

  30. Code – Our solution • The buildsystem does the dependency checking / compilation • No need for make... • Therefore you are free to use any IDE you want • We can create project files for Visual Studio (VS2003/VS2005/VS2008) and eclipse (easy to add different IDEs) • Very easy to integrate with CruiseControl • Just setup a CruiseControl server, check out the project sources and trigger the build Alternatives

  31. Why don't we use library projects? • Library projects don't include configuration settings • The user of the library has to do the configuration • Library projects are not intelligent • Our modules can react on the context, settings and options (they can execute ruby code!) • hard to create and maintain • At least harder than modules in our system! • Doesn’t help with compilers / sdks

  32. Module definition • Each module is contained in a directory with the definition file and all sources of the module • The definition file specifies the dependencies, configuration and sources of this module • Each module can import other modules • no circular-dependencies obviously

  33. Module definition • Source files are tagged to specify the compiler that should be used • Common tags are for example • '.cpp' – C++ source • 'CG_FRAGMENT_COMPILER' – Cg fragment shader • 'SPU_CPP_COMPILER' – PS3 SPU C++ Compiler • Most tags are automatically determined through the file extension

  34. Compiler modules • All the binaries of the compiler are packaged into the module • Was a bit problematic for some compilers but works for all of them: • Visual C++ 2003, 2005 and 2008 for win32 • GCC for PS3 (PPU, SPU) and PSP • Visual C++ for xbox 360 • Codewarrior for Wii and Nintendo DS • The module defines the compiler and connects it to certain tags • No need to install compilers anymore...

  35. Third party Libraries / SDKs • Third party modules (like mfc, directx, fmod, bink, maxsdk, freetype2) are (mostly) easy • Normally just a bunch of header files (set the include path) and libraries (add the libraries to the linker) • Lots of warnings though … • SDKs • Tend to be rather large... • On some platforms the compiler and platform sdk is not cleanly separable • Windows platform sdk includes (old) directx headers ...

  36. Building the Project • You can just build the project from the command line • Or you can create project files for visual studio • Create Visual studio project containing all sources of all modules (including header files for thirdparty libs, sdks and compilers) → Visual Assist is happy ;) • Takes some seconds for a middle size project with 10 configurations • We support Eclipse CDT btw ...

  37. Modules – Nearly there • This solves most of the requirements • Define (re-)usable modules • Define a project out of modules • Compilers, sdks and thirdparty libs • Very flexible due to the underlying programming language • No solution for the versioning problem (yet) • No solution for sharing the modules in different projects (yet)

  38. Solving the version problem • The best solution we found is svn:externals • We use subversion as our source code storage system • With svn:externals you can specify that a file or directory should come from a different path in the repository

  39. svn:externals for modules • all shared modules in one place • reference the modules into the project with svn:externals • This means, that you just have to checkout your project (it automatically gets the externals) • You can choose the version and revision of a module in your project • The shared source is still only in one place (the library)

  40. Modules - Advantages • Easy to lock the revisions of modules to a certain revision number • Every checkout of the project in the future will create exactly the same code • One Version number for the project • You can still branch a module and fix last minute things or make project specific adjustments • Setup of a new pc is really easy …

  41. Modules - Disadvantages • HUGE checkouts • A normal project can easily have some GB of code/libraries coming in • As a compromise we put converters that are project specific into the project folder and use the same modules • Not too bad in practice • Buy big and fast hard drives ;) • Currently only works with subversion (easy to implement on other SCC software that support a similar feature)

  42. Build system - Summary • Works really well in practice • No one wants to go back to manually define project settings in visual studio • Really easy to create a new project, import some modules and have something on the screen & running (Nice for prototypes) • Pushes Developers to think in modules → improves code architecture/quality • A life saver when you have to archive/restore a project

  43. Last but not Least • Our modular code build system is available as open source at • http://sf.net/projects/lace-build/ • Try it, use it and improve it for your use case • Released under the MIT License • Ask us if you prefer another license • You can find contact information on the source-forge page.

  44. Questions ?

More Related