Feeding the monster advanced data packaging for consoles
1 / 78

Feeding the Monster Advanced Data Packaging for Consoles - PowerPoint PPT Presentation

  • Uploaded on

Feeding the Monster Advanced Data Packaging for Consoles. Bruno Champoux Nicolas Fleury. Outline. Next-Generation Loading Problems LIP: A Loading Solution Packaging C++ Objects Demo: LIP Data Viewer Questions. Some things never change…. Next-Gen Loading Problem.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Feeding the Monster Advanced Data Packaging for Consoles' - mahola

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Feeding the monster advanced data packaging for consoles
Feeding the MonsterAdvanced Data Packaging for Consoles

Bruno ChampouxNicolas Fleury


  • Next-Generation Loading Problems

  • LIP: A Loading Solution

  • Packaging C++ Objects

  • Demo: LIP Data Viewer

  • Questions

Next gen loading problem
Next-Gen Loading Problem

  • Processing power up by 10-40X

  • Memory size up by 8-16X

  • Optical drive performance up by 1-4X

Next gen loading problem1
Next-Gen Loading Problem

  • Xbox 360

    • 12X dual-layer DVD drive

    • Outer edge speed: 15 MB/s

    • Average seek: 115 ms

  • PlayStation 3

    • Blu-ray performance still unknown

    • 1.5X is the most likely choice

    • CAV drive should give 6 to 16 MB/s

    • Average seek time might be worse than DVD

Next gen loading problem3
Next-Gen Loading Problem

  • In order to feed the next-gen data needs, loading will need to be more frequent

  • Hard drives are optional for PS3 and Xbox 360

  • Optical drive performance does not scale with the memory/CPU power increase

  • Conclusion:

    • Loading performance must be optimal; any processing other than raw disc transfers must be eliminated

Did i hear loading screen
Did I Hear “Loading Screen”?

  • Disruptive

  • Boring as hell

  • Non-skippable cutscenes are not better!

  • Conclusion:

    • Loading screens must not survive the current generation

Background loading
Background Loading

  • Game assets are loaded during gameplay

  • Player immersion is preserved

  • Solution:

    • Use blocking I/O in a thread or another processor

Background loading1
Background Loading

  • Requirements:

    • Cannot be much slower than a loading screen

    • Must have low CPU overhead

    • Must not block other streams

  • Conclusion:

    • Once again, loading performance must be optimal; any processing other than raw disc transfers must be eliminated

Proposing a solution
Proposing A Solution

  • Requirements for a next-generation packaging and loading solution:

    • Large amounts of assets must be loaded at speeds nearing the hardware transfer limit

    • Background loading must be possible at little CPU cost

    • Data assets must be streamed in and phased out without causing memory fragmentation

Understanding loading times
Understanding Loading Times

  • Freeing memory space

    • Unloading

    • Defragmenting

  • Seek time

  • Read time

  • Allocations

  • Parsing

  • Relocation (pointers, hash ID lookups)

  • Registration (e.g. physics system)

Reducing loading times
Reducing Loading Times

  • Always load compressed files

    • Using N:1 compression will load N times faster

    • Double-buffering hides decompression time

    • Plenty of processing power available for decompression on next-gen consoles

Reducing loading times1
Reducing Loading Times

  • Compression algorithm choice

    • Favor incremental approach

    • Use an algorithm based on bytes, not bits

      • Lempel-Ziv family

      • LZO

Reducing loading times2
Reducing Loading Times

  • Take advantage of spatial and game flow coherence

    • Batch related data together in one file to save seek time

    • Place related files next to each other on the disc to minimize seek time

Reducing loading times3
Reducing Loading Times

  • Take advantage of optical disc features

    • Store frequently accessed data in the outer section of the disc

    • Store music streams in the middle (prevents full seek)

    • Store single-use data near the center (videos, cutscenes, engine executable)

    • Beware of layer switching (0.1 seconds penalty)

Reducing loading times4
Reducing Loading Times

  • Use the “flyweight” design pattern

    • Geometry instancing

    • Animation sharing

  • Favor procedural techniques

    • Parametric surfaces

    • Textures (fire, smoke, water)

Reducing loading times5
Reducing Loading Times

  • Always prepare data offline

    • Eliminate text or intermediate format parsing in the engine

    • Engine time spent converting or interpreting data is wasted

    • Load native hardware and middleware formats

    • Load C++ objects directly

Why load c objects
Why Load C++ Objects?

  • More natural way to work with data

  • Removes any need for parsing or interpreting assets

  • Creation is inexpensive

    • Pointer relocation

    • Hash ID conversion

    • Object registration

Loading c objects
Loading C++ Objects

  • Requires a very smart packaging system

    • Member pointers

    • Virtual tables

    • Base classes

    • Alignment issues

    • Endianness

Loading non c objects
Loading Non-C++ Objects

  • Must be in a format that is ready to use after being read to memory

    • Texture/normal maps

    • Havok structures

    • Audio

    • Script bytecode

  • Pretty straightforward

Load in place lip
Load-In-Place (LIP)

  • Our solution for packaging and loading game assets

  • Framework for defining, storing and loading native C++ objects

  • Dynamic Storage: a self-defragmenting game asset container

Load in place lip item
Load-In-Place: “LIP Item”

  • 1 LIP item  1 game asset

  • 1 LIP item  unique hash ID (64-bit)

    • 32 bits for the type ID and properties

    • 32 bits for the hashed asset name (CRC-32)

  • The smallest unit of data that can be

    • queried

    • moved by defragmentation

    • unloaded

  • Supports both C++ objects and binary blocks

Examples of lip items
Examples of LIP Items

  • Joint Animation

  • Character Model

  • Environment Model Section

  • Collision Floor Section

  • Game Object (hero, enemy, trigger, etc.)

  • Script

  • Particle Emitter

  • Texture

C based lip items
C++-Based LIP Items

  • Can be made of any number of C++ objects and arrays

  • On the disc, all internal pointers are kept relative to the LIP item block

    • Pointer relocation starts with a placement new on a “relocation constructor”

    • Internal pointers are relocated automatically through “constructor chaining”

Placement new operator
Placement “new” Operator

  • Syntax

    • new(<address>) <type>;

  • Calls the constructor but does not allocate memory

  • Initializes the virtual table

  • Called once for each LIP item on the main class relocation constructor

Relocation constructors
Relocation Constructors

  • Required by all classes and structures that

    • can get loaded by the LIP framework

    • contain members that require relocation

  • 3 constructors

    • Loading relocation constructor

    • Moving relocation constructor (defragmentation)

    • Dynamic constructor (optional, can be dummy)

  • No default constructor!

Object members relocation
Object Members Relocation

  • Internal pointer

    • Must point within the LIP item block

    • Converted into absolute pointer

  • External reference (LIP items only)

    • Stored as a LIP item hash ID

    • Converted into a pointer in the global asset table entry that points to the referenced LIP item

  • LIP framework provides wrapper classes with appropriate constructors for all pointer types

Relocation example
Relocation Example

class GameObject {


GameObject(const LoadContext& ctx);

GameObject(const MoveContext& ctx);

GameObject(HASHID id,

Script* pScript);


lip::RelocPtr<Transfo> mpLocation;

lip::LipItemPtr<Script> mpScript;


Relocation example cont d
Relocation Example (cont’d)

GameObject::GameObject(const LoadContext& ctx) :


mpScript(ctx) {}

GameObject::GameObject(const MoveContext& ctx) :


mpScript(ctx) {}

GameObject::GameObject(HASHID id,

Script* pScript) :

mpLocation(new Transfo),

mpScript(pScript) { SetHashId(id); }

Relocation example cont d1
Relocation Example (cont’d)


void PlacementNew(

lip::LoadContext& loadCtx)





loadCtx.pvBaseAddr = pvLoadMemory;


Load in place load unit
Load-In-Place: “Load Unit”

  • Group of LIP items

  • The smallest unit of data that can be loaded

    • 1 load unit  1 load command

  • Number of files is minimized

    • 1 language-independent file

      • Models, animations, scripts, environments, …

    • N language-dependent files

      • Fonts, in-game text, some textures, audio, …

  • Load unit files are compressed

Load unit table
Load Unit Table

  • Each LIP item has an entry in the table

    • Hash ID

    • Offset to LIP Item

Dynamic storage
Dynamic Storage

  • Loading process

    • Load unit files are read and decompressed to available storage memory

    • Load unit table offsets are relocated

    • Load unit table entries are merged in the global asset table

    • A placement new is called for each LIP item

    • Some LIP item types may require a second initialization pass (e.g. registration)

Dynamic storage1
Dynamic Storage

  • Unloading process

    • Each LIP item can be removed individually

    • All LIP items of a load unit can be removed together

    • Destructors are called on C++ LIP items

    • Dynamic storage algorithm will defragment the new holes later

  • Locking

    • LIP items can be locked

    • Locked items cannot be moved or unloaded

Platform specific issues
Platform-Specific Issues

  • GameCube

    • Special ARAM load unit files

      • Animations

      • Collision floors

    • Small disc  compression

  • Xbox/Xbox 360

    • Special LIP items for DirectX buffers

      • Vertex, index and texture buffers

      • 4KB-aligned LIP items (binary blocks)

      • Buffer headers in separate LIP items (C++ objects)

Load in place other uses
Load-In-Place: Other Uses

  • Network-based asset editing

    • LIP items can be transferred from and to our level editor during gameplay

    • Changes in asset sizes do not matter

  • Used by Maya exporters to store our intermediate art assets

    • LIP is much more efficient than parsing XML!

Packaging c objects
Packaging C++ Objects

Nicolas Fleury

Our previous python based system
Our Previous Python-Based System

  • class MyClass(LipObject): x = Member(UInt32) y = Member(UInt32, default=1) p = Member(makeArrayType( makePtrType(SomeClass)))

Cool things with this system
Cool Things with this System

  • Not too complex to implement.

  • Python is easy to use.

  • Introspection support.

  • A lot of freedom in corresponding C++ classes.

Problems with this system
Problems with this System

  • Python and C++ structures must be synchronized.

  • Exporters must be written, at least partly, in Python.

  • Validations limited (unless you parse C++ code).

  • We just invented a Python/C++ hybrid.

C based system
C++-based system

  • class MyClass : public MyBaseCls { ... LIP_DECLARE_REGISTER(MyClass); uint32 x;};



  • Exporters are now written in C++.

  • Class content written twice, but synchronization fully validated.

  • Dummy engine .DLL must be compiled (not a working engine, provides only reflection/introspection).

  • Need a good build system.

  • We just added reflection/introspection to C++.

Member registration information
Member Registration Information

  • Name

  • Offset

  • Type

  • Special flags (exposed in level editor, etc.)

Non empty base class registration information
(Non Empty) Base Class Registration Information

  • Name

  • Type

  • Offset; calculated with:

    • (size_t)(BaseClassType*)(SubClassType*)1 - 1

Member type deduction
Member Type Deduction

  • In IntrospectorBase class:template < typename TypeT, typename MemberT>void RegisterMember( const char* name, MemberT(TypeT::*memberPtr));

Member type deduction arrays
Member Type Deduction (Arrays)

  • In IntrospectorBase class:template < typename TypeT, typename MemberTypeT, int sizeT>void RegisterMember( const char* name, MemberT(TypeT::*memberPtr)[sizeT]);

Needed information in tools
Needed Information in Tools

  • Every class base class (to write their members too).

  • Every class member.

  • Every base class offset (to detect missing base class registration).

  • Every member name, size, type and special flags.

  • For every type, the necessary alignment and if it is polymorphic.

Introspection classes base classes
Introspection Classes (Base Classes)

Result member introspection
Result: Member Introspection

  • Able to know all types and their members.

  • Can be used for both writing and reading binary data.

  • Same class used in tools to fill the data as in engine.


  • Data of any platform, endianness, pointer size (binary files have a header with platform id).

  • Both for engine data and tools binary formats.

  • Hexadecimal viewer integration, edition support.

  • Excellent learning and debugging tool.

Restrictions for simplification of implementation
Restrictions for Simplification of Implementation

  • Polymorphic types must begin with a vtable pointer (their first non-empty base class must be polymorphic).

  • Can’t inherit twice from same class indirectly (or offset trick doesn’t work).

  • No virtual base classes.

  • All padding is explicit.

Explicit padding
Explicit Padding

class MyClass {...LIP_PADDING(mP1, LIP_PS3(12));uint16 mSomeMember;lip::Padding<4> mP2;uint32 mSomeOtherMember;LIP_PADDING(mP3,LIP_PC(4) LIP_PS3(8));...


Particular things to handle
Particular Things to Handle

  • Endianness.

  • 64 bits pointers (no more!).

  • VTable padding.

  • Type alignment.

Vtable padding
VTable Padding

  • __declspec(align(16)) class Matrix {…};

  • class MyClass { uint32 x, y, z; Matrix m;};

  • 32 bytes on PS3, 48 bytes on PC.

Automatic versioning
Automatic Versioning

  • Create a huge string with member names/types and member names/types of pointed classes.

  • In the case of polymorphic pointers, all sub-types must also be included.

  • Hash the huge string.

  • Can be integrated in tools dependency tree mechanism.

Needed information in engine
Needed Information in Engine

  • A hash map of objects to do the placement new of the appropriate type.

  • Smart pointers/arrays handle the rest.

Type ids
Type Ids

  • VTable pointers are replaced by a type id.

  • LIP_DECLARE_TYPE_ID(MyClass, id) in .hpp.

    • Defines a compile-time mechanism to get id.

    • Declares a global object.

  • LIP_DEFINE_TYPE_ID(MyClass) in .cpp.

    • Defines the global object. Its constructor adds itself as a hash node to a hash map. This object class is templated to make operations with the good type (example: placement new).


  • Normal pointers like T* must be set to 0 when exporting an object.

  • For relocated pointers, smart pointer classes must be used.

  • Different types of smart pointers/arrays:

    • Ownership of pointed data?

    • Relocation of pointed data?

Smart members
Smart Members

  • Classes can derive from a lip::SmartMember class to implement a custom writing/reading in tools.

  • Class only used as a tag, it doesn’t have any virtual function.

  • Classes deriving from lip::SmartMember are expected to implement a compile-time interface.

  • Useful for smart pointers (normal pointers cannot be load-in-placed).

Smart members full control over writing and reading
Smart Members: Full Control Over Writing and Reading.

class MySmartArray : lip::SmartMember {


void Write(lip::LipWriter&) const;

void WriteExternalData( lip::LipWriter&)const;

void Stream(lip::LipReader&);

void StreamExternalData( lip::LipReader&);


Relocptr type

Relocation assumes pointed data is not of a sub-type and does directly a placement new of Type.

Relocpolymorphicptr type

Relocation looks in the hash map to do placement new of the good type (involves a search and a virtual function call).


  • Concept of exclusivity group masks to regroup values in mask in a radio button group in GUI.


Other solutions
Other Solutions

  • Parse debug info files.

  • Compile as C++/CLI.

  • Parse source code.


  • Latest slides

    • http://www.a2m.com/gdc/

  • How to reach us

    • [email protected]

    • [email protected]