feeding the monster advanced data packaging for consoles n.
Skip this Video
Loading SlideShow in 5 Seconds..
Feeding the Monster Advanced Data Packaging for Consoles PowerPoint Presentation
Download Presentation
Feeding the Monster Advanced Data Packaging for Consoles

Loading in 2 Seconds...

play fullscreen
1 / 78

Feeding the Monster Advanced Data Packaging for Consoles - PowerPoint PPT Presentation

  • Uploaded on

Feeding the Monster Advanced Data Packaging for Consoles. Bruno Champoux Nicolas Fleury. Outline. Next-Generation Loading Problems LIP: A Loading Solution Packaging C++ Objects Demo: LIP Data Viewer Questions. Some things never change…. Next-Gen Loading Problem.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Feeding the Monster Advanced Data Packaging for Consoles' - ellis

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
  • Next-Generation Loading Problems
  • LIP: A Loading Solution
  • Packaging C++ Objects
  • Demo: LIP Data Viewer
  • Questions
next gen loading problem
Next-Gen Loading Problem
  • Processing power up by 10-40X
  • Memory size up by 8-16X
  • Optical drive performance up by 1-4X
next gen loading problem1
Next-Gen Loading Problem
  • Xbox 360
    • 12X dual-layer DVD drive
    • Outer edge speed: 15 MB/s
    • Average seek: 115 ms
  • PlayStation 3
    • Blu-ray performance still unknown
    • 1.5X is the most likely choice
    • CAV drive should give 6 to 16 MB/s
    • Average seek time might be worse than DVD
next gen loading problem3
Next-Gen Loading Problem
  • In order to feed the next-gen data needs, loading will need to be more frequent
  • Hard drives are optional for PS3 and Xbox 360
  • Optical drive performance does not scale with the memory/CPU power increase
  • Conclusion:
    • Loading performance must be optimal; any processing other than raw disc transfers must be eliminated
did i hear loading screen
Did I Hear “Loading Screen”?
  • Disruptive
  • Boring as hell
  • Non-skippable cutscenes are not better!
  • Conclusion:
    • Loading screens must not survive the current generation
background loading
Background Loading
  • Game assets are loaded during gameplay
  • Player immersion is preserved
  • Solution:
    • Use blocking I/O in a thread or another processor
background loading1
Background Loading
  • Requirements:
    • Cannot be much slower than a loading screen
    • Must have low CPU overhead
    • Must not block other streams
  • Conclusion:
    • Once again, loading performance must be optimal; any processing other than raw disc transfers must be eliminated
proposing a solution
Proposing A Solution
  • Requirements for a next-generation packaging and loading solution:
    • Large amounts of assets must be loaded at speeds nearing the hardware transfer limit
    • Background loading must be possible at little CPU cost
    • Data assets must be streamed in and phased out without causing memory fragmentation
understanding loading times
Understanding Loading Times
  • Freeing memory space
    • Unloading
    • Defragmenting
  • Seek time
  • Read time
  • Allocations
  • Parsing
  • Relocation (pointers, hash ID lookups)
  • Registration (e.g. physics system)
reducing loading times
Reducing Loading Times
  • Always load compressed files
    • Using N:1 compression will load N times faster
    • Double-buffering hides decompression time
    • Plenty of processing power available for decompression on next-gen consoles
reducing loading times1
Reducing Loading Times
  • Compression algorithm choice
    • Favor incremental approach
    • Use an algorithm based on bytes, not bits
      • Lempel-Ziv family
      • LZO
reducing loading times2
Reducing Loading Times
  • Take advantage of spatial and game flow coherence
    • Batch related data together in one file to save seek time
    • Place related files next to each other on the disc to minimize seek time
reducing loading times3
Reducing Loading Times
  • Take advantage of optical disc features
    • Store frequently accessed data in the outer section of the disc
    • Store music streams in the middle (prevents full seek)
    • Store single-use data near the center (videos, cutscenes, engine executable)
    • Beware of layer switching (0.1 seconds penalty)
reducing loading times4
Reducing Loading Times
  • Use the “flyweight” design pattern
    • Geometry instancing
    • Animation sharing
  • Favor procedural techniques
    • Parametric surfaces
    • Textures (fire, smoke, water)
reducing loading times5
Reducing Loading Times
  • Always prepare data offline
    • Eliminate text or intermediate format parsing in the engine
    • Engine time spent converting or interpreting data is wasted
    • Load native hardware and middleware formats
    • Load C++ objects directly
why load c objects
Why Load C++ Objects?
  • More natural way to work with data
  • Removes any need for parsing or interpreting assets
  • Creation is inexpensive
    • Pointer relocation
    • Hash ID conversion
    • Object registration
loading c objects
Loading C++ Objects
  • Requires a very smart packaging system
    • Member pointers
    • Virtual tables
    • Base classes
    • Alignment issues
    • Endianness
loading non c objects
Loading Non-C++ Objects
  • Must be in a format that is ready to use after being read to memory
    • Texture/normal maps
    • Havok structures
    • Audio
    • Script bytecode
  • Pretty straightforward
load in place lip
Load-In-Place (LIP)
  • Our solution for packaging and loading game assets
  • Framework for defining, storing and loading native C++ objects
  • Dynamic Storage: a self-defragmenting game asset container
load in place lip item
Load-In-Place: “LIP Item”
  • 1 LIP item  1 game asset
  • 1 LIP item  unique hash ID (64-bit)
    • 32 bits for the type ID and properties
    • 32 bits for the hashed asset name (CRC-32)
  • The smallest unit of data that can be
    • queried
    • moved by defragmentation
    • unloaded
  • Supports both C++ objects and binary blocks
examples of lip items
Examples of LIP Items
  • Joint Animation
  • Character Model
  • Environment Model Section
  • Collision Floor Section
  • Game Object (hero, enemy, trigger, etc.)
  • Script
  • Particle Emitter
  • Texture
c based lip items
C++-Based LIP Items
  • Can be made of any number of C++ objects and arrays
  • On the disc, all internal pointers are kept relative to the LIP item block
    • Pointer relocation starts with a placement new on a “relocation constructor”
    • Internal pointers are relocated automatically through “constructor chaining”
placement new operator
Placement “new” Operator
  • Syntax
    • new(<address>) <type>;
  • Calls the constructor but does not allocate memory
  • Initializes the virtual table
  • Called once for each LIP item on the main class relocation constructor
relocation constructors
Relocation Constructors
  • Required by all classes and structures that
    • can get loaded by the LIP framework
    • contain members that require relocation
  • 3 constructors
    • Loading relocation constructor
    • Moving relocation constructor (defragmentation)
    • Dynamic constructor (optional, can be dummy)
  • No default constructor!
object members relocation
Object Members Relocation
  • Internal pointer
    • Must point within the LIP item block
    • Converted into absolute pointer
  • External reference (LIP items only)
    • Stored as a LIP item hash ID
    • Converted into a pointer in the global asset table entry that points to the referenced LIP item
  • LIP framework provides wrapper classes with appropriate constructors for all pointer types
relocation example
Relocation Example

class GameObject {


GameObject(const LoadContext& ctx);

GameObject(const MoveContext& ctx);

GameObject(HASHID id,

Script* pScript);


lip::RelocPtr<Transfo> mpLocation;

lip::LipItemPtr<Script> mpScript;


relocation example cont d
Relocation Example (cont’d)

GameObject::GameObject(const LoadContext& ctx) :


mpScript(ctx) {}

GameObject::GameObject(const MoveContext& ctx) :


mpScript(ctx) {}

GameObject::GameObject(HASHID id,

Script* pScript) :

mpLocation(new Transfo),

mpScript(pScript) { SetHashId(id); }

relocation example cont d1
Relocation Example (cont’d)


void PlacementNew(

lip::LoadContext& loadCtx)





loadCtx.pvBaseAddr = pvLoadMemory;


load in place load unit
Load-In-Place: “Load Unit”
  • Group of LIP items
  • The smallest unit of data that can be loaded
    • 1 load unit  1 load command
  • Number of files is minimized
    • 1 language-independent file
      • Models, animations, scripts, environments, …
    • N language-dependent files
      • Fonts, in-game text, some textures, audio, …
  • Load unit files are compressed
load unit table
Load Unit Table
  • Each LIP item has an entry in the table
    • Hash ID
    • Offset to LIP Item
dynamic storage
Dynamic Storage
  • Loading process
    • Load unit files are read and decompressed to available storage memory
    • Load unit table offsets are relocated
    • Load unit table entries are merged in the global asset table
    • A placement new is called for each LIP item
    • Some LIP item types may require a second initialization pass (e.g. registration)
dynamic storage1
Dynamic Storage
  • Unloading process
    • Each LIP item can be removed individually
    • All LIP items of a load unit can be removed together
    • Destructors are called on C++ LIP items
    • Dynamic storage algorithm will defragment the new holes later
  • Locking
    • LIP items can be locked
    • Locked items cannot be moved or unloaded
platform specific issues
Platform-Specific Issues
  • GameCube
    • Special ARAM load unit files
      • Animations
      • Collision floors
    • Small disc  compression
  • Xbox/Xbox 360
    • Special LIP items for DirectX buffers
      • Vertex, index and texture buffers
      • 4KB-aligned LIP items (binary blocks)
      • Buffer headers in separate LIP items (C++ objects)
load in place other uses
Load-In-Place: Other Uses
  • Network-based asset editing
    • LIP items can be transferred from and to our level editor during gameplay
    • Changes in asset sizes do not matter
  • Used by Maya exporters to store our intermediate art assets
    • LIP is much more efficient than parsing XML!
packaging c objects
Packaging C++ Objects

Nicolas Fleury

our previous python based system
Our Previous Python-Based System
  • class MyClass(LipObject): x = Member(UInt32) y = Member(UInt32, default=1) p = Member(makeArrayType( makePtrType(SomeClass)))
cool things with this system
Cool Things with this System
  • Not too complex to implement.
  • Python is easy to use.
  • Introspection support.
  • A lot of freedom in corresponding C++ classes.
problems with this system
Problems with this System
  • Python and C++ structures must be synchronized.
  • Exporters must be written, at least partly, in Python.
  • Validations limited (unless you parse C++ code).
  • We just invented a Python/C++ hybrid.
c based system
C++-based system
  • class MyClass : public MyBaseCls { ... LIP_DECLARE_REGISTER(MyClass); uint32 x;};
  • Exporters are now written in C++.
  • Class content written twice, but synchronization fully validated.
  • Dummy engine .DLL must be compiled (not a working engine, provides only reflection/introspection).
  • Need a good build system.
  • We just added reflection/introspection to C++.
member registration information
Member Registration Information
  • Name
  • Offset
  • Type
  • Special flags (exposed in level editor, etc.)
non empty base class registration information
(Non Empty) Base Class Registration Information
  • Name
  • Type
  • Offset; calculated with:
    • (size_t)(BaseClassType*)(SubClassType*)1 - 1
member type deduction
Member Type Deduction
  • In IntrospectorBase class:template < typename TypeT, typename MemberT>void RegisterMember( const char* name, MemberT(TypeT::*memberPtr));
member type deduction arrays
Member Type Deduction (Arrays)
  • In IntrospectorBase class:template < typename TypeT, typename MemberTypeT, int sizeT>void RegisterMember( const char* name, MemberT(TypeT::*memberPtr)[sizeT]);
needed information in tools
Needed Information in Tools
  • Every class base class (to write their members too).
  • Every class member.
  • Every base class offset (to detect missing base class registration).
  • Every member name, size, type and special flags.
  • For every type, the necessary alignment and if it is polymorphic.
result member introspection
Result: Member Introspection
  • Able to know all types and their members.
  • Can be used for both writing and reading binary data.
  • Same class used in tools to fill the data as in engine.
  • Data of any platform, endianness, pointer size (binary files have a header with platform id).
  • Both for engine data and tools binary formats.
  • Hexadecimal viewer integration, edition support.
  • Excellent learning and debugging tool.
restrictions for simplification of implementation
Restrictions for Simplification of Implementation
  • Polymorphic types must begin with a vtable pointer (their first non-empty base class must be polymorphic).
  • Can’t inherit twice from same class indirectly (or offset trick doesn’t work).
  • No virtual base classes.
  • All padding is explicit.
explicit padding
Explicit Padding

class MyClass {...LIP_PADDING(mP1, LIP_PS3(12));uint16 mSomeMember;lip::Padding<4> mP2;uint32 mSomeOtherMember;LIP_PADDING(mP3,LIP_PC(4) LIP_PS3(8));...


particular things to handle
Particular Things to Handle
  • Endianness.
  • 64 bits pointers (no more!).
  • VTable padding.
  • Type alignment.
vtable padding
VTable Padding
  • __declspec(align(16)) class Matrix {…};
  • class MyClass { uint32 x, y, z; Matrix m;};
  • 32 bytes on PS3, 48 bytes on PC.
automatic versioning
Automatic Versioning
  • Create a huge string with member names/types and member names/types of pointed classes.
  • In the case of polymorphic pointers, all sub-types must also be included.
  • Hash the huge string.
  • Can be integrated in tools dependency tree mechanism.
needed information in engine
Needed Information in Engine
  • A hash map of objects to do the placement new of the appropriate type.
  • Smart pointers/arrays handle the rest.
type ids
Type Ids
  • VTable pointers are replaced by a type id.
  • LIP_DECLARE_TYPE_ID(MyClass, id) in .hpp.
    • Defines a compile-time mechanism to get id.
    • Declares a global object.
  • LIP_DEFINE_TYPE_ID(MyClass) in .cpp.
    • Defines the global object. Its constructor adds itself as a hash node to a hash map. This object class is templated to make operations with the good type (example: placement new).
  • Normal pointers like T* must be set to 0 when exporting an object.
  • For relocated pointers, smart pointer classes must be used.
  • Different types of smart pointers/arrays:
    • Ownership of pointed data?
    • Relocation of pointed data?
smart members
Smart Members
  • Classes can derive from a lip::SmartMember class to implement a custom writing/reading in tools.
  • Class only used as a tag, it doesn’t have any virtual function.
  • Classes deriving from lip::SmartMember are expected to implement a compile-time interface.
  • Useful for smart pointers (normal pointers cannot be load-in-placed).
smart members full control over writing and reading
Smart Members: Full Control Over Writing and Reading.

class MySmartArray : lip::SmartMember {


void Write(lip::LipWriter&) const;

void WriteExternalData( lip::LipWriter&)const;

void Stream(lip::LipReader&);

void StreamExternalData( lip::LipReader&);


relocptr type

Relocation assumes pointed data is not of a sub-type and does directly a placement new of Type.

relocpolymorphicptr type

Relocation looks in the hash map to do placement new of the good type (involves a search and a virtual function call).

  • Concept of exclusivity group masks to regroup values in mask in a radio button group in GUI.
other solutions
Other Solutions
  • Parse debug info files.
  • Compile as C++/CLI.
  • Parse source code.
  • Latest slides
    • http://www.a2m.com/gdc/
  • How to reach us
    • bruno.champoux@a2m.com
    • nicolas.fleury@a2m.com