feeding the monster advanced data packaging for consoles
Skip this Video
Download Presentation
Feeding the Monster Advanced Data Packaging for Consoles

Loading in 2 Seconds...

play fullscreen
1 / 78

Feeding the Monster Advanced Data Packaging for Consoles - PowerPoint PPT Presentation

  • Uploaded on

Feeding the Monster Advanced Data Packaging for Consoles. Bruno Champoux Nicolas Fleury. Outline. Next-Generation Loading Problems LIP: A Loading Solution Packaging C++ Objects Demo: LIP Data Viewer Questions. Some things never change…. Next-Gen Loading Problem.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Feeding the Monster Advanced Data Packaging for Consoles' - mahola

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
  • Next-Generation Loading Problems
  • LIP: A Loading Solution
  • Packaging C++ Objects
  • Demo: LIP Data Viewer
  • Questions
next gen loading problem
Next-Gen Loading Problem
  • Processing power up by 10-40X
  • Memory size up by 8-16X
  • Optical drive performance up by 1-4X
next gen loading problem1
Next-Gen Loading Problem
  • Xbox 360
    • 12X dual-layer DVD drive
    • Outer edge speed: 15 MB/s
    • Average seek: 115 ms
  • PlayStation 3
    • Blu-ray performance still unknown
    • 1.5X is the most likely choice
    • CAV drive should give 6 to 16 MB/s
    • Average seek time might be worse than DVD
next gen loading problem3
Next-Gen Loading Problem
  • In order to feed the next-gen data needs, loading will need to be more frequent
  • Hard drives are optional for PS3 and Xbox 360
  • Optical drive performance does not scale with the memory/CPU power increase
  • Conclusion:
    • Loading performance must be optimal; any processing other than raw disc transfers must be eliminated
did i hear loading screen
Did I Hear “Loading Screen”?
  • Disruptive
  • Boring as hell
  • Non-skippable cutscenes are not better!
  • Conclusion:
    • Loading screens must not survive the current generation
background loading
Background Loading
  • Game assets are loaded during gameplay
  • Player immersion is preserved
  • Solution:
    • Use blocking I/O in a thread or another processor
background loading1
Background Loading
  • Requirements:
    • Cannot be much slower than a loading screen
    • Must have low CPU overhead
    • Must not block other streams
  • Conclusion:
    • Once again, loading performance must be optimal; any processing other than raw disc transfers must be eliminated
proposing a solution
Proposing A Solution
  • Requirements for a next-generation packaging and loading solution:
    • Large amounts of assets must be loaded at speeds nearing the hardware transfer limit
    • Background loading must be possible at little CPU cost
    • Data assets must be streamed in and phased out without causing memory fragmentation
understanding loading times
Understanding Loading Times
  • Freeing memory space
    • Unloading
    • Defragmenting
  • Seek time
  • Read time
  • Allocations
  • Parsing
  • Relocation (pointers, hash ID lookups)
  • Registration (e.g. physics system)
reducing loading times
Reducing Loading Times
  • Always load compressed files
    • Using N:1 compression will load N times faster
    • Double-buffering hides decompression time
    • Plenty of processing power available for decompression on next-gen consoles
reducing loading times1
Reducing Loading Times
  • Compression algorithm choice
    • Favor incremental approach
    • Use an algorithm based on bytes, not bits
      • Lempel-Ziv family
      • LZO
reducing loading times2
Reducing Loading Times
  • Take advantage of spatial and game flow coherence
    • Batch related data together in one file to save seek time
    • Place related files next to each other on the disc to minimize seek time
reducing loading times3
Reducing Loading Times
  • Take advantage of optical disc features
    • Store frequently accessed data in the outer section of the disc
    • Store music streams in the middle (prevents full seek)
    • Store single-use data near the center (videos, cutscenes, engine executable)
    • Beware of layer switching (0.1 seconds penalty)
reducing loading times4
Reducing Loading Times
  • Use the “flyweight” design pattern
    • Geometry instancing
    • Animation sharing
  • Favor procedural techniques
    • Parametric surfaces
    • Textures (fire, smoke, water)
reducing loading times5
Reducing Loading Times
  • Always prepare data offline
    • Eliminate text or intermediate format parsing in the engine
    • Engine time spent converting or interpreting data is wasted
    • Load native hardware and middleware formats
    • Load C++ objects directly
why load c objects
Why Load C++ Objects?
  • More natural way to work with data
  • Removes any need for parsing or interpreting assets
  • Creation is inexpensive
    • Pointer relocation
    • Hash ID conversion
    • Object registration
loading c objects
Loading C++ Objects
  • Requires a very smart packaging system
    • Member pointers
    • Virtual tables
    • Base classes
    • Alignment issues
    • Endianness
loading non c objects
Loading Non-C++ Objects
  • Must be in a format that is ready to use after being read to memory
    • Texture/normal maps
    • Havok structures
    • Audio
    • Script bytecode
  • Pretty straightforward
load in place lip
Load-In-Place (LIP)
  • Our solution for packaging and loading game assets
  • Framework for defining, storing and loading native C++ objects
  • Dynamic Storage: a self-defragmenting game asset container
load in place lip item
Load-In-Place: “LIP Item”
  • 1 LIP item  1 game asset
  • 1 LIP item  unique hash ID (64-bit)
    • 32 bits for the type ID and properties
    • 32 bits for the hashed asset name (CRC-32)
  • The smallest unit of data that can be
    • queried
    • moved by defragmentation
    • unloaded
  • Supports both C++ objects and binary blocks
examples of lip items
Examples of LIP Items
  • Joint Animation
  • Character Model
  • Environment Model Section
  • Collision Floor Section
  • Game Object (hero, enemy, trigger, etc.)
  • Script
  • Particle Emitter
  • Texture
c based lip items
C++-Based LIP Items
  • Can be made of any number of C++ objects and arrays
  • On the disc, all internal pointers are kept relative to the LIP item block
    • Pointer relocation starts with a placement new on a “relocation constructor”
    • Internal pointers are relocated automatically through “constructor chaining”
placement new operator
Placement “new” Operator
  • Syntax
    • new(<address>) <type>;
  • Calls the constructor but does not allocate memory
  • Initializes the virtual table
  • Called once for each LIP item on the main class relocation constructor
relocation constructors
Relocation Constructors
  • Required by all classes and structures that
    • can get loaded by the LIP framework
    • contain members that require relocation
  • 3 constructors
    • Loading relocation constructor
    • Moving relocation constructor (defragmentation)
    • Dynamic constructor (optional, can be dummy)
  • No default constructor!
object members relocation
Object Members Relocation
  • Internal pointer
    • Must point within the LIP item block
    • Converted into absolute pointer
  • External reference (LIP items only)
    • Stored as a LIP item hash ID
    • Converted into a pointer in the global asset table entry that points to the referenced LIP item
  • LIP framework provides wrapper classes with appropriate constructors for all pointer types
relocation example
Relocation Example

class GameObject {


GameObject(const LoadContext& ctx);

GameObject(const MoveContext& ctx);

GameObject(HASHID id,

Script* pScript);


lip::RelocPtr<Transfo> mpLocation;

lip::LipItemPtr<Script> mpScript;


relocation example cont d
Relocation Example (cont’d)

GameObject::GameObject(const LoadContext& ctx) :


mpScript(ctx) {}

GameObject::GameObject(const MoveContext& ctx) :


mpScript(ctx) {}

GameObject::GameObject(HASHID id,

Script* pScript) :

mpLocation(new Transfo),

mpScript(pScript) { SetHashId(id); }

relocation example cont d1
Relocation Example (cont’d)


void PlacementNew(

lip::LoadContext& loadCtx)





loadCtx.pvBaseAddr = pvLoadMemory;


load in place load unit
Load-In-Place: “Load Unit”
  • Group of LIP items
  • The smallest unit of data that can be loaded
    • 1 load unit  1 load command
  • Number of files is minimized
    • 1 language-independent file
      • Models, animations, scripts, environments, …
    • N language-dependent files
      • Fonts, in-game text, some textures, audio, …
  • Load unit files are compressed
load unit table
Load Unit Table
  • Each LIP item has an entry in the table
    • Hash ID
    • Offset to LIP Item
dynamic storage
Dynamic Storage
  • Loading process
    • Load unit files are read and decompressed to available storage memory
    • Load unit table offsets are relocated
    • Load unit table entries are merged in the global asset table
    • A placement new is called for each LIP item
    • Some LIP item types may require a second initialization pass (e.g. registration)
dynamic storage1
Dynamic Storage
  • Unloading process
    • Each LIP item can be removed individually
    • All LIP items of a load unit can be removed together
    • Destructors are called on C++ LIP items
    • Dynamic storage algorithm will defragment the new holes later
  • Locking
    • LIP items can be locked
    • Locked items cannot be moved or unloaded
platform specific issues
Platform-Specific Issues
  • GameCube
    • Special ARAM load unit files
      • Animations
      • Collision floors
    • Small disc  compression
  • Xbox/Xbox 360
    • Special LIP items for DirectX buffers
      • Vertex, index and texture buffers
      • 4KB-aligned LIP items (binary blocks)
      • Buffer headers in separate LIP items (C++ objects)
load in place other uses
Load-In-Place: Other Uses
  • Network-based asset editing
    • LIP items can be transferred from and to our level editor during gameplay
    • Changes in asset sizes do not matter
  • Used by Maya exporters to store our intermediate art assets
    • LIP is much more efficient than parsing XML!
packaging c objects
Packaging C++ Objects

Nicolas Fleury

our previous python based system
Our Previous Python-Based System
  • class MyClass(LipObject): x = Member(UInt32) y = Member(UInt32, default=1) p = Member(makeArrayType( makePtrType(SomeClass)))
cool things with this system
Cool Things with this System
  • Not too complex to implement.
  • Python is easy to use.
  • Introspection support.
  • A lot of freedom in corresponding C++ classes.
problems with this system
Problems with this System
  • Python and C++ structures must be synchronized.
  • Exporters must be written, at least partly, in Python.
  • Validations limited (unless you parse C++ code).
  • We just invented a Python/C++ hybrid.
c based system
C++-based system
  • class MyClass : public MyBaseCls { ... LIP_DECLARE_REGISTER(MyClass); uint32 x;};
  • Exporters are now written in C++.
  • Class content written twice, but synchronization fully validated.
  • Dummy engine .DLL must be compiled (not a working engine, provides only reflection/introspection).
  • Need a good build system.
  • We just added reflection/introspection to C++.
member registration information
Member Registration Information
  • Name
  • Offset
  • Type
  • Special flags (exposed in level editor, etc.)
non empty base class registration information
(Non Empty) Base Class Registration Information
  • Name
  • Type
  • Offset; calculated with:
    • (size_t)(BaseClassType*)(SubClassType*)1 - 1
member type deduction
Member Type Deduction
  • In IntrospectorBase class:template < typename TypeT, typename MemberT>void RegisterMember( const char* name, MemberT(TypeT::*memberPtr));
member type deduction arrays
Member Type Deduction (Arrays)
  • In IntrospectorBase class:template < typename TypeT, typename MemberTypeT, int sizeT>void RegisterMember( const char* name, MemberT(TypeT::*memberPtr)[sizeT]);
needed information in tools
Needed Information in Tools
  • Every class base class (to write their members too).
  • Every class member.
  • Every base class offset (to detect missing base class registration).
  • Every member name, size, type and special flags.
  • For every type, the necessary alignment and if it is polymorphic.
result member introspection
Result: Member Introspection
  • Able to know all types and their members.
  • Can be used for both writing and reading binary data.
  • Same class used in tools to fill the data as in engine.
  • Data of any platform, endianness, pointer size (binary files have a header with platform id).
  • Both for engine data and tools binary formats.
  • Hexadecimal viewer integration, edition support.
  • Excellent learning and debugging tool.
restrictions for simplification of implementation
Restrictions for Simplification of Implementation
  • Polymorphic types must begin with a vtable pointer (their first non-empty base class must be polymorphic).
  • Can’t inherit twice from same class indirectly (or offset trick doesn’t work).
  • No virtual base classes.
  • All padding is explicit.
explicit padding
Explicit Padding

class MyClass {...LIP_PADDING(mP1, LIP_PS3(12));uint16 mSomeMember;lip::Padding<4> mP2;uint32 mSomeOtherMember;LIP_PADDING(mP3,LIP_PC(4) LIP_PS3(8));...


particular things to handle
Particular Things to Handle
  • Endianness.
  • 64 bits pointers (no more!).
  • VTable padding.
  • Type alignment.
vtable padding
VTable Padding
  • __declspec(align(16)) class Matrix {…};
  • class MyClass { uint32 x, y, z; Matrix m;};
  • 32 bytes on PS3, 48 bytes on PC.
automatic versioning
Automatic Versioning
  • Create a huge string with member names/types and member names/types of pointed classes.
  • In the case of polymorphic pointers, all sub-types must also be included.
  • Hash the huge string.
  • Can be integrated in tools dependency tree mechanism.
needed information in engine
Needed Information in Engine
  • A hash map of objects to do the placement new of the appropriate type.
  • Smart pointers/arrays handle the rest.
type ids
Type Ids
  • VTable pointers are replaced by a type id.
  • LIP_DECLARE_TYPE_ID(MyClass, id) in .hpp.
    • Defines a compile-time mechanism to get id.
    • Declares a global object.
  • LIP_DEFINE_TYPE_ID(MyClass) in .cpp.
    • Defines the global object. Its constructor adds itself as a hash node to a hash map. This object class is templated to make operations with the good type (example: placement new).
  • Normal pointers like T* must be set to 0 when exporting an object.
  • For relocated pointers, smart pointer classes must be used.
  • Different types of smart pointers/arrays:
    • Ownership of pointed data?
    • Relocation of pointed data?
smart members
Smart Members
  • Classes can derive from a lip::SmartMember class to implement a custom writing/reading in tools.
  • Class only used as a tag, it doesn’t have any virtual function.
  • Classes deriving from lip::SmartMember are expected to implement a compile-time interface.
  • Useful for smart pointers (normal pointers cannot be load-in-placed).
smart members full control over writing and reading
Smart Members: Full Control Over Writing and Reading.

class MySmartArray : lip::SmartMember {


void Write(lip::LipWriter&) const;

void WriteExternalData( lip::LipWriter&)const;

void Stream(lip::LipReader&);

void StreamExternalData( lip::LipReader&);


relocptr type

Relocation assumes pointed data is not of a sub-type and does directly a placement new of Type.

relocpolymorphicptr type

Relocation looks in the hash map to do placement new of the good type (involves a search and a virtual function call).

  • Concept of exclusivity group masks to regroup values in mask in a radio button group in GUI.
other solutions
Other Solutions
  • Parse debug info files.
  • Compile as C++/CLI.
  • Parse source code.