1 / 78

Feeding the Monster Advanced Data Packaging for Consoles

Feeding the Monster Advanced Data Packaging for Consoles. Bruno Champoux Nicolas Fleury. Outline. Next-Generation Loading Problems LIP: A Loading Solution Packaging C++ Objects Demo: LIP Data Viewer Questions. Some things never change…. Next-Gen Loading Problem.

mahola
Download Presentation

Feeding the Monster Advanced Data Packaging for Consoles

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Feeding the MonsterAdvanced Data Packaging for Consoles Bruno ChampouxNicolas Fleury

  2. Outline • Next-Generation Loading Problems • LIP: A Loading Solution • Packaging C++ Objects • Demo: LIP Data Viewer • Questions

  3. Some things never change…

  4. Next-Gen Loading Problem • Processing power up by 10-40X • Memory size up by 8-16X • Optical drive performance up by 1-4X

  5. Next-Gen Loading Problem • Xbox 360 • 12X dual-layer DVD drive • Outer edge speed: 15 MB/s • Average seek: 115 ms • PlayStation 3 • Blu-ray performance still unknown • 1.5X is the most likely choice • CAV drive should give 6 to 16 MB/s • Average seek time might be worse than DVD

  6. Next-Gen Loading Problem

  7. Next-Gen Loading Problem • In order to feed the next-gen data needs, loading will need to be more frequent • Hard drives are optional for PS3 and Xbox 360 • Optical drive performance does not scale with the memory/CPU power increase • Conclusion: • Loading performance must be optimal; any processing other than raw disc transfers must be eliminated

  8. Did I Hear “Loading Screen”? • Disruptive • Boring as hell • Non-skippable cutscenes are not better! • Conclusion: • Loading screens must not survive the current generation

  9. Background Loading • Game assets are loaded during gameplay • Player immersion is preserved • Solution: • Use blocking I/O in a thread or another processor

  10. Background Loading • Requirements: • Cannot be much slower than a loading screen • Must have low CPU overhead • Must not block other streams • Conclusion: • Once again, loading performance must be optimal; any processing other than raw disc transfers must be eliminated

  11. Proposing A Solution • Requirements for a next-generation packaging and loading solution: • Large amounts of assets must be loaded at speeds nearing the hardware transfer limit • Background loading must be possible at little CPU cost • Data assets must be streamed in and phased out without causing memory fragmentation

  12. Understanding Loading Times • Freeing memory space • Unloading • Defragmenting • Seek time • Read time • Allocations • Parsing • Relocation (pointers, hash ID lookups) • Registration (e.g. physics system)

  13. Reducing Loading Times • Always load compressed files • Using N:1 compression will load N times faster • Double-buffering hides decompression time • Plenty of processing power available for decompression on next-gen consoles

  14. Reducing Loading Times • Compression algorithm choice • Favor incremental approach • Use an algorithm based on bytes, not bits • Lempel-Ziv family • LZO

  15. Reducing Loading Times • Take advantage of spatial and game flow coherence • Batch related data together in one file to save seek time • Place related files next to each other on the disc to minimize seek time

  16. Reducing Loading Times • Take advantage of optical disc features • Store frequently accessed data in the outer section of the disc • Store music streams in the middle (prevents full seek) • Store single-use data near the center (videos, cutscenes, engine executable) • Beware of layer switching (0.1 seconds penalty)

  17. Reducing Loading Times • Use the “flyweight” design pattern • Geometry instancing • Animation sharing • Favor procedural techniques • Parametric surfaces • Textures (fire, smoke, water)

  18. Reducing Loading Times • Always prepare data offline • Eliminate text or intermediate format parsing in the engine • Engine time spent converting or interpreting data is wasted • Load native hardware and middleware formats • Load C++ objects directly

  19. Why Load C++ Objects? • More natural way to work with data • Removes any need for parsing or interpreting assets • Creation is inexpensive • Pointer relocation • Hash ID conversion • Object registration

  20. Loading C++ Objects • Requires a very smart packaging system • Member pointers • Virtual tables • Base classes • Alignment issues • Endianness

  21. Loading Non-C++ Objects • Must be in a format that is ready to use after being read to memory • Texture/normal maps • Havok structures • Audio • Script bytecode • Pretty straightforward

  22. Load-In-Place (LIP) • Our solution for packaging and loading game assets • Framework for defining, storing and loading native C++ objects • Dynamic Storage: a self-defragmenting game asset container

  23. Load-In-Place: “LIP Item” • 1 LIP item  1 game asset • 1 LIP item  unique hash ID (64-bit) • 32 bits for the type ID and properties • 32 bits for the hashed asset name (CRC-32) • The smallest unit of data that can be • queried • moved by defragmentation • unloaded • Supports both C++ objects and binary blocks

  24. Examples of LIP Items • Joint Animation • Character Model • Environment Model Section • Collision Floor Section • Game Object (hero, enemy, trigger, etc.) • Script • Particle Emitter • Texture

  25. C++-Based LIP Items • Can be made of any number of C++ objects and arrays • On the disc, all internal pointers are kept relative to the LIP item block • Pointer relocation starts with a placement new on a “relocation constructor” • Internal pointers are relocated automatically through “constructor chaining”

  26. Placement “new” Operator • Syntax • new(<address>) <type>; • Calls the constructor but does not allocate memory • Initializes the virtual table • Called once for each LIP item on the main class relocation constructor

  27. Relocation Constructors • Required by all classes and structures that • can get loaded by the LIP framework • contain members that require relocation • 3 constructors • Loading relocation constructor • Moving relocation constructor (defragmentation) • Dynamic constructor (optional, can be dummy) • No default constructor!

  28. Object Members Relocation • Internal pointer • Must point within the LIP item block • Converted into absolute pointer • External reference (LIP items only) • Stored as a LIP item hash ID • Converted into a pointer in the global asset table entry that points to the referenced LIP item • LIP framework provides wrapper classes with appropriate constructors for all pointer types

  29. Relocation Example class GameObject { public: GameObject(const LoadContext& ctx); GameObject(const MoveContext& ctx); GameObject(HASHID id, Script* pScript); protected: lip::RelocPtr<Transfo> mpLocation; lip::LipItemPtr<Script> mpScript; };

  30. Relocation Example (cont’d) GameObject::GameObject(const LoadContext& ctx) : mpLocation(ctx), mpScript(ctx) {} GameObject::GameObject(const MoveContext& ctx) : mpLocation(ctx), mpScript(ctx) {} GameObject::GameObject(HASHID id, Script* pScript) : mpLocation(new Transfo), mpScript(pScript) { SetHashId(id); }

  31. Relocation Example (cont’d) template<typenameLipItemT> void PlacementNew( lip::LoadContext& loadCtx) { new(loadCtx.pvBaseAddr) LipItemT(loadCtx); } loadCtx.pvBaseAddr = pvLoadMemory; PlacementNew<GameObject>(loadCtx);

  32. Relocation Example (cont’d)

  33. Load-In-Place: “Load Unit” • Group of LIP items • The smallest unit of data that can be loaded • 1 load unit  1 load command • Number of files is minimized • 1 language-independent file • Models, animations, scripts, environments, … • N language-dependent files • Fonts, in-game text, some textures, audio, … • Load unit files are compressed

  34. Load Unit Table • Each LIP item has an entry in the table • Hash ID • Offset to LIP Item

  35. Dynamic Storage • Loading process • Load unit files are read and decompressed to available storage memory • Load unit table offsets are relocated • Load unit table entries are merged in the global asset table • A placement new is called for each LIP item • Some LIP item types may require a second initialization pass (e.g. registration)

  36. Dynamic Storage • Unloading process • Each LIP item can be removed individually • All LIP items of a load unit can be removed together • Destructors are called on C++ LIP items • Dynamic storage algorithm will defragment the new holes later • Locking • LIP items can be locked • Locked items cannot be moved or unloaded

  37. Platform-Specific Issues • GameCube • Special ARAM load unit files • Animations • Collision floors • Small disc  compression • Xbox/Xbox 360 • Special LIP items for DirectX buffers • Vertex, index and texture buffers • 4KB-aligned LIP items (binary blocks) • Buffer headers in separate LIP items (C++ objects)

  38. Load-In-Place: Other Uses • Network-based asset editing • LIP items can be transferred from and to our level editor during gameplay • Changes in asset sizes do not matter • Used by Maya exporters to store our intermediate art assets • LIP is much more efficient than parsing XML!

  39. Packaging C++ Objects Nicolas Fleury

  40. Our Previous Python-Based System • class MyClass(LipObject): x = Member(UInt32) y = Member(UInt32, default=1) p = Member(makeArrayType( makePtrType(SomeClass)))

  41. Cool Things with this System • Not too complex to implement. • Python is easy to use. • Introspection support. • A lot of freedom in corresponding C++ classes.

  42. Problems with this System • Python and C++ structures must be synchronized. • Exporters must be written, at least partly, in Python. • Validations limited (unless you parse C++ code). • We just invented a Python/C++ hybrid.

  43. C++-based system • class MyClass : public MyBaseCls { ... LIP_DECLARE_REGISTER(MyClass); uint32 x;}; • // In .cppLIP_DEFINE_REGISTER(MyClass) { LIP_REGISTER_BASE_CLASS(MyBaseCls); LIP_REGISTER_MEMBER(x);}

  44. Consequences • Exporters are now written in C++. • Class content written twice, but synchronization fully validated. • Dummy engine .DLL must be compiled (not a working engine, provides only reflection/introspection). • Need a good build system. • We just added reflection/introspection to C++.

  45. Member Registration Information • Name • Offset • Type • Special flags (exposed in level editor, etc.)

  46. (Non Empty) Base Class Registration Information • Name • Type • Offset; calculated with: • (size_t)(BaseClassType*)(SubClassType*)1 - 1

  47. Member Type Deduction • In IntrospectorBase class:template < typename TypeT, typename MemberT>void RegisterMember( const char* name, MemberT(TypeT::*memberPtr));

  48. Member Type Deduction (Arrays) • In IntrospectorBase class:template < typename TypeT, typename MemberTypeT, int sizeT>void RegisterMember( const char* name, MemberT(TypeT::*memberPtr)[sizeT]);

  49. Needed Information in Tools • Every class base class (to write their members too). • Every class member. • Every base class offset (to detect missing base class registration). • Every member name, size, type and special flags. • For every type, the necessary alignment and if it is polymorphic.

  50. Introspection Classes (Members)

More Related