1 / 37

AS-AP Archive and Preservation Early Draft

AS-AP Archive and Preservation Early Draft. Oliver Morgan Metaglue Corporation August 2010. AS-AP Archive and Preservation. Goals define a standard MXF format for A&P discuss and reach consensus on image formats, file structure, and metadata reconcile with implementation practice

jonco
Download Presentation

AS-AP Archive and Preservation Early Draft

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AS-AP Archive and PreservationEarly Draft Oliver MorganMetaglue CorporationAugust 2010

  2. AS-AP Archive and Preservation • Goals • define a standard MXF format for A&P • discuss and reach consensus on image formats, file structure, and metadata • reconcile with implementation practice • review and publish through AMWA • eventually, due-process standardization

  3. Profiling MXF • MXF is a massive standard • wide coverage • many details • many options • needs focus to specific applications • Application Specification • defines broad choices for an application area • ex: Program Delivery, Versioning, Craft Editing, Contribution,... • Shim • detailed constraints for specific uses

  4. the universe of media formats media formats that can be carried in MXF Archive and Preservation (Application Spec) Umatic (Shim)

  5. the universe of media formats media formats that can be carried in MXF Archive and Preservation (Application Spec) Umatic Film (Shim) NAVCC? (Sub-Shim)

  6. the universe of media formats media formats that can be carried in MXF Archive and Preservation (Application Spec) Photo Umatic Film (Shim) NARA? NAVCC? (Sub-Shim)

  7. Details and AS-AP Topics • AS-03 case study/example • slides follow below • AS-AP Strawman • “AMWA-AS-AP-Archive&Preservation-Spec-1a-20100811.doc”

  8. AS-03 Program Delivery Oliver MorganMetaglue CorporationJune 2009

  9. AS-03 MXF Program Delivery • File-Based Workflow • Which file format? • Based on SMPTE Material Exchange Format (MXF) • The SMPTE MXF standards cover a very wide range of parameters and user needs • AS-03 is a vendor-neutral interoperability profile of MXF • The NGIS HD and NGIS SD “shims” focus the AS-03 profile on PBS programming distribution requirements • Other “shims” are anticipated for other organizations

  10. Why MXF? • There are many file formats to choose from • almost as many formats as there are recording devices • that is fine so long as we all use the same server… • The industry has been searching for a format that can become as ubiquitous as SDI • Goal is a non-proprietary specification • SMPTE started work in earnest in 1997 • Universal Wrapper • Compression Agnostic, Bitrate Agnostic • Application Agnostic • Allow Mix and Match • Universal Metadata • Both Standard and User-Defined

  11. MXF History • MXF was published in 2004 • 700 pages of specifications • a disruptive technology • Developed 1997 to 2003 with lots of testing • published documents needed corrigenda & amendment • initial implementations were rough • Nonetheless, user uptake gathered momentum

  12. MXF Today • Libraries of AAF and MXF files exist today • have existed since 2002 • Primarily essence files • also include valuable customer metadata • Some examples • Digital Cinema Releasing: 50,000 per year ? • Digital Cinema Post Production: scads • Commercial TV: 50,000 per month ? • Shoah Foundation: 50,000 per year • Culpeper Archive: will be 250,000 per year • Government: 1 per minute per aircraft

  13. MasterMob FileSourceMob MasterMob FileSourceMob MasterMob FileSourceMob UMID UMID UMID UMID UMID UMID UMID UMID DM DM DM DM DM DM V1 V1 V1 V1 V1 V1 A1 A1 A1 A1 A1 A1 A2 A2 A2 A2 A2 A2 AX AX AX AX AX AX TC 23 24 00 01 02 03 TC 23 24 00 01 02 03 TC 23 24 00 01 02 03 TC 23 24 00 01 02 03 TC 23 24 00 01 02 03 TC 23 24 00 01 02 03 EssenceData EssenceData Some of the (many) kinds of MXF Files

  14. FileSourceMob PhysicalSourceMob MasterMob PhysicalSourceMob MasterMob FileSourceMob UMID UMID UMID UMID UMID UMID UMID UMID DM DM DM DM V1 V1 V1 V1 A1 A1 A1 A1 A2 A2 A2 A2 AX AX AX AX TC 23 24 00 01 02 03 TC 23 24 00 01 02 03 TC 23 24 00 01 02 03 TC 23 24 00 01 02 03 EssenceData DM DM V1 V1 A1 A1 A2 A2 AX AX TC 23 24 00 01 02 03 TC 23 24 00 01 02 03 EssenceData Some more…

  15. MasterMob FileSourceMob UMID UMID UMID DM DM V1 V1 A1 A1 A2 A2 AX AX TC 23 24 00 01 02 03 TC 23 24 00 01 02 03 EssenceData AS-03 File (MXF OP1a)

  16. Overall View Of An AS-03 File

  17. Package Media Object Package Identifier Descriptive Metadata (DM) SMPTE UMID –Software GUID –User-defined – – any number, any scheme– SMPTE, MXF, User-defined Tracks Sequences – any length any number –Picture, Sound –Descriptive Metadata –Data, User-defined– Links to other Mobs Timecodes – as needed – internal or external any number, any frame rate –SMPTE, Keycode, GPS– UMID DM V1 A1 A2 AX TC23 24 00 01 02 03 MXF “Package”

  18. AS-03 Goals • The goal of the AS-03 Program Delivery project is create specifications and sample files for MXF distribution of finished programs and program segments with associated metadata. • The content of these files is not intended for further processing before delivery to the consumer • Direct Playout, from any server • Specifications will be composed of a set of general constraints plus one or more "shims" that are directed to specific applications

  19. How? • AS-03 defines sets of constraints upon the use of the features of the MXF file format for use in the program delivery chain. • The intent of these constraints is to minimize the number of options that must be considered by implementers of individual devices within the system, thus increasing the reliability and interoperability of the overall system.

  20. Specification • A set of general constraints plus one or more "shims" that are directed to specific applications: for example, one for "NGIS HD", one for "NGIS SD", one for "Broadcaster A", and so on. • This allows us to remain flexible while maintaining our focus on delivering this project according to the proposed schedule • We can move issues between the general text and the "shims" to optimize the split between generality and specific application requirements

  21. AS-03 Coverage • The specifications will cover both SD and HD program content encoded in long GOP MPEG2 or H.264 constrained to specific bit rates and GOP structures. • Captioning and VBI data will be carried separately from video. • Program audio will be constrained to specific channel repertoires and bit rates. • Essence will be interleaved to ensure preservation of sync during distribution • MPEG V-ES in SMPTE 381 • PCM in SMPTE 382 • AC-3 ES in SMPTE 339 in SMPTE 382 • VBI in SMPTE 436 • Alll multiplexed in SMPTE 379M Generic Container

  22. AS-03 Metadata • Different metadata sets appropriate to each distribution application will be carried in the MXF header • A small summary metadata set will be present in all files • will carry a few metadata items gathered from the application-specific metadata • to facilitate content identification, routing and multiplexing • DMS-AS-03 • For NGIS HD and SD • Slate metadata • Program ID (P123456-001) • Total Run Time (TRT) • Start/End of Message (SOM and EOM) pairs • Active Format Description (AFD) • Feature Indicators (CC, DVS, SAP, Vchip, Rating) • Language Codes • NOLA code • Title • Media type (signal type) • Optional PB/Core

  23. AS-03 Constraint Tabulation - Picture

  24. AS-03 Constraint Tabulation - Sound

  25. More Constraints – Captions, Metadata

  26. More Constraints – MXF-specific

  27. Reference Implementation • Utilities have been produced to wrap test essence and metadata into AS-03 files, to unwrap the files and compare the inputs and outputs. These will be used throughout the project to illuminate design issues and to perform regression testing • Using mxflib open source • scripts to wrap and unwrap • design testing is performed with a custom scripting application • simpler functions will be included in NGIS workflow

  28. Sample Files • Test files will be created during the development, including “golden” (engineered good), and “silver” (real-world, proven good) • Silver streams encoded by users • for NGIS HD and SD encoded by PBS • Several encoders • Actual “Cringely” and other programming • Golden test streams • Golden test sequence uses synthetic test patterns • MPEG Video, AC3, PCM, CEA 608 and 708

  29. AS-03 Testing • AS-03 builds on earlier work on AS-PBS • Basic methodology • make specs and sample files available to all vendors • regular telephone discussion • iterative testing • eventually, publish via the AMWA • Server vendors • Omneon, Harris, Thomson, Seachange,GVG, 360 Systems , Sony, Pinnacle, … • Now starting testing with transcoder vendors

  30. Segmentation • Some files cut to length • SOM = 0, EOM = total duration • e.g. AS-03 NGIS • Some files include preroll/postroll • SOM, EOM traditionally specified using timecode • Duration is less error-prone than EOM • SOM and EOM/Duration pairs • sequence of non-overlapping segments • may be separated by black fill, or back-to-back • (not AS-03) Segments may be in individual files

  31. Package Media Object Package Identifier Descriptive Metadata (DM) SMPTE UMID –Software GUID –User-defined – – any number, any scheme– SMPTE, MXF, User-defined Tracks Sequences – any length any number –Picture, Sound –Descriptive Metadata –Data, User-defined– Links to other Mobs Timecodes – as needed – internal or external any number, any frame rate –SMPTE, Keycode, GPS– UMID DM V1 A1 A2 AX TC23 24 00 01 02 03 MXF “Package”

  32. MasterPkg CompositionPkg FileSourcePkg FileSourcePkg PhysicalSourcePkg UMID UMID UMID UMID UMID UMID UMID DM DM DM V1 V1 V1 V2 A1 A1 A1 A2 A2 A2 FX AX AX AX V1 TC 23 24 00 01 02 03 TC 23 24 00 01 02 03 TC 23 24 00 01 02 03 A1 A2 EssenceData DM V1 A1 A2 AX TC 23 24 00 01 02 03 EssenceData The Package Derivation Chain

  33. MXF Operational Patterns

  34. SOM/EOM in MXF Structural Metadata (1) Descriptive Metadata Composition Package Picture Sound Descriptive Metadata Picture Material Package Sound Timecode 01:00:00:00 01:05:00:00 01:10:00:00 01:15:00:00 File Package Picture Sound <!--partial Reg-XML--> <compositionpackage> <track> <sequence> <datadefinition> Picture </datadefinition> <fill> <duration> 2:00:00 </duration> </fill> <sourceclip> <duration> 4:30:00 </duration> </sourceclip> <fill> <duration> 1:00:00 </duration> </fill> <sourceclip> <duration> 6:00:00 </duration> </sourceclip> <sourceclip> <duration> 1:20:00 </duration> </sourceclip> <fill> <duration> 20:00 </duration> </fill> <sourceclip> <duration> 2:20:00 </duration> </sourceclip> </sequence> </track> <track> … </track></compositionpackage>

  35. SOM/EOM in MXF Structural Metadata (2) Descriptive Metadata Picture Material Package Sound Timecode 01:00:00:00 01:05:00:00 01:10:00:00 01:15:00:00 File Package Picture Sound <!--partial Reg-XML--> <materialpackage> <track> <sequence> <datadefinition> Descriptive Metadata </datadefinition> <fill> <duration> 2:00:00 </duration> </fill> <dmsegment> <duration> 4:30:00 </duration> </dmsegment> <fill> <duration> 1:00:00 </duration> </fill> <dmsegment> <duration> 6:00:00 </duration> </dmsegment> <dmsegment> <duration> 1:20:00 </duration> </dmsegment> <fill> <duration> 20:00 </duration> </fill> <dmsegment> <duration> 2:20:00 </duration> </dmsegment> </sequence> </track> <track> … </track></materialpackage>

  36. CRC and Signature • Requirements for verification of file content? • transport/storage/delivery errors • authentication • file corruption during processing • integrity of repurposing (e.g. track swapping) • Techniques in AS-02 • MD5 • CCITT CRC-16 • Castagnoli CRC-32 • Techniques from SMPTE 429-6 DCP Encryption • SHA-256/SHA-512 Secure Hash • full MIC and encryption • Other techniques • Signature • Hash essence together with the author’s public key

  37. Thank YouQuestions? Oliver Morgan oliver@metaglue.com http://www.metaglue.com

More Related