1 / 54

Contribution of SAFE to XFDU

Contribution of SAFE to XFDU. CCSDS/MOIMS/IPR Workshop (11-15/04/2005 Athens). Stéphane MBAYE GAEL Consultant Stephane.Mbaye@gael.fr. Rationale. SAFE Format Specifications 1.5 – [SAFE-FMT] ESA/CNES XFDU I/O Library – [XFDU-I/O] SAFE I/O library – [SAFE-I/O]

theta
Download Presentation

Contribution of SAFE to XFDU

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Contribution of SAFE to XFDU CCSDS/MOIMS/IPR Workshop(11-15/04/2005 Athens) Stéphane MBAYEGAEL ConsultantStephane.Mbaye@gael.fr

  2. Rationale • SAFE Format Specifications 1.5 – [SAFE-FMT] • ESA/CNES XFDU I/O Library – [XFDU-I/O] • SAFE I/O library – [SAFE-I/O] • XFDU Specifications – Issues and Proposals

  3. SAFE Format Specifications 1.5[SAFE-FMT]

  4. SAFE - Origin and goals (reminder) • Standard Archive Format for Europe • Provide ESA with a single format for all EO data • All Level 0 (Raw) and Level 1 • Level 2 (as far as applicable) • Ground Segment Auxiliary files) • Should become an ESA AIP • Candidate for SIP and DIP (TBV) • Shall be compatible with XFDU

  5. SAFE – 111 Data types to be supported NOAA (6)AVHRR/TOVS (6) IRS P3 (1)MOS (1) Nimbus (2)CZCS (2) MOS (1)MESSR (1) SPOT (3)HRV (1), HRVIR (1), HRG (1) • ENVISAT (59)ASAR (23), AATSR (5), DORIS (3), GOMOS (7), MERIS (13), MIPAS (6), MWR (1), RA2 (8), SCIAMACHY (5) • ERS (25)RA (4), AMI (12), ATSR (3), MWR (1), GOME (2), PRARE (3) • Landsat (3)MSS (1), TM (1), ETM+(1), RBV (0) • JERS (3)OPS (1), SAR (2) • Terra/Aqua (1)MODIS (1) • NOAA (6)AVHRR/TOVS (6) • Orbview (5)SeaWifs (5)

  6. SAFE, a restriction of XFDU <!-- SAFE Root element type. The type of the SAFE root derives from XFDU root type. --> <xsd:complexType name="safeType"> <xsd:complexContent> <xsd:restriction base="XFDUType"> <xsd:sequence> <xsd:element form="unqualified" name="packageHeader«  type="safe:packageHeaderType" minOccurs="1« />  <xsd:element form="unqualified" name="informationPackageMap" type="safe:informationPackageMapType" maxOccurs="1« />  <xsd:element form="unqualified" name="metadataSection" type="safe:metadataSectionType" minOccurs="1« />  <xsd:element form="unqualified" name="dataObjectSection" type="dataObjectSectionType" minOccurs="0« />  <xsd:element form="unqualified" name="behaviorSection" type="behaviorSectionType" minOccurs="0« /> </xsd:sequence> </xsd:restriction> </xsd:complexContent> </xsd:complexType> <!-- SAFE Root element --> <xsd:element name="safe" type="safe:safeType"/>

  7. XML Schemas / Namespace policy http://www.esa.int/safe/1.0 http://www.esa.int/safe/1.0/core-1.0.xsd http://www.esa.int/safe/1.0/passive-sensor-1.0.xsd http://www.esa.int/safe/1.0/active-sensor-1.0.xsd http://www.esa.int/safe/1.0/impl/xxx Specializations…

  8. SAFE Information Package Map Section One and only one Information Package Map Section

  9. SAFE – Core Metadata Objects • Acquisition Period Object • Platform Objects • Processing Log Objects • Quality Information Objects • Orbit Reference Objects • FrameSet Objects • XML Schemas Objects

  10. SAFE - Acquisition Period Object

  11. SAFE – Platform Object

  12. SAFE - Specialized Core Objects <!-- Platform description --> <metadataObject ID="platform" classification="DESCRIPTION" category="DMD"> <mdWrap textInfo="Platform Description" vocabluaryMdType="OTHER" otherMdType="SAFE" mimeType="text/xml"> <xmlData> <safe:platform> <!-- Platform identification --> <safe:nssdcId>2003-009A</safe:nssdcId> <safe:familyName>ENVISAT</safe:familyName> <!-- Instrument identification --> <safe:instrument> <safe:familyName>MERIS</safe:familyName> <meris:mode vcid="6" opsMode="4"/> </safe:instrument> </safe:platform> </xmlData> </mdWrap> </metadataObject>

  13. SAFE – Orbit Reference Object

  14. SAFE – Processing Log Object

  15. SAFE – Quality Information Use of XPath/XQuery foridentifying the corrupted/ Missing areas of theData Objects <!-- Quality information of the measurements --> <metadataObject ID="measurementQualityInformation" classification="DESCRIPTION" category="DMD"> <mdWrap textInfo="Quality Information" vocabluaryMdType="OTHER" otherMdType="SAFE" mimeType="text/xml"> <xmlData> <safe:qualityInformation> <!-- 24 records are missing from after the 1233rd record --> <safe:missingUnits> <safe:location>mdsr[1234]</safe:location> <safe:count>24</safe:count> </safe:missingUnits> <!-- The records 456 to 468 have a bad application ID --> <safe:corruptedUnits> mdsr[456 to 468]/isp/pkHd/PkId/appId </safe:corruptedUnits> </safe:qualityInformation> </xmlData> </mdWrap> </metadataObject>

  16. SAFE – Frame Sets

  17. SAFE – XML Schema Objects • All Data Objects shall have an associated XML Schemas • Post-processing (sub-setting, reformatting, …) • Annotation (quality information, cloud cover) • Indexing • Documentation / Presentation

  18. SAFE - Preservation of Binary Objects • Annotated XML Schemas • Structured Data File (SDF) markups <xsd:complexType> <xsd:sequence> <xsd:element name="record" type="xs:int" minOccurs="0" maxOccurs="unbounded"> <xsd:annotation> <xsd:appinfo> <sdf:block> <sdf:occurrence query="../header/numRec + 1"/> <sdf:length unit=‘bit’>12</sdf:length> </sdf:block> </xsd:appinfo> </xsd:annotation> </xsd:element> </xsd:sequence> <xsd:complexType/>

  19. SDF – Hierarchy of data blocks

  20. SDF - Location of a block within the stream

  21. SDF - Physical representation of a node

  22. SDF - Physical representation of a complex content

  23. SDF - Markups

  24. SDF - Example of Binary interpreter (Derby) Detailed content of an ENVISAT MERIStelemetry Source Packet(CCSDS standard)described by SDF

  25. Active Sensors Metadata Objects • Pulse Repetition Frequency • Polarization • Pulse Length

  26. SAFE Behavior Objects • No SAFE Behavior Object has been defined • Some may be added for configuring Stitching? • Should be investigated with CNES

  27. XFDU I/O library – [XFDU-I/O]

  28. XFDU I/O - General architecture

  29. Getting XFDU I/O library –Web site http://www.gael.fr/xfdu

  30. XFDU I/O Basic principles • Based on interactions with the MANIFEST file • Each interaction is not automatically comitted (possible rollback): the result shall be explicitely saved • 100% Java • C++ Wrapper based on Java Native Interface (JNI) • No support of Behavior section (to be investigated with CNES)

  31. XFDU I/O Implemented Usecases • Creation of a new XFDU packet • Opening of an existing XFDU packet • Removal of a XFDU packet • Moving a XFDU packet (with or without the attached objects) • Management of the Content Units • Management of Metadata Objects (only XML wrap) • Management of Data Objects (only Flocat references) • Browse Data Object Content (if associated XML Schema) • Validation of XFDU packet • Dumping XFDU Manifest in XML

  32. XFDU I/O – Configuration Status • Configuration identifiers: • safe-1-0-beta-2: first functional release • safe-1-0-beta-1: API only • Package identifier: esa.xfdu (TBV CNES)

  33. SAFE I/O library – [SAFE-I/O]

  34. SAFE I/O - General architecture

  35. Getting SAFE I/O library – SAFE Web site http://www.gael.fr/safe

  36. SAFE I/O Basic principles (Cont’) • Delegates standard XFDU access to ESA XFDU I/O • Based on interactions with the MANIFEST file • Each interaction is not automatically comitted (possible rollback): the result shall be explicitely saved

  37. SAFE I/O Implemented Usecases • Creation of a new SAFE packet • Opening of an existing SAFE packet • Removal of a SAFE packet • Moving a SAFE packet (with or without the attached objects) • Management of the Content Units • Management of the Core Metadata Objects • Adding specialized Metadata • Management of Data Objects • Extraction or pointing to a subpart of a Data Object • Identifying missing units and corrupted areas • Browse Data Object Content • Validation of SAFE packet • Dumping SAFE Manifest in XML

  38. SAFE I/O – Configuration Status • Configuration identifiers: • safe-1-0-release : an official release of version 1.0 • safe-1-0-rc2 : second release candidate of version 1.0 • safe-1-0-rc1 : a release candidate of version 1.0 • safe-1-0-beta-1: unconsolidated distribution of version 1.0 • Package identifier: esa.safe

  39. XFDU SpecificationsIssues and Proposals

  40. XFDU – Environment Information • What is teh exact definition of the environmentInfo/@specVersion attribute? • May the environmentInfo contain package history (e.g. SAFE processing logs)?

  41. XFDU – Information Package Map • What is the presice role of contentUnit/@order ? • GAEL understanding: user defined order • CNES understanding: package sequence number? • A Map section ?shall? Reference all metadata/data objects of the package? • Meaning of multiple Map sections?

  42. XFDU – Single file packages • For DIPs ESA requests single file products • ZIP, Jar or other standard archive format are not suitable for large data sets (e.g. 5 Gb for ENVISAT-ASAR format) • ESA proposes to append binary data objects to the MANIFEST file (not fully compliant with XML 1.0 or XML 1.0)

  43. XFDU – How to design restrictions? • The XML Schema requires global elements and attributes for restricting types in a different target namespace (E.g. SAFE) • May xfdu.xsd propose more global element for that purpose? • Should implementers redefine the xfdu.xsd? • May the XFDU specification provide a clarification/recommendation on that subject?

  44. XFDU – Metadata/Data Object references • metadataObject/@mdRef =? bytestream/@FLocat • metadataObject/@mdWrap =? bytestream/@FContent • May these differences be homogeneized?

  45. XFDU – Checksums • Should the CRC moved or add to the Flocat element (i.e. one CRC per file)? • What is the difference between dataObject/@checksum and bytestream/@checksum?

  46. XFDU – Reference to XLink • Requires an xlink.xsd in the same directory as the xfdu.xsd • References • http://www.w3.org/TR/XLink ? • http://www.w3.org/1999/XLink ? • Where to download the right XML Schema?

  47. XFDU - Typos • vocabluaryMdType ? • Comments of all items of the enumerations: e.g.EAD? • mdSecType may be renamed metadataSectionType?

  48. Thank you for your attention !

  49. XFDU – XML Schema Overview

  50. XFDU - Package Header

More Related