1 / 44

Support for NPP/NPOESS by The HDF Group

Support for NPP/NPOESS by The HDF Group. Mike Folk, Elena Pourmal, Peter Cao The HDF Group June 30, 2009. To develop and adapt HDF5 software and provide HDF5 support for NPP/NPOESS. Goal. NPOESS Priorities and HDF5. Data calibration/validation Data accessibility and usability

chad
Download Presentation

Support for NPP/NPOESS by The HDF Group

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Support for NPP/NPOESS by The HDF Group Mike Folk, Elena Pourmal, Peter Cao The HDF Group June 30, 2009 NPOESS Data Formats Working Group

  2. To develop and adapt HDF5 software and provide HDF5 support for NPP/NPOESS. Goal NPOESS Data Formats Working Group

  3. NPOESS Priorities and HDF5 • Data calibration/validation • Data accessibility and usability • Tool development • System maintenance • User support NPOESS Data Formats Working Group

  4. Areas of immediate need • Ensure easy and intuitive data access and use by diverse communities • Enable producers and consumers to view content, manage metadata, and convert data to other formats. • Provide high-quality, rapid-response support of HDF5 for NPOESS users at all levels. NPOESS Data Formats Working Group

  5. Areas of immediate need • Ensure easy and intuitive data access and use by diverse communities Specifically: Develop high level APIs and tools to support NPP/NPOESS data management • Enable producers and consumers to view content, manage metadata, and convert data to other formats. Specifically: Modify HDFView to enable viewing of region references and quality flags • Provide high-quality, rapid-response support of HDF5 for NPOESS users at all levels. NPOESS Data Formats Working Group

  6. Update on High-Level APIs to support NPOESS data NPOESS Data Formats Working Group

  7. Overview NPOESS Data Formats Working Group

  8. Overview • Under current agreement The HDF Group will • Propose, develop, and implement approaches to access data pointed to by region references, rewrite region references in datasets and access individual quality flags. • Gather user needs and match to proposed plan, design, and design implementation. Address requirements, and inform the user communities on NPOESS HDF5 implementation and use. • This part of the presentation describes our progress on those two tasks. NPOESS Data Formats Working Group

  9. Overview • The HDF Group has been gathering requirements through • HDF-EOS and HDF Workshops and other conferences and meetings • The HDF Group HelpDeskhelp@hdfgroup.org • Personal contacts • The result of this work is a set of 10 new High-Level APIs to support data access in NPOESS files NPOESS Data Formats Working Group

  10. Overview • Links to RFCs and source code: • Current version of RFCs and feedback from the users http://hdf5dev.pbworks.com/NPOESS-RFCs • Snapshots of the HDF5 library code ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/NPOESS/source • Tested on Linux 32 and 64-bit, Solaris and Windows XP • Repository http://svn.hdfgroup.uiuc.edu/hdf5/branches/NPOESS NPOESS Data Formats Working Group

  11. Overview • We need your help to • Evaluate current APIs • Use them • Brake them • Tell us what you like and do not like including documentation • Give feedback on the implemented features • Is anything missing? • Did we misinterpret some of the requirements? • Phase II work will be based on feedback we get from you! NPOESS Data Formats Working Group

  12. New APIs Highlights NPOESS Data Formats Working Group

  13. New APIs Highlights • 10 new C APIs to support HDF5 region references and packed bits in support of NPOESS quality flags • New APIs are wrappers around several HDF5 library calls • The latest versions of the documentation are available from http://hdf5dev.pbworks.com/NPOESS-RFCs • HL APIs • Region References • Bit-fields • Source code is based on the HDF5 development branch • Tested daily on Linux 32 and 64-bit, Solaris, and weekly on Windows XP NPOESS Data Formats Working Group

  14. New APIs Highlights • There will be no decision made which APIs will be brought back to the mainstream HDF5 library until we hear from NPOESS and general HDF5 community. • Consider current version as a PROTOTYPE that will help to gather and refine requirements. NPOESS Data Formats Working Group

  15. New APIs Highlights • New region reference APIs • Query information about data • Get information about data pointed to by region reference (full path of the datatset, datatype, dimensionality, description of selected region, i.e., corners of the rectangular region.) • Access region reference raw data • Read data pointed to by a region reference to an application buffer. • Read data described by its location in the dataset to an application buffer. • Create dataset with raw data pointed by the array of region references. NPOESS Data Formats Working Group

  16. New APIs Highlights • New region reference APIs (continue) • Create region references • Create array of region references using list of datasets names and hyperslab descriptions • Traverse a group tree and create a new dataset with region references to all found datastes • Gather raw data in one dataset • Copy raw data pointed by region references to a specified location • Copy references • Copy raw data pointed by region reference to a specified location and create a reference to it NPOESS Data Formats Working Group

  17. New APIs Highlights • New packed bit-field APIs • Extracts bit-field values from a dataset into application buffer • Deficiencies: • HDF5 doesn’t have a knowledge about the meaning of a bit-field and currently doesn’t store any information about the packed data • NPOESS stores information in a separate file that user has to access to find and interpret bit-filed data • HDF5 provides several way for storing “low range data” • N-bit data • Different compression methods • This API is very specific to NPOESS data NPOESS Data Formats Working Group

  18. Update on h5dump NPOESS Data Formats Working Group

  19. h5dump and region reference data • Added new flag –R to display data pointed by region references • Can be used in combinations with other flags to display only one dataset, or to display selection in the dataset, or export data to a text or binary file NPOESS Data Formats Working Group

  20. h5dump example > h5dump FileA.h5HDF5 "FileA.h5" {... DATASET "R1" { DATATYPE H5T_REFERENCE DATASPACE SIMPLE { ( 4 ) / ( 4 ) } DATA { (0): DATASET /Group_1D/DS1 {(1), (6), (9), (10), (14)}, ... (3): DATASET /Group_3D/DS3 {(3,3,3)-(4,4,4)} } }… NPOESS Data Formats Working Group

  21. h5dump example (first element) > h5dump –R FileA.h5HDF5 "FileA.h5" { ...DATASET "R1" { DATATYPE H5T_REFERENCE DATASPACE SIMPLE { ( 4 ) / ( 4 ) } DATA { (0): DATASET /Group_1D/DS1 { (0): REGION_TYPE POINT (1), (6), (9), (10), (14) (0): DATATYPE H5T_STD_I32LE (0): DATASPACE SIMPLE { ( 17 ) / ( 17 ) }(0): DATA { (1): 1, (6): 6, (9): 9, (10): 10, (14): 14 … NPOESS Data Formats Working Group

  22. h5dump example (fourth element) > h5dump –R FileA.h5HDF5 "FileA.h5" { ...(3): DATASET /Group_3D/DS3 { (3): REGION_TYPE BLOCK (3,3,3)-(4,4,4) (3): DATATYPE H5T_STD_I32LE (3): DATASPACE SIMPLE { ( 6, 6, 6 ) / ( 6, 6, 6 ) } (3): DATA { (3,3,3): 444, 445, (3,4,3): 454, 455, (4,3,3): 544, 545, (4,4,3): 554, 555 (3): } (3): } … NPOESS Data Formats Working Group

  23. h5dump and packed bit-field data • Added new flag –M <L> to display packed bit-field data, where <L> is a paired list of integers the first number of which is the offset and the second number is the number of bits to display • Deficiencies: • Information in <L> has to be provided by a user • HDF5 library doesn’t have a knowledge how data is packed. NPOESS Data Formats Working Group

  24. h5dump example > h5dump -d /Data_Products/Gran_1 sds.h5HDF5 "sds.h5" { DATASET "/Data_Products/Gran_1" { DATATYPE H5T_STD_U8BE DATASPACE SIMPLE { ( 768, 3200 ) / ( H5S_UNLIMITED, H5S_UNLIMITED ) } DATA { (0,0): 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, (0,19): 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, (0,38): 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,... (767,3191): 65533, 65533, 65533, 65533, 65533, 65533, 65533, (767,3198): 65533, 65533 } }} NPOESS Data Formats Working Group

  25. h5dump example > h5dump -d /Data_Products/Gran_1 -M 0,2 sds.h5HDF5 "sds.h5" { DATASET "/Data_Products/Gran_1" { DATATYPE H5T_STD_U8BE DATASPACE SIMPLE { ( 768, 3200 ) / ( H5S_UNLIMITED, H5S_UNLIMITED ) }PACKED_BITS OFFSET=0 LENGTH=2 DATA { (0,0): 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, (0,19): 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, (0,38): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,... (767,3191): 3, 3, 3, 3, 3, 3, 3, (767,3198): 3, 3 } }} NPOESS Data Formats Working Group

  26. Update on HDFViewSupport Region References and Quality Flags NPOESS Data Formats Working Group

  27. NPOESS Region References • Data products are organized by : • Reference objects (aggregations). • Reference regions (granules). NPOESS Data Formats Working Group

  28. NPOESS Quality Flags • NPOESS quality flags: • Quality flags are qualifications of one or more consecutive bits in each byte. • The number of arrays is dependant on the number and size of each data product's quality flag definitions. • Multiple bit-level flags are packed in a single byte. • Byte alignment is the same for every quality flag array: first bit is the LSB. NPOESS Data Formats Working Group

  29. HDFView 2.5 : Region References HDFView 2.5 or earlier versions: • Show region references as strings. • Nothing more, you cannot do anything with the strings. NPOESS Data Formats Working Group

  30. Phase-1 Work: Region References Show data pointed by a region reference: • Double click to show data in a separate table. • Use different border color and text title to distinguish the table. NPOESS Data Formats Working Group

  31. Phase-1 Work: Region References Options to show data in image or table • Right click to show a popup window to show data in table or image. NPOESS Data Formats Working Group

  32. Phase-1 Work: Region References Open multiple cells at one time: • Select multiple cells • Drag mouse to select continuous cells. • CTR + mouse click to select non-continuous cells. • Right-mouse click to open selected cell at one time. NPOESS Data Formats Working Group

  33. Phase-1 Work: Region References Show multiple regions pointed by a single reference: • Separate table to show each region. NPOESS Data Formats Working Group

  34. Phase-1 Work: Quality Flags Use bitmask to show individual quality flags: • Bitmask only applies to byte data. • By default, show quality flags as 8-bit decimal integers. NPOESS Data Formats Working Group

  35. Phase-1 Work: Quality Flags Use bitmask to extract quality flags: • Use “Open As” on the dataset of quality flags. • Enter bitmask, e.g. 00000011. • Click “Help” icon for help. NPOESS Data Formats Working Group

  36. Phase-1 Work: Quality Flags • The mask shown at top-right, e.g. {0, 1} NPOESS Data Formats Working Group

  37. Phase-1 Work: Quality Flags • Data can be displayed in decimal, hexadecimal, or binary. NPOESS Data Formats Working Group

  38. Phase-1 Work: What Is Left • Open data pointed by object references (aggregation). • Show the name/path of the dataset pointed by a reference. • Add check boxes to select specific bit(s) (or mask). NPOESS Data Formats Working Group

  39. Phase-2 Work • So far, the right click menu on the object IDs gives me the options: "Show as Table" and "Show as Image". I wonder if the menu could be closer to the right click menu on an All_Data dataset. That is, give me the "Open", "Open as", "Copy", "Save to", "Show Properties" ... but only the ones that makes sense. The "Delete" might not apply since that would alter the structure of the references • Use definition of quality flgas, which are in two separate locations: 1) The Common Data Format Control Book – External (CDFCB-X), Volumes II, III and IV. 2) XML files provided as part of the CDFCB-X delivery. • Apply the structure of the XML to determine whether a field is a packed quality flag byte. • Production quality: testing, documentation, and distribution. NPOESS Data Formats Working Group

  40. Phase-2 Work: plugin Two options • Part of general HDFView • Separate plugin NPOESS Data Formats Working Group

  41. Phase-2 Work: plugin Part of general HDFView • Features for region references and quality flags are part of the HDFView. • Pros: • No extra work for release and distribution. • Cons: • Add extra work to maintain the general HDFView (changing/adding NPOESS features will affect the whole HDFView). • Some features may not be needed for general users. NPOESS Data Formats Working Group

  42. Phase-2 Work: plugin Separate plugins • the features will be implemented in separate plugins, which will be added to the general HDFView as requested at runtime. • Pros: • Modify (add/change features) will not affect the general HDFView. • Easy to maintain (change/add/delete) NPOESS related features. • Cons: • Extra work for distributing/maintaining the source and binaries for the plugins. NPOESS Data Formats Working Group

  43. Phase-2 Work: plugin To support plugin • Need to extend the NPOESS plugins from default implementation so that changes (new features and bug fixes) in the general HDFView will be automatically applied to the plugins. • Need to redesign/implement the current default implementation of TreeView and TableView. The current default implementation is not extendable. This work will be done in phase-2. • For phase-1 plugin, we will copy the default implementation source code and add the new features to copy. NPOESS Data Formats Working Group

  44. NPOESS Data Formats Working Group

More Related