1 / 10

HDF5 Q4 Demo

HDF5 Q4 Demo. Architecture. Full VOL Plugin Support. Skeletal IOD Artificial Data Read at server New Features/Functionality: Attributes (H5A*) Links (H5L*) Objects (H5O*) Datatype Conversion: Done at server, avoids extra buffer allocation at client

sorcha
Download Presentation

HDF5 Q4 Demo

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HDF5 Q4 Demo

  2. Architecture Friday Seminar

  3. Full VOL Plugin Support • Skeletal IOD • Artificial Data Read at server • New Features/Functionality: • Attributes (H5A*) • Links (H5L*) • Objects (H5O*) • Datatype Conversion: • Done at server, avoids extra buffer allocation at client • Examples (BE to LE, 16 to 32 bit, etc…) • Non-contiguous memory selections: • Hyperslabs, point selections • Now supported with Mercury non-contiguous transfers

  4. Asynchronous Execution • Nearly all HDF5 API routines are completely asynchronous now • Dependencies built at the client, executed at the server • No wait at client for previous operations to complete to be able to forward a child operation • Few API routines still cause partly asynchronous behavior • Child operations need to wait for them to complete at the client. • A small number of routines have to be synchronous: • Iterate/visit • H5Oopen • Using event queue (EQ) objects now to manage requests instead of individual requests: • Very cumbersome to manage a request for every operation issued. • Event Queue can be though of as “bags” to wait/test a bunch of requests. • There is no effect on dependencies or influence on execution order for requests in an EQ. • User can still pop off the last inserted request in EQ to manage it separately.

  5. Unimplemented Functionality • Deferred • Iterate/visit (Does not make sense without a storage backend – Too much work to generate artificial metadata for them) • A few metadata query (H5*get_*) routines (same reason) • Variable Length Datatypes • Probably or Definitely will not support: • External Links - probably • References (H5R*) – probably • H5*_by_indexoperations – probably • File Mounting – definitely • H5Oopen_by_addr – definitely • Some Query operations (H5Dget_offset) – definitely

  6. Testing Status • Testing is improved: • Verifying new VOL routines are shipped and call the correct IOD routines at server. • Regression testing framework is setup (Cmake only now, autotools later). • Need to beef up regression testing, but won’t be able to do much until we get something for a storage backend. • Temporary Native HDF5 backend: • Call in to the native HDF5 storage format to verify routines and their parameters are shipped from the client and executed correctly at the server. • Very limited: • 1 client & 1 server possible for this now • Not asynchronous

  7. Data Integrity • Support in our level of the stack. • Added a public routine for applications to generate checksums: H5checksum(). • Added properties and routines to set properties for data integrity: • herr_tH5Pset_dxpl_checksum(hid_tdxpl_id, uint32_t value); • herr_tH5Pset_dxpl_checksum_ptr(hid_tdxpl_id, uint32_t *value); • herr_tH5Pset_dxpl_inject_corruption(hid_tdxpl_id, hbool_tflag);

  8. ACG Dynamic Data Structures • Added 4 new routines and their FF version: • herr_tH5DOappend(hid_tdataset_id, hid_tdxpl_id, unsigned axis, size_t extension, hid_tmemtype, constvoid *buffer); • herr_tH5DOsequence(hid_tdataset_id, hid_tdxpl_id, unsigned axis, hsize_tstart, size_tsequence, hid_tmemtype, void *buffer); • herr_tH5DOset(hid_tdataset_id, hid_tdxpl_id, consthsize_tcoord[], hid_tmemtype, • constvoid *buffer); • herr_tH5DOget(hid_tdataset_id, hid_tdxpl_id, consthsize_tcoord[],hid_tmemtype, void *buffer);

  9. H5DOappend() 3x5 Dataset H5DOappend(did, dxpl, 0, 3 , memtype, buf); H5DOappend(did, dxpl, 1, 2 , memtype, buf2); After 1: 6x5 Dataset After 2: 6x7 Dataset • H5DOsequence is very similar: • Read instead of write (no extension). • Accepts a starting position to read from.

  10. ACG support for Next Quarter • Support for VL Datatypes: • Each array element in the dataset corresponds to an IOD BLOB object • Add an Append only hint to the Dataset Creation property list: • Optimizes access and storage • Only 1 BLOB object needed • Each array element stores an offset into the BLOB object • Map Objects: • KV stores at the HDF5 API level.

More Related