1 / 11

The HDF5 Virtual File Layer (VFL) and Virtual File Drivers (VFDs)

The HDF5 Virtual File Layer (VFL) and Virtual File Drivers (VFDs). Dana Robinson The HDF Group. Efficient Use of HDF5 With High Data Rate X-Ray Detectors Paul Scherrer Institut. HDF5 Library. HDF5 API. Virtual File Layer. VFD. VFD. User-Supplied VFD. d isk. HDF5 API.

ledell
Download Presentation

The HDF5 Virtual File Layer (VFL) and Virtual File Drivers (VFDs)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The HDF5Virtual File Layer (VFL)andVirtual File Drivers (VFDs) Dana Robinson The HDF Group Efficient Use of HDF5 With High Data Rate X-Ray Detectors Paul ScherrerInstitut

  2. HDF5 Library HDF5 API Virtual File Layer VFD VFD User-Supplied VFD disk

  3. HDF5 API internal generic I/O call H5FDwrite() Virtual File Layer function pointer Virtual File Driver VFD-specific I/O call H5FD_sec2_write() disk

  4. HDF5 VFDs log split multi family sec21(default) Windows2 STDIO3 core (in-memory) MPI MPI-POSIX "logical" VFDs which perform no I/O themselves 1) Uses POSIX I/O (sec2 = "POSIX section 2") 2) Currently a wrapper for SEC2. There is no driver which uses Win32 API calls. 3) "How to write a VFD" demo driver. Not intended for production use.

  5. The core VFD allows you to create/open HDF5 files in memory. HDF5 API H5FDwrite() Virtual File Layer disk Core VFD H5FD_core_write() memory File is (optionally) written to disk on close

  6. The family VFD allows you to split a logical HDF5 file among many smaller physical files. HDF5 API H5FDwrite() Virtual File Layer Family VFD H5FD_family_write() disk file A file B file C … file Z

  7. The multi (and split) VFD allows you to direct various categories of HDF5 data to different files and disks. Multi VFD H5FD_multi_write() global and local heap data, B-trees, object headers raw data superblock disk 1 disk 2 file A file B file C

  8. Selecting a "Terminal" VFD A "terminal" VFD does not require a second VFD 1) Create the property list. hid_tfapl_id = H5Pcreate(H5P_FILE_ACCESS); 2) Set the VFD using the appropriate API calls. herr_terr = H5Pset_fapl_sec2(fapl_id); 3) Create/open your file. hid_t fid = H5Fcreate("foo.h5", 0, H5P_DEFAULT, fapl_id);

  9. Selecting a "Logical" VFD A "logical" VFD requires a second, underlying VFD Step 1 Create a file access property list (FAPL) and set the VFD to use the terminal VFD. Step 2 Create a second FAPL and set the VFD to use the logical VFD (passing in the first VFD). Step 3 Open your file using the second (logical) FAPL. Logical VFD Terminal VFD Disk

  10. 1) Create the first property list. hid_tfapl_id = H5Pcreate(H5P_FILE_ACCESS); 2) Set the terminal VFD using the appropriate API calls. herr_terr = H5Pset_fapl_sec2(terminal); 3) Create the second file create/access property list. hid_tfapl_id = H5Pcreate(H5P_FILE_ACCESS); 4) Set the logical VFD using the appropriate API calls. herr_terr = H5Pset_fapl_family(logical, SIZE, terminal); 5) Create/open your file. hid_t fid = H5Fcreate("foo.h5", 0, H5P_DEFAULT, logical);

  11. Stackable VFDs Virtual File Layer Non-Terminal VFD Non-Terminal VFD Terminal VFD Disk

More Related