HDF5 and AIO

1. HDF5 and AIO Current status and future work HDF5 Workshop at PSI

2. HDF5 and AIO • Asynchronous I/O (AIO), or non-blocking I/O, is a form of input/output processing that permits other processing to continue while the transmission occurs (i.e., overlapping compute with I/O) • Current HDF5 I/O calls are synchronous or blocking • On read, call doesn’t complete until the desired data has been read from the file and written to the application buffer • On write, the call doesn’t complete until the outgoing data buffer has been transferred to OS HDF5 Workshop at PSI

3. HDF5 and AIO • HDF5 may hide most of I/O overhead under application computation • Support asynchronous I/O access to data in HDF5 file: • I/O is initiated within the library in response to an API call • I/O operation completes in the backgroundafter API call has returned • Beneficial for both raw data and HDF5 metadata I/O HDF5 Workshop at PSI

4. Implementation considerations • Based on POSIX Asynchronous I/O routines • APIs for applications to initiate write/read/file sync • Return immediately without waiting for requested I/O operation to complete • Facilities to: • Query OS to determine if AIO operation is complete • Stall, pending completion of AIO operation HDF5 Workshop at PSI

5. Implementation considerations • Requirements of AIO library • Portability • Robustness • System libraries we tested didn’t perform well and in some cases were not even POSIX compliant! • We have been working with the “AIO-Lite” library from Argonne built on top of Pthreads • Preliminary tests show significant reduction in application I/O time for AIO HDF5 Workshop at PSI

6. Current status • Modifications completed to HDF5 Virtual File Layer • Define a set of new VFD calls (general enough to support different AIO implementation, e.g., POSIX AIO and MPI non-blocking calls) • Initiating of asynchronous read, write, file sync • Obtaining status of an asynchronous operation • Blocking pending completion of an asynchronous operation • Finishing an asynchronous operation • Canceling an asynchronous operation HDF5 Workshop at PSI

7. Current status • Modify H5FD_class_t in H5FDpublic.h to define these calls as optional calls that file drivers may choose to implement: herr_t (*aio_read)(H5FD_t *file, …); herr_t(*aio_write)(H5FD_t *file, …); herr_t(*aio_test)(hbool_t *done_ptr, …); herr_t(*aio_wait)(void *ctlblk_ptr); herr_t(*aio_finish)(int *errno_ptr, …); herr_t(*aio_fsync)(H5FD_t *file, …); herr_t(*aio_cancel)(void *ctlblk_ptr); herr_t(*fsync)(H5FD_t *file, …); HDF5 Workshop at PSI

8. Current status • Modifications completed to HDF5 Virtual File Layer (continued) • Implement VFD AIO calls as top level VFD calls (sec2, etc.) • If underlying driver supports the desired AIO operation, these functions just pass request to it • Otherwise simulate AIO by translating the required operations into functionally equivalent SIO • Modify the family and multi file drivers to implement AIO VFD calls by passing AIO VFD calls to the underlying file drivers HDF5 Workshop at PSI

9. Current status • Modifications completed to HDF5 Virtual File Layer (continued) • Modify configure to enable and control the AIO extensions to file drivers • --enable/disable_aio • --enable/disable_64_bit_posix_aio • --enable/disable_posix_aio_error_recovery • Testing • Code is available from https://svn.hdfgroup.uiuc.edu/hdf5/branches/aio_vfd/ HDF5 Workshop at PSI

10. Future work • Design and implement clients to use AIO for writing HDF5 metadata and raw data • Design and implement public APIs to control AIO • Those tasks are in a planning stage HDF5 Workshop at PSI

11. Thank You! Questions? HDF5 Workshop at PSI