Coupling Parallel IO with Remote Data Access

1. Coupling Parallel IO with Remote Data Access Ekow Otoo, Arie Shoshani, Doron Rotem, and Alex Sim Lawrence Berkeley National Lab.

2. Outline • Project objectives • Status and accomplishments • Usage in an application • Extensions • Other future work SciDAC All Hands Meeting October’05

3. Project Objectives • Development of the MpiioSrm library • mpiiosrm.h libmpiiosrm.a, libmpiiosrm.so • Allows near-online access to files on mass storage system (e.g., HPSS), from MPI applications on a linux cluster • Access files from local and remote MSS with MPI applications. • Applications on a Linux cluster having • Local parallel file system (PVFS2) and • HPSS as the remote mass storage system SciDAC All Hands Meeting October’05

4. HPSS pNetCDF HDF5 SAM MPI-IO SRM Jasmin Castor BDB Other Status – 1Libmpiiosrm Module Dependencies MPI Applications High Level Access and Control mpiiosrm Record Structured File Access Low Level File System Access ADIO PVFS2 GPFS UFS XFS Other SciDAC All Hands Meeting October’05

5. Status – 2Main Functions • Functions in libmpiiosrm.a: • MPI_File_srm_proxy_init(); • MPI_File_srm_open(); [in place of MPI_File_open()] • MPI_File_srm_close(); [in place of MPI_File_close()] • MPI_File_srm_delete(); [in place of MPI_File_delete()] • MPI_File_srm_proxy_destroy() • Functions (2) and (4) take a file name as one of its parameters. • Note name changes from last meeting. SciDAC All Hands Meeting October’05

6. Status – 3Major Changes Since Last AHM • Function names revised • MPI_File_srm_proxy_init() function starts an SRM client as a detached thread. • Only the process with the proxy_rank spawns this thread. • Use of PVFS2 • srm_put() implemented for MPI_File_writes, i.e., • Files can now migrate from PVFS2 to HPSS. • Still being tested. SciDAC All Hands Meeting October’05

7. Usage in an Application • Steps for reading remote files: • Prepare an input file for the program • A file containing the file names to be read from HPSS if not found in local parallel file system. • Initiate grid-proxy-init(); password, etc. • User requires a grid certificate • Start a namesrver, drmServer and a trmServer • Compile the program to be executed • Run “mpiexec –n XX <progname>” to access files given in the input file. SciDAC All Hands Meeting October’05

8. Usage in an ApplicationInput file Layout • Implicit layout of parallel files • Uses default PVFS configuration • Alternatively use keys of MPI-IO File hints • Specify only pairs of source and destination URLs • Explicit layout specifies for each file: • Pairs of source and destination URL • Start_IO_Node • Striping factor • Striping unit SciDAC All Hands Meeting October’05

9. Usage in an ApplicationProgram Skeleton … MPI_Init(); … MPI_Info_create(); MPI_Info_set(); … MPI_File_srm_proxy_init(); … MPI_File_srm_open(); … <Process file read with standard MPI_File_* operations> … MPI_File_srm_close(); MPI_File_srm_proxy_destroy(); … MPI_Finalize(); SciDAC All Hands Meeting October’05

10. HPSS SAM Jasmin Castor Extensions • File Control • Data vs File Access • Multi-site Access • Fault tolerance & Failsafe MPI Applications Srm-Server Srm- -Client MPI-IO DRM TRM SciDAC All Hands Meeting October’05

11. Extensions - 2 • Control of Prefetching of File Bundles • Process files, one at a time, by availability • Process files, one at a time, by sequence • Process files by bundles • Data Access instead of File Access only • Allow for file filtering at the source SRM • Use of select criteria and indexes to generate only relevant data SciDAC All Hands Meeting October’05

12. Extension - 3 • Multi-Site Access • Extend access to other MSS implementing SRM specs. • Access files from multiple sites in a session • Extensions to Xrootd servers • Fault Tolerance and Failsafe Operations • Easier now with multiple srm_client proxies being spawn as threads • Access from C++ and Fortran SciDAC All Hands Meeting October’05

13. Other Future Work • Parallel Multidimensional Index Schemes • Repertoire of high and low dimensional indexing methods for scientific applications • High dimensions: • Bitmaps (John, Kurt, etc) • Others • Low Dimensions (1 ~ 8) • R-Tree, Order Preserving Extendible hashing, • Multi-level Grid File • String Searching Methods – Suffix trees, PATRICIA, etc. SciDAC All Hands Meeting October’05

14. Other Future Work - cont. • Extendible Multidimensional Array Files • With extendibility in all dimensions, not just one • For both dense arrays and sparse arrays • Efficiently accessible in MPI with irregular distributed array method using map arrays. • Multi-resolution array files SciDAC All Hands Meeting October’05

15. Other Proposed Activities cont. SciDAC All Hands Meeting October’05

16. Example of Mapping Function 0 1 2 3 4 5 6 7 8 9 i1 0 1 0 1 2 12 16 35 42 49 72 81 0 3 4 5 13 17 36 43 50 73 82 1 6 7 8 14 18 37 44 51 74 83 2 5 4 3 9 10 11 15 19 38 45 52 75 84 >= >= < < 8 4 39 46 53 76 85 20 21 22 23 24 7 3 3 1 0 0 40 47 54 86 5 25 26 27 28 29 77 < 41 48 55 6 30 31 32 34 77 87 33 1 9 72 8 1 4 20 4 56 57 58 59 60 61 62 63 78 88 7 1 8 1 56 7 6 35 5 8 64 65 66 67 68 69 70 71 79 89 Red-Black-Like Binary Tree i0 SciDAC All Hands Meeting October’05

17. The End SciDAC All Hands Meeting October’05