The Kangaroo Approach to Data Movement on the Grid

1. The Kangaroo Approach to Data Movement on the Grid Author: D. Thain, J. Basney, S.-C. Son, and M. Livny From: HPDC 2001 Presenter: NClab, KAIST, Hyonik Lee

2. Motivation • In many Grid applications, client usually gets data from data server, processes them and sends the results back to data server.

3. Problem • While data are being transferred to destination server, error or latency can occur due to • Server crash • Performance variation of server • Exhausted resources -> notresilient • The throughput of grid application is not good because the client should be blocked until the data are transferred in such a hostile environment.

4. Solution Approach • Use intermediary memory or disk buffer. • Background processes move data and handling errors. • Interface • Get • Put • Commit • push

5. Advantages • One-hop Kangaroo • Insulates the client form many difficulties • Network failure, destination machine crash • Exhausted resources (e.g. BW) • Read operation can be satisfied from cached data. • Multi-hop Kangaroo • Multiple hops help avoid the need to co-allocate network resources along all hops. • Multiple hops increase the available spooling space.

6. File System Architecture Application open, read, write, close, fsync Adaptation get, put, push, commit open, read, write, close, fsync Consistency Consistency Consistency get, put, commit, push get, put, push,commit ack ack Kangaroo Transport Kangaroo Transport Kangaroo Transport TCP/IP TCP/IP TCP/IP

7. Adaptation Layer • It is unreasonable to expect programmers to convert existing applications to work with Kangaroo. • Adaptation Layer converts standard POSIX operations into Kangaroo operations. • A number of operations are missing in the Kangaroo interface, but it is sufficient to admit many grid applications that simply must read and write data. read write fsync exit open without create open with create lseek close get put push commit Kangaroo POSIX Just update table

8. Consistency Layer • Commit causes the caller to block until all outstanding changes have been written to some stable storage. • Push causes the caller to block until all outstanding changes have been delivered to their respective destinations.

9. Kangaroo Transport Layer • Performance • Uses all available resources (net, memory, disk) to maximize throughput • Error management • Retry • Delay

10. Experiments

11. Blocking procedure call Non-blocking message