1 / 9

The Kangaroo Approach to Data Movement on the Grid

The Kangaroo Approach to Data Movement on the Grid Douglas Thain, Jim Basney, Se-Chang Son, and Miron Livny http://www.cs.wisc.edu/condor/kangaroo “Half-Interactive” Jobs on the Grid The “Grid” is a vision of an international, highly available, robust, shared computation service.

LeeJohn
Download Presentation

The Kangaroo Approach to Data Movement on the Grid

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Kangaroo Approach to Data Movementon the Grid Douglas Thain, Jim Basney, Se-Chang Son, and Miron Livny http://www.cs.wisc.edu/condor/kangaroo

  2. “Half-Interactive” Jobson the Grid • The “Grid” is a vision of an international, highly available, robust, shared computation service. • Grid applications generally… • ..are half-interactive. • ..are brittle wrt errors. • ..have weak consistency requirements. • Can we perform on-line data access without subjecting apps to unexpected errors? App Unreliable Network

  3. The Kangaroo Approach To Data Movement • Make a third party responsible for executing each application’s I/O operations. • Use all available resources to hide latencies. • Never return an error to the application. • (Maybe tell the user or scheduler.) • Benefits: Fault tolerance, higher throughput. • Cost: Weaker consistency.

  4. App File System File System File System File System The Vision: A Data Grid K K K Data Movement System K K K K Disk

  5. Kangaroo Prototype An application may contact any node in the system and perform partial-file reads and writes. The node may then execute or buffer operations as conditions warrant. K K K App Buffer Buffer Disk

  6. User Interface • An interposition agent converts POSIX operations into the Kangaroo protocol. % vi /kangaroo/coral.cs.wisc.edu/etc/hosts % gcc -c /gsiftp/ftp.cs.wisc.edu/test.c -o /kangaroo/coral/tmp/test.o K Agent App read, write get, put Kangaroo POSIX

  7. Macrobenchmark:Image Processing • Post-processing of satellite image data: Need to compute various enhancements and produce output for each. • Read input image • For I=1 to N • Compute transformation of image • Write output image • Example: • Image size about 5 MB • Compute time about 6 sec • IO-cpu ratio about 0.9 MB/s

  8. I/O Models Compared CPU Released Task Done Offline I/O: INPUT CPU CPU CPU CPU OUTPUT OUTPUT OUTPUT OUTPUT CPU Released Online I/O: INPUT CPU OUTPUT CPU OUTPUT CPU OUTPUT CPU OUTPUT CPU Released Task Done Task Done Kangaroo: INPUT CPU CPU CPU CPU PUSH OUTPUT OUTPUT OUTPUT OUTPUT

More Related