the kangaroo approach to data movement on the grid l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
The Kangaroo Approach to Data Movement on the Grid PowerPoint Presentation
Download Presentation
The Kangaroo Approach to Data Movement on the Grid

Loading in 2 Seconds...

play fullscreen
1 / 9

The Kangaroo Approach to Data Movement on the Grid - PowerPoint PPT Presentation


  • 466 Views
  • Uploaded on

The Kangaroo Approach to Data Movement on the Grid Douglas Thain, Jim Basney, Se-Chang Son, and Miron Livny http://www.cs.wisc.edu/condor/kangaroo “Half-Interactive” Jobs on the Grid The “Grid” is a vision of an international, highly available, robust, shared computation service.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'The Kangaroo Approach to Data Movement on the Grid' - LeeJohn


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
the kangaroo approach to data movement on the grid

The Kangaroo Approach to Data Movementon the Grid

Douglas Thain, Jim Basney,

Se-Chang Son, and Miron Livny

http://www.cs.wisc.edu/condor/kangaroo

half interactive jobs on the grid
“Half-Interactive” Jobson the Grid
  • The “Grid” is a vision of an international, highly available, robust, shared computation service.
  • Grid applications generally…
    • ..are half-interactive.
    • ..are brittle wrt errors.
    • ..have weak consistency requirements.
  • Can we perform on-line data access without subjecting apps to unexpected errors?

App

Unreliable

Network

slide3

The Kangaroo Approach

To Data Movement

  • Make a third party responsible for executing each application’s I/O operations.
  • Use all available resources to hide latencies.
  • Never return an error to the application.
    • (Maybe tell the user or scheduler.)
  • Benefits: Fault tolerance, higher throughput.
  • Cost: Weaker consistency.
the vision a data grid

App

File

System

File

System

File

System

File

System

The Vision: A Data Grid

K

K

K

Data Movement

System

K

K

K

K

Disk

kangaroo prototype
Kangaroo Prototype

An application may contact any node in the system and perform partial-file reads and writes.

The node may then execute or buffer operations as conditions warrant.

K

K

K

App

Buffer

Buffer

Disk

user interface
User Interface
  • An interposition agent converts POSIX operations into the Kangaroo protocol.

% vi /kangaroo/coral.cs.wisc.edu/etc/hosts

% gcc -c /gsiftp/ftp.cs.wisc.edu/test.c -o

/kangaroo/coral/tmp/test.o

K

Agent

App

read, write

get, put

Kangaroo

POSIX

macrobenchmark image processing
Macrobenchmark:Image Processing
  • Post-processing of satellite image data: Need to compute various enhancements and produce output for each.
    • Read input image
    • For I=1 to N
      • Compute transformation of image
      • Write output image
  • Example:
    • Image size about 5 MB
    • Compute time about 6 sec
    • IO-cpu ratio about 0.9 MB/s
i o models compared
I/O Models Compared

CPU Released

Task Done

Offline I/O:

INPUT

CPU

CPU

CPU

CPU

OUTPUT

OUTPUT

OUTPUT

OUTPUT

CPU Released

Online I/O:

INPUT

CPU

OUTPUT

CPU

OUTPUT

CPU

OUTPUT

CPU

OUTPUT

CPU Released

Task Done

Task Done

Kangaroo:

INPUT

CPU

CPU

CPU

CPU

PUSH

OUTPUT

OUTPUT

OUTPUT

OUTPUT