GAT
This presentation is the property of its rightful owner.
Sponsored Links
1 / 37

GAT PowerPoint PPT Presentation


  • 95 Views
  • Uploaded on
  • Presentation posted in: General

GAT. Alexander Beck-Ratzka. OFFIS, 23.06.06. GAT: Agenda. Introduction Why a new Grid-API? What GAT offers, what not? Examples Architecture The single API groups File (-stream) management, logical file management, AdvertService management, Job management Adaptors Conclusion.

Download Presentation

GAT

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Gat

GAT

Alexander Beck-Ratzka

OFFIS, 23.06.06

GAT


Gat agenda

GAT: Agenda

  • Introduction

    • Why a new Grid-API?

    • What GAT offers, what not?

    • Examples

    • Architecture

  • The single API groups

    • File (-stream) management, logical file management,

    • AdvertService management,

    • Job management

  • Adaptors

  • Conclusion

GAT


Why a new grid api

Why a new Grid-API?

GAT as a uniform API for the access to heterogeneous grid

technologies / grid middleware.

GAT is only a framework; the real grid operationen must be done

by adaptors. GAT offers the possibility for accessing adpaptors.

New grid technologies only needs to be coupled to GAT via

adaptors -> No changes in the source sode necessary, even not

If using new grid technologies.

GAT enables a simple access to grid technologies.

GAT uses the grid middleware, which is available at the moment, where is should be used, and this by only typing a single statement for GAT_JobSubmit or GATFileCopy...

GAT


What gat offers 1

What GAT offers? (1)

GAT is not a new grid middleware.

  • GAT enables an easy to different grid infrastructures.

    • GRAM

    • Condor

    • Unicore

    • GridFTP

    • ...

GAT is an OpenSource project.

GAT


What gat offers 2

What GAT offers? (2)

  • Applications call the GAT-API for grid operations.

    • Applications hav to be linked against the GAT.

  • Applications are independend of the underlying grid infrastructure.

    • GAT Engine loads available adaptors at runtime

    • During a calls of the GAT-API the GAT engine decides, which adaptor performs the grid operation.

    • In error case during a „grid operation“, another adaptor will be accessed.

  • default adaptors offer local operations

    • grid applications can be compiled, linked and tested without available grid services.

    • The same application can run in a „complete grid environment“: without a new build.

GAT


What gat offers 3

What GAT offers? (3)

The GAT-API doesn‘t change. Changes e.g. in Globus job submit are included in the GAT Globus adaptor.

GAT offers reliability. If one grid service is not available, another grid service is used.

GAT is much eaysier to install as Globus.

GAT offers grid with a minimum effort for the end user.

GAT


What gat not offers

What GAT not offers?

GAT doesn‘t replace any function of a grid middleware.

Without an adaptor for a grid middleware GAT is useless for this middleware.

GAT has no resource broker.

GAT


Examples java gat

Examples (Java-GAT)

File operations

Job submit

GAT


File copy java gat

File copy: Java-GAT

import org.gridlab.gat.*;

import org.gridlab.gat.io.File;

public class RemoteCopy

{

public static void main(String[] args) throws Exception

{

GATContext context = new GATContext();

URI src = new URI(args[0]);

URI dest = new URI(args[1]);

File file = GAT.createFile(context, src); // create file object

file.copy(dest); // and copy it

GAT.end();

}

}

GAT


File write java gat

File Write (Java-GAT)

import java.io.PrintWriter;

import org.gridlab.gat.GAT.*;

import org.gridlab.gat.io.FileOutputStream;

public class FileStreamSimple {

public static void main(String[] args) {

GATContext context = new GATContext();

URI src = null;

FileOutputStream stream = null;

PrintWriter p;

src = new URI(args[0]);

stream = GAT.createFileOutputStream(context, null, src);

p = new PrintWriter(stream);

String toBeStreamed = "hello world\n";

p.println(toBeStreamed);

p.close();

GAT.end();

}

}

GAT


Job submit java gat

Job Submit (Java-GAT)

public class SubmitJobToHost {

public static void main(String[] args) {

GATContext context = new GATContext();

Preferences prefs = new Preferences();

prefs.put("ResourceBroker.adaptor.name", "globus");

prefs.put("ResourceBroker.jobmanager", "sge");

URI exe = null;

File outFile = null;

File errFile = null;

File pre1 = null;

File pre1Dest = null;

try {

exe = new URI("file:////bin/hostname");

outFile = GAT.createFile(context, prefs, new URI(

"any://fs0.das2.cs.vu.nl/out"));

errFile = GAT.createFile(context, prefs, new URI(

"any://fs0.das2.cs.vu.nl/err"));

pre1 = GAT.createFile(context, prefs, new URI(

"any://fs0.das2.cs.vu.nl//bin/echo"));

pre1Dest = GAT.createFile(context, prefs, new URI(

"any://fs0.das2.cs.vu.nl//home/rob/my_temp_file"));

} catch (Exception e) {

System.err.println("error: " + e);

System.exit(1);

}

SoftwareDescription sd = new SoftwareDescription();

sd.setLocation(exe);

sd.setStdout(outFile);

sd.setStderr(errFile);

sd.addPreStagedFile(pre1, pre1Dest);

Hashtable hardwareAttributes = new Hashtable();

hardwareAttributes.put("machine.node", "fs0.das2.cs.vu.nl");

ResourceDescription rd = new HardwareResourceDescription(

hardwareAttributes);

JobDescription jd = null;

ResourceBroker broker = null;

try {

jd = new JobDescription(sd, rd);

broker = GAT.createResourceBroker(context, prefs);

} catch (Exception e) {

System.err.println("Could not create Job description: " + e);

System.exit(1);

}

Job job = null;

try {

job = broker.submitJob(jd);

} catch (Exception e) {

System.err.println("submission failed: " + e);

e.printStackTrace();

System.exit(1);

}

GAT


Gat architecture

GAT architecture

API thin layer; only a frame.

Adaptor implements “grid functionality” as “ordered” by the API.

Engine mediates betwen API u. adaptor:

  • the adaptors are called at runtime

  • error tracing and “Fallback” (default local adaptors)

    CPI for Implementation of single adaptors.

GAT


Gat architecture1

Application layer

Application

GAT layer

GAT API

GAT Engine

GAT Adapter

GTK4

PBS

SGE

Globus 2/3.x

DRMAA

Unicore

GAT architecture

User Space

„Grid“ Space

GAT


Implementation engine

Implementation (Engine)

C version fully implemented

C++ wrapper fully implemented

Python wrapper fully implemented

JAVA version fully implemented

GAT


Implementation adaptors c gat

Implementation (Adaptors) C-GAT

Globus:

  • gram, gridftp, RLS, gsiscp, gsissh

Unicore:

  • Job Submit, Job Monitoring

DRMAA (Distributed Resource Management Application API)

SGE (Sun Grid Engine)

PBS (Portable Batch System)

GAT


Implementation adaptors java gat

Implementation (Adaptors) Java-GAT

Globus:

  • With Java CoG Kit; Globus 3.y complete, GTK 4 up to now without WebServices.

Unicore:

  • In work

SGE

PBS

GAT


Implementation adaptors java gat and c gat

Implementation (Adaptors) Java-GAT and C-GAT

Local adpaptors:

  • ssh, scp, ftp, sftp,

  • File-Adapter: (cp, mv, read, write, etc...)

  • Job-Adapter: fork, exec, auch über ssh...

GAT


File copy schematic run through

File.copy(dest)

FileCPI.copy(dest)

Adapter1 copy

Adapter2 copy

Adapter3 copy

File.copy: schematic run through

GAT


Gat api overview

GAT API overview

File operations

(Monitoring and events)

Resources, jobs

(Information exchange)

(Utility classes: error handling, security, preferences)

GAT


File sub system gatfile class

File sub systemGATFile class

GATObject

GATFile

Create

Destroy

Copy

Move

Delete

IsReadable

IsWritable

GetLength

LastWriteTime

GetLocation

GAT


File sub system gatfilestream class

File sub systemGATFileStream class

GATObject

GATFileStream

Create

Destroy

Read

Write

Seek

GAT


File sub system gatlogicalfile class

File sub systemGATLogicalFile class

GATObject

GATLogicalFile

Create

Destroy

GetFiles

GetLocations

Remove

RemoveFile

AddFile

AddLocation

Replicate

GAT


Advert package

Advert package

Goal: Advertisement of important informations at a central place, e.g. job finished, job rc...

Realized via an SQL adaptor in C-GAT.

GAT


Advert package1

Advert package

GATObject

GATAdvertService

Add

Delete

AddMetadata

GetMetadata

GetAdertisable

Find

SetPWD

GetPWD

GAT


Job management classes

GATJob

Unschedule

CheckPoint

CloneJob

Migrate

Stop

GetJobDescription

GetState

GetJobIDGetNativeID

Job managementclasses

GATResourceBroker

ReserveResource

FindResources

SubmitJob

GAT


Job management available job status

Job managementavailable job status

GAT


Job management structures

GATSoftwareResourceDescription

Executable

Arguments

Stdin

Stdout

Stderr

Pre-Stage-Files

Post-Stage-Files

GATJobDescription

Job managementstructures

GATHardwareResourceDescription

Machine type

Memory

CPU-Time

Nodes needed

GAT


Job management example pbs adaptor

Job managementexample (PBS adaptor)

GATHardwareResourceDescription

machine.queue= destination (-q)

jobname = jobname (-N)

yeo = join (-j)

memory.size = mem (-l)

file.size = file (-l)

cpu.walltime = walltime (-l)

cpu.nodes = nodes (-l)

GAT


Job management example pbs adaptor1

Job managementexample (PBS adaptor)

GATSoftwareResourceDescription

Stdin = stdin (exec < input)

Stdout = stdout (-o)

Stderr = stderr (-e)

Executable = executable

Arguments = arglist[]

GAT


Job management example pbs adaptor2

#PBS -q [email protected]

#PBS -l walltime=12:00:00, \

-mem=1G,file=2G,[email protected]

#PBS -N testjob

#PBS -o out.testjob

#PBS -e err.testjob

#PBS -j eo

/bin/prog arg1 arg2 lastarg < input

.

.

Job managementexample (PBS adaptor)

Realization in QSUB script

machine.queue= [email protected]

jobname= TestJob

yeo= eo

memory.size= 1G

file.size= 2G

cpu.walltime= 12:00:00

cpu.nodes= 8

stdin= input

stdout= out.testjob

stderr= err.testjob

executable= /bin/prog

arglist[0]= arg1

arglist[1]= arg2

arglist[2]= lastarg

GAT


Job management lacks

Job managementlacks

Problem:

Defintions in HardwareResourceDescription quite arbitrarily.

Solution:

Orientation at the JSDL standard.

GAT


Job management lacks1

Job managementlacks

Problem:

To less JOB status.

Solution:

Orientation at DRMAA.

GAT


Gat future

GAT future

C-GAT is now in maintenance mode

Java-GAT is still further developed

SAGA (Simple API for Grid Applications) is the upcoming GGF standard. First SAGA engine with GAT wrapper and GTK4 adaptor in autumn 2006

GAT


Usage example proc mpa garching

Usage exampleProC MPA Garching

GAT


Gat

SAGA

To be integrated: GAT, Java-CoG, DRMAA, RealityGrid, JSDL, GridRPC, OSGA-BES, GridCPR, gLite, HDF5

Developed by: GAT, RealityGrid UK Science, OMII Grid UK Science, CCT Louisana, VU Netherlands, NAREGI Japan, Globus/CoG, GGF DRMAA, GGF GridRPC

Important: Please remarks needs on adaptors!

GAT


Gat users

GAT users

C-GAT

Java-GAT

SuperScalar (Univ. of Barcelona, ESP) SURA-Grid (63 partners, US) SCOOP project (LSU + 9 partners, US) UCOMS project (LSU + 4 partners, US) Cactus (LSU, US)Clusterix Project (PSNC, Univ. of Krakow, PL) Amolf (Vl-e, NL) Frank Seinstra (UvA, NL)Triana group (Cardiff, UK)

MPA in Garching

ProC-Workflows on the Grid

LSU in Baton Rouge

Chemistry project

AMOLF NL

Fourier Transform Mass Spectrometry (FTMS) Analyse. FTMS data are delivered to the Grid with JavaGAT (ssh, sftp, gridftp).

Multimedian Project NL

Start of parallel Jobs

ZIB Berlin

INRIA Frankreich

GAT


Links for gat

Links for GAT

GAT overall: http://www.gridlab.org/WorkPackages/wp-1/

GAT CVS: cvs.gridlab.org

cvsroot: pserver:[email protected]:/cvs/gridlab

Password: anon

GAT sources: wp-1/Codes

GAT documents: wp-1/Documents

Only GATEngine: wp-1/Codes/GATEngine

Download tarball:

http://www.gridlab.org/WorkPackages/wp-1/gatreleases.html

http://www.gridlab.org/WorkPackages/wp-1/adaptorreleases.html

German GAT Mailing-Liste:[email protected] (www.listserv.dfn.de)

GAT


  • Login