Condor birdbath soap interface to condor
Download
1 / 11

Condor BirdBath SOAP Interface to Condor - PowerPoint PPT Presentation


  • 124 Views
  • Uploaded on

Condor BirdBath SOAP Interface to Condor. Charaka Goonatilake Department of Computer Science University College London [email protected] Background. Motivations. Adding Web Service interfaces to Condor daemons

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Condor BirdBath SOAP Interface to Condor' - JasminFlorian


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Condor birdbath soap interface to condor

Condor BirdBathSOAP Interface to Condor

Charaka Goonatilake

Department of Computer Science

University College London

[email protected]



Motivations
Motivations

  • Adding Web Service interfaces to Condor daemons

  • Facilitate the development of third-party applications capable of interacting with Condor (remotely).

    • E.g. build higher-level application specific scheduler that submits jobs to multiple Condor pools based on application semantics

    • These can be built using a wide range of languages/SOAP packages

    • BirdBath has been tested on:

      • Java (Apache Axis, XSUL)

      • Python (ZSI)

      • C# (.Net)

      • C/C++ (gSOAP)

  • Condor accessible from platforms where its command-line tools are not supported/installed


The hypothetical big picture

Schedd

Schedd

Schedd

The (Hypothetical) Big Picture

Site D

Condor-G

Globus/gridSAM

Client

SOAP

SOAP

flocking

Site A

Site B

Site C


Condor soap interfaces

Shadow

Condor SOAP Interfaces

  • Now available in Condor 6.7.5

  • Obtain resource ClassAds

  • Obtain all daemon ClassAds

  • Begin/commit/abort transaction

  • Submit Job

  • Obtain Job ClassAds

  • Get/Send file

  • Removed/hold/release job

Execution

machine

Execution

machine

Submission

machine(s)

Central manager

Schedd

Collector

Negotiator

Execution

machine

Execution

machine


Clients for birdbath
Clients for BirdBath

  • WSDLs available that describe each daemon interface

  • SOAP libraries (stub code) can be generated automatically from WSDLs

Custom Client

holdJob()

Submission

machine(s)

SOAP library

Schedd


Example query collector
Example: Query Collector

  • Obtain information (ClassAds) about all resources that have over 512MB RAM

    collectorLocator = new CondorCollectorLocator();

    collector = locator.getcondorCollector(new URL(“http://kotturoti.cs.ucl.ac.uk:9618”);

    classAds = collector.queryStartdAds(“Memory>512”);


Example job submission
Example: Job Submission

  • Create New Transaction

  • Create New JobId

  • Send Files ({input files, binaries})

  • Create Job Description

  • Submit Job

  • Commit Transaction

  • Transaction based process

  • Uses two phase commit to promote reliability and robustness

  • Files transferred in chunks – Base64 encoded SOAP attachments

  • DAGMan jobs supported


  • Omii collaboration gridsam plugin

    Client

    Client

    <?xml version="1.0" encoding="UTF-8"?>

    <JobDefinition xmlns="http://…">

    <JobDescription>

    <Application>

    <Executable>/bin/echo</Executable>

    <Argument>hello world</Argument>

    </Application>

    </JobDescription>

    </JobDefinition>

    JSDL

    JSDL

    SOAP

    SOAP

    GridSAM

    GridSAM

    Condor

    Condor

    SGE

    SGE

    Fork

    Fork

    OMII Collaboration: gridSAM plugin

    • Job Submission & Monitoring Web Service that submits to range of resource managers (e.g. Condor, SGE, Globus)

    • Uses JSDL (Job Submission Description Language) for standardised job description


    Gridsam plugin

    Condor-WS

    • Building JSDL to ClassAd conversion libraries

    ClassAd

    SOAP

    Submission

    machine(s)

    Schedd

    gridSAM plugin

    • Existing Condor plugin uses shell-based submission via command-line tools and monitoring of log files

    Client

    JSDL

    SOAP

    Middle-tier

    GridSAM

    Condor

    SGE

    Fork


    Future work
    Future Work

    • Bundle plug-in into gridSAM

      • Also provide separate JSDL to ClassAd conversion library

    • Extensions to other daemons (what functionality to expose?)

      • For example, allow checkpoint data to be retrieved through the shadow in order to migrate jobs across pools

    • Roll out daemons on e-minerals mini-grid


    ad