Condor BirdBath SOAP Interface to Condor

1. Condor BirdBathSOAP Interface to Condor Charaka Goonatilake Department of Computer Science University College London c.goonatilake@cs.ucl.ac.uk

2. Background

3. Motivations • Adding Web Service interfaces to Condor daemons • Facilitate the development of third-party applications capable of interacting with Condor (remotely). • E.g. build higher-level application specific scheduler that submits jobs to multiple Condor pools based on application semantics • These can be built using a wide range of languages/SOAP packages • BirdBath has been tested on: • Java (Apache Axis, XSUL) • Python (ZSI) • C# (.Net) • C/C++ (gSOAP) • Condor accessible from platforms where its command-line tools are not supported/installed

4. Schedd Schedd Schedd The (Hypothetical) Big Picture Site D Condor-G Globus/gridSAM Client SOAP SOAP flocking Site A Site B Site C

5. Shadow Condor SOAP Interfaces • Now available in Condor 6.7.5 • Obtain resource ClassAds • Obtain all daemon ClassAds • Begin/commit/abort transaction • Submit Job • Obtain Job ClassAds • Get/Send file • Removed/hold/release job Execution machine Execution machine Submission machine(s) Central manager Schedd Collector Negotiator Execution machine Execution machine

6. Clients for BirdBath • WSDLs available that describe each daemon interface • SOAP libraries (stub code) can be generated automatically from WSDLs Custom Client holdJob() Submission machine(s) SOAP library Schedd

7. Example: Query Collector • Obtain information (ClassAds) about all resources that have over 512MB RAM collectorLocator = new CondorCollectorLocator(); collector = locator.getcondorCollector(new URL(“http://kotturoti.cs.ucl.ac.uk:9618”); classAds = collector.queryStartdAds(“Memory>512”);

8. Example: Job Submission • Create New Transaction • Create New JobId • Send Files ({input files, binaries}) • Create Job Description • Submit Job • Commit Transaction • Transaction based process • Uses two phase commit to promote reliability and robustness • Files transferred in chunks – Base64 encoded SOAP attachments • DAGMan jobs supported

9. Client Client <?xml version="1.0" encoding="UTF-8"?> <JobDefinition xmlns="http://…"> <JobDescription> <Application> <Executable>/bin/echo</Executable> <Argument>hello world</Argument> </Application> </JobDescription> </JobDefinition> JSDL JSDL SOAP SOAP GridSAM GridSAM Condor Condor SGE SGE Fork Fork … … OMII Collaboration: gridSAM plugin • Job Submission & Monitoring Web Service that submits to range of resource managers (e.g. Condor, SGE, Globus) • Uses JSDL (Job Submission Description Language) for standardised job description

10. Our plugin accesses Condor via the SOAP interfaces Condor-WS • Building JSDL to ClassAd conversion libraries ClassAd SOAP Submission machine(s) Schedd gridSAM plugin • Existing Condor plugin uses shell-based submission via command-line tools and monitoring of log files Client JSDL SOAP Middle-tier GridSAM Condor SGE Fork …

11. Future Work • Bundle plug-in into gridSAM • Also provide separate JSDL to ClassAd conversion library • Extensions to other daemons (what functionality to expose?) • For example, allow checkpoint data to be retrieved through the shadow in order to migrate jobs across pools • Roll out daemons on e-minerals mini-grid