1 / 25

Current status of

Current status of. IGTMD meeting, Lyon Sylvain Reynaud. cluster. Use cases. User may access several grid infrastructures when he needs… some data available on a given grid a very large number of computing resources resources with specific constraints super-computer confidentiality

lorieh
Download Presentation

Current status of

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Current status of IGTMD meeting, Lyon Sylvain Reynaud

  2. cluster Use cases • User may access several grid infrastructures when he needs… • some data available on a given grid • a very large number of computing resources • resources with specific constraints • super-computer • confidentiality • small overhead (e.g. for consolidation) • interactivity • … JSAGA

  3. hide middleware heterogeneity (e.g. gLite, Globus, Unicore) JDL RSL What is JSAGA ? job desc. JSAGA gLite plug-ins Globus plug-ins JSAGA

  4. hide middleware heterogeneity (e.g. gLite, Globus, Unicore) EGEE OPlast delegate selection & files staging WMS SRM input data hide grid infrastructures heterogeneity (e.g. EGEE, OSG, DEISA) GridFTP LCG-CE LCG-CE WS-GRAM WS-GRAM firewall job job What is JSAGA ? job desc. JSAGA gLite plug-ins Globus plug-ins JDL RSL staging graph JSAGA

  5. For end-users: ready-to-use software adapted to their needs For application developers: hide infrastructure heterogeneity For application developers: hide middleware heterogeneity For plug-in developers: as many interfaces as ways to implement the functionalities SAGA Plug-ins interfaces Global architecture • Class URLFactory introduced in release "1.0rc2". • This will enable to implement : • support for windows paths • support for relative paths • protocol-dependant URL encoding • … JSAGA

  6. close to application developer needs object oriented uniform interface to all the supported technologies but << certainly not simple to implement >> (T. Kielmann) code is twice bigger than the code for all the plug-ins close to existing middleware API service oriented as many interfaces as ways to provide the functionalities optional interfaces for technology-specific… optimizations additional features SAGA Plug-ins interfaces Design of plug-in interfaces Need standard plug-in interfaces to enable to factorize efforts of wrapping middleware API. JSAGA

  7. done construction planned Plug-in interfaces for execution Streaming Plug-in interfaces: direct/buffered/redirected streams used before/during/after execution Monitoring Plug-in interfaces: querying / listening individual job / list of jobs / filtered jobs set stream for interactive set stream for non- interactive get stream for interactive query status for individual job listen status for individual job query status for filtered jobs getInput getOutput getError getState waitFor SAGA user interface: getInput / getOutput SAGA user interface: getState / waitFor Job control Job monitoring gatekeeper gLite-WMS wsgram unicore6 ssh fork cream gridvm PBS proxy gatekeeper gLite-LB wsgram unicore6 ssh fork cream … JSAGA

  8. done construction planned Physical files Logical files Provided plug-ins Security X509 SSH InMemCred Globus G. Legacy G. RFC820 MyProxy VOMS Login / pwd JKS Data catalog rns lfn srb / irods http https sftp rbyteio file zip gsiftp tar ftp mail cache srm Exec. (control) Exec. (monitor) Job control gatekeeper gLite-WMS wsgram unicore6 ssh local cream gridvm PBS remote gatekeeper gLite-LB wsgram unicore6 ssh local cream … Expression Language basic default JEP BeanShell JSDL+ext. SAGA JDL RSL-2 RSL-4 JSAGA

  9. done construction planned Job Collection parametric Job termination rules JSDL retry rules data staging extensions language specific Plug-ins out of SAGA scope piece of a sample JSDL job description in JDL description in JSDL description in RSL <POSIXApplication> <Executable>myexec</Executable> <Argument>-from @{date('dd/MM/yyyy', now+24*INDEX)}</Argument> </POSIXApplication> … <POSIXApplication> <Executable>myexec</Executable> <Argument>-from 30/05/2008</Argument> </POSIXApplication> … <POSIXApplication> <Executable>myexec</Executable> <Argument>-from 31/05/2008</Argument> </POSIXApplication> … <POSIXApplication> <Executable>myexec</Executable> <Argument>-from 01/06/2008</Argument> </POSIXApplication> … Expression Language basic default JEP BeanShell JSDL+ext. SAGA JDL RSL-2 RSL-4 JSAGA

  10. Using SAGA API security context data management execution management Gatekeeper Unicore 6 SSH fork Demo (part 1) hide middleware heterogeneity (e.g. gLite, Globus, Unicore) JSAGA

  11. For end-users: ready-to-use software adapted to their needs For application developers: hide infrastructure heterogeneity For application developers: hide middleware heterogeneity For plug-in developers: as many interfaces as ways to implement the functionalities Job Collection SAGA Plug-ins interfaces Global architecture JSAGA

  12. Hide middleware heterogeneity Uniform interface (SAGA) Hide infrastructure heterogeneity Selection of the right security context Selection of the right client API configuration Transport of the job input/output data to/from worker nodes Describe the capabilities of the infrastructures you want to use VOMS VOMS Globus Describe grid infrastructures requirements CC-IN2P3 WMS gsiftp:// EGEE wsgram OpenPlast Grid World localhost JSAGA

  13. Hide middleware heterogeneity Job management technologies e.g. CREAM, WMS, SSH, GK Hide infrastructure heterogeneity Computing Elements (e.g. GK) Different grid or site policies e.g. network filtering, shared FS Different environment variables e.g. $VO_?_SW_DIR, /usr/local Different configuration attributes e.g. monitor service URL, shell path on cygwin, default SE URL Commands available on worker e.g. globus-url-copy, srmcp, Scp, wget, tar VOMS VOMS Globus Describe grid infrastructures examples gatekeeper srb:// srm:// CC-IN2P3 lfn:// WMS gsiftp:// EGEE gatekeeper wsgram OpenPlast Grid http:// tar:// gatekeeper World localhost JSAGA

  14. VOMS gatekeeper srb:// srm:// CC-IN2P3 lfn:// WMS gsiftp:// EGEE VOMS gatekeeper wsgram OpenPlast Grid http:// Globus tar:// gatekeeper World localhost plug-ins Transfer path depends on… • Using a single grid • all files can be transferred to the worker through a single node • Using several grids • need to dynamically build a transfer graph, according to… url:// job desc. JSAGA JSAGA

  15. VOMS gatekeeper srb:// srm:// CC-IN2P3 lfn:// WMS gsiftp:// EGEE VOMS gatekeeper wsgram OpenPlast Grid http:// tar:// Globus gatekeeper localhost World plug-ins Transfer path depends on… • grid or site • network filtering policy • commands available on workers • services available from workers (close Storage Element, shared FS) • supported context instances • data to stage • shared by several jobs • installed on some worker nodes • file size • required data protection level • execution service • protocols supported for staging • transfer protocol • access mode (RO, WO, RW) • third-party transfer • supported data protection level url:// job desc. JSAGA JSAGA

  16. C C' common result R1 std-error E1 R1 C' job C SMTP SRB GSIFTP GSIFTP HTTP job OPlast EGEE CA OPlast OPlast OPlast OPlast OPlast E1 job VOMS gatekeeper srb:// srm:// CC-IN2P3 lfn:// WMS gsiftp:// EGEE VOMS GSIFTP gatekeeper wsgram OpenPlast OpenPlast Grid http:// tar:// Globus gatekeeper localhost World OpenPlast Transfer path depends on… • grid or site • network filtering policy • commands available on workers • services available from workers (close Storage Element, shared FS) • supported context instances • data to stage • shared by several jobs • installed on some worker nodes • file size • required data protection level • execution service • protocols supported for staging • transfer protocol • access mode (RO, WO, RW) • third-party transfer • supported data protection level JSAGA

  17. data to stage shared by several jobs installed on some worker nodes file size required data protection level grid or site network filtering policy commands available on workers services available from workers (close Storage Element, shared FS) supported context instances E E src executable input data D1 SMTP SRB GSIFTP GSIFTP HTTP R1 C' D1 job C GSIFTP job OPlast EGEE CA OPlast OPlast OPlast OPlast OPlast TAR TAR E1 job C C' C'' common Transfer path depends on… result R1 • execution service • protocols supported for staging • transfer protocol • access mode (RO, WO, RW) • third-party transfer • supported data protection level std-error E1 E src C" E iGet JSAGA

  18. Using SAGA API security context data management execution management Gatekeeper Unicore 6 SSH fork Using Job Collection API execution mgt. EGEE OpenPlast my laptop VOMS gatekeeper srb:// srm:// CC-IN2P3 lfn:// WMS gsiftp:// EGEE VOMS gatekeeper wsgram OpenPlast Grid http:// tar:// Globus gatekeeper localhost World Demo (part 2) hide grid infrastructures heterogeneity (e.g. EGEE, OSG, DEISA) hide middleware heterogeneity (e.g. gLite, Globus, Unicore) JSAGA

  19. Fully automated build process download build tools, modules generate code, build, run tests Generated project web site : http://grid.in2p3.fr/jsaga/ documentation user guides (from annotations) specification (from code) design (from code) reports static code analysis test results and coverage tasks list Provide wizard to generate skeleton of new plug-in projects Provide reusable test suite for validating plug-ins Software quality # SAGA protocols test-suite configuration gsiftp.base=gsiftp://ccrugceli01.in2p3.fr/tmp/ gsiftp.base2=gsiftp://agena.c-s.fr/grid/tmp/ gsiftp.context=OpenPlast_proxy lfn.base=lfn:///grid/dteam/ lfn.context=EGEE_proxy http.base=http://grid.in2p3.fr/html/Public/ file.base=file:///c:/tmp/ file.base2=file:///c:/ JSAGA

  20. Minimize dependencies resolve conflicts between versions of libraries with maven regenerate stubs from WSDL with Axis 1.4 remove external dependencies no gLite-UI for EGEE, no OS-specific dependency Provide an installer install only what you need… Software quality JSAGA

  21. Software metrics • Specification • SAGA, version 1.0 (January 15, 2008) : 323 pages • Official Java Binding, version "snapshot-1" • Implementation • 46 third-party libraries (only 5 are for the core engine) • 644 java classes (+ 508 generated java classes) • some code in other languages • C • XSL • XML Schema • LL grammar • Bash shell • MS-DOS batch JSAGA

  22. VOMS gatekeeper srb:// srm:// CC-IN2P3 JSAGA lfn:// WMS EGEE gsiftp:// VOMS gatekeeper wsgram OpenPlast Grid http:// tar:// Globus gatekeeper localhost World plug-ins Main assets of JSAGA • does not sacrifice scalability and efficiency for abstraction • thanks to design (in particular the definition of plug-in interfaces) • thanks to caching mechanisms • does not only hide heterogeneity of middleware, but also hides heterogeneity of grid infrastructures • thanks to • use existing grid infrastructures as they are url:// job desc. JSAGA

  23. For end-users: ready-to-use software adapted to their needs For application developers: hide infrastructure heterogeneity For application developers: hide middleware heterogeneity For plug-in developers: as many interfaces as ways to implement the functionalities Applications Job Collection SAGA Plug-ins interfaces Global architecture JSAGA

  24. Applications Related projects • JSAGA is used by… • Elis@ • a web portal for submitting jobs to industrial and research grid infrastructures • JJS (Java Job Submission) • a tool for submitting job collections to EGEE • optimized for short-life jobs (resource selection based on QoS observed while submitting jobs) • JUX (Java Universal eXplorer) • a multi-protocol file browser • is currently testing it / JSAGA

  25. JJS plug-ins for CREAM, LCG-CE (extends GK plug-in), finish plug-ins for SRM, LFC plug-ins for JDL, tar etc. Perspectives • Improvement of the core engine (IGTMD project) • move to SAGA Java Binding release candidate 2 (and next...) • bug-fixes, new features, etc. • Development of plug-ins (contribution to other projects) • RA-Grid • plug-ins for OAR, CiGri • NAREGI • plug-ins for Super Scheduler, (GridVM ?) • for BQS users who want to progressively use grid • plug-ins for BQS JSAGA

More Related