1 / 16

A module to customize CREAM jobs according to site policies

JWGEN :. A module to customize CREAM jobs according to site policies. Tsukuba, KEK, 21 December 2010 Sylvain Reynaud. Adapt the job… to user needs set user job environment middleware version, operating system, environment variable set LRMS queue set hardware resources CPU, memory, disk

vern
Download Presentation

A module to customize CREAM jobs according to site policies

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. JWGEN : A module to customize CREAM jobs according to site policies Tsukuba, KEK, 21 December 2010 Sylvain Reynaud

  2. Adapt the job… to user needs set user job environment middleware version, operating system, environment variable set LRMS queue set hardware resources CPU, memory, disk set logical resources hpss, xrootd, dCache, oracle set permissions (sw manager) to site policy accept/deny job submission enforcement of security policy options and information for : quota, priorities, accounting According to the information associated with this job Computing Element (CE) CE Queue User's virtual organization User's role JDL requirements (if specified) CPU memory disk The problem

  3. JWGEN : a module to adapt grid job to… user needs site policy …according to information associated with this job CE, queue, VO, requirements… A solution : JWGEN CREAM-CE BLAH APEL GIP job control job monitor info. provider accoun -ting JWGEN BQS

  4. A solution : JWGEN CREAM-CE CREAM-CE Padova BLAH APEL BLAH GIP APEL GIP CNAF job control job monitor info. provider accoun -ting job control job monitor info. provider accoun -ting JWGEN JWGEN BQS Grid Engine

  5. JSDL options + wrapper A solution : JWGEN JDL CREAM-CE CREAM-CE Padova BLAH APEL BLAH GIP APEL GIP CNAF job control job monitor info. provider accoun -ting job control job monitor info. provider accoun -ting JWGEN JWGEN BQS Grid Engine

  6. Input : JSDL

  7. Output : SGE options + job wrapper $ ./libexec/jwgen-core/generate_wrapper.sh /tmp/753627923.jsdl #!/bin/bash #$ -q admin #$ -v SITE_NAME=Test #$ -A "GRID=EGI SITE=Test TIER=tier3 VO=dteam ROLEVOMS=&2Fdteam&2FRole=NULL&2FCapability=NULL DN=&2FO=GRID-FR&2FC=FR&2FO=CNRS&2FOU=CC-LYON&2FCN=Sylvain&20Reynaud" #$ -l os=sl5 #$ -N ccsyre_crm01_753627923 #$ -l s_cpu=00:01:40 #$ -l s_vmem=20M #$ -l s_fsize=5G #$ -l dcache [ -r /afs/in2p3.fr/grid/profiles/glite/3.1.25-0/SL4_64/WN32/lcg-prologue.sh ] && . /afs/in2p3.fr/grid/profiles/glite/3.1.25-0/… … etc …

  8. Configuration file CREAM-CE CREAM-CE Padova BLAH APEL BLAH GIP APEL GIP CNAF job control job monitor info. provider accoun -ting job control job monitor info. provider accoun -ting JWGEN JWGEN BQS Grid Engine config file

  9. Configuration file • Need to factorize information • thanks to CE profiles config file

  10. Configuration file • Need to factorize information • thanks to CE profiles CC-IN2P3 site CC-IN2P3 Tier 1 EGI LCG profile A profile B LCG-CE 01 LCG-CE 02 LCG-CE 04 CREAM 02 LCG-CE 03 LCG-CE 07 LCG-CE 08 CC-IN2P3 Tier 2 Test ROC-FR LCG-CE 05 CREAM 05 LCG-CE 11 CREAM 03 CREAM VM-07 LCG-CE 06

  11. Configuration file • Need to factorize information • thanks to CE profiles • thanks to operations on sets • Sets are used for • queues, groups, users, logical resources • Sets can be • created : ignore ancestor sets • subtracted : exclude some items • refined : without adding new items • extended : with adding new items

  12. Query configuration file $ ./sbin/jwgen-ce-query.sh -h ./sbin/jwgen-ce-query.sh: option requires an argument -- h Usage: ./sbin/jwgen-ce-query.sh [-h <hostname>] [-q <queue>] [-g <group>] [-u <user>] [-x] Where: -h : hostname of the CE (default value: 'ccsyre.in2p3.fr') -q : CE queue (default value: '_' = any value) -g : unix group (default value: '_' = any value) -u : unix user account (default value: '_default' = non-special users) -x : print output as XML

  13. Query configuration file $ ./sbin/jwgen-ce-query.sh -h cclcgceli05.in2p3.fr -u atlas099 CEInstance 'cclcgceli05.in2p3.fr' CEQueue 'long' Group 'atlas' User 'atlas099' LogicalResourceSet = {dcache,xrootd} OperatingSystem = sl5 ProfilesDirectory=/afs/in2p3.fr/grid/profiles/glite/3.2.2-0/SL5_64/WN64 Project = P_atlas_T2ana ResourceCPUMax = max ResourceFileSizeMax = max ResourceMemoryMax = max ResourceQueue = long SubmissionPolicy = ACCEPT

  14. Core package independent of • grid middleware • based on JSDL • can be used with CREAM job control plug-in (BLAH) or Globus job manager for example • LRMS • developed, tested and used with BQS • will now be used with Grid Engine • site policy • configuration keywords and templates are provided separately

  15. Conclusion • JWGEN has proven to be useful with BQS, now with GE • It can be reused with other grid middlewares, LRMS and site policies • But… • configuration is of course more complex that the BLAH flat file • customization is not so easy to do (even if customizable files are well identified) • We plan to propose JWGEN as an optional module for grid sites that need this kind of advanced features

  16. Questions ?

More Related