1 / 25

GROWL Scripts: Lightweight Access to Grid Resources

GROWL Scripts: Lightweight Access to Grid Resources. John Kewley Grid Technology Group e-Science Centre STFC Daresbury Laboratory j.kewley@dl.ac.uk. GROWL.

aquarius
Download Presentation

GROWL Scripts: Lightweight Access to Grid Resources

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GROWL Scripts: Lightweight Access to Grid Resources John Kewley Grid Technology Group e-Science Centre STFC Daresbury Laboratory j.kewley@dl.ac.uk UK e-Science All Hands Meeting

  2. GROWL • Collaborative project between CCLRC (now STFC) Daresbury Laboratory and the Universities of Cambridge and Lancaster, funded by the JISC VRE programme. • Project Objectives: to produce a lightweight client-side Grid connection toolkit. • Project completed January 2007 UK e-Science All Hands Meeting

  3. GROWL • GROWL addresses the three barriers that newcomers find when using the Grid for the first time: • Setting up the client-side middleware • Handling of certificates • Job submission in the presence of firewalls This talk looks at GROWL Scripts, GROWL Web Services takes a different approach UK e-Science All Hands Meeting

  4. Client Middleware: Problems • Typically need to be root to install (according to documentation) • Software must be downloaded from various locations • There are many choices for type of installation (too many options?) UK e-Science All Hands Meeting

  5. Installation • GROWL scripts provide an alternative way of installing Grid middleware on your client Linux machine to that given on the NGS website: • Advantages: • Don't need to be a privileged user • Will download client middleware packages for your system (assuming it is supported) • Minimal setup/configuration • About 10–15 mins (if all goes well !) UK e-Science All Hands Meeting

  6. The Virtual Data Toolkit (VDT) is an easy to install and configure ensemble of grid middleware • http://vdt.cs.wisc.edu • GROWL Scripts installs the pre-WS globus client from VDT, as well as gsi-enabled openssl and the best known (IGTF accredited) CA certificates. UK e-Science All Hands Meeting

  7. Installing Grid Client using GROWL • Download GROWL Scripts $ cd $ wget http://www.growl.org.uk/Growl.tar.gz • Install into home directory $ tar -zxvf Growl.tar.gz • Build VDT client (a software distribution that includes globus) $ cd Growl; make VDT • Before using any GROWL Scripts, bash users should source ~/Growl/setup.sh • while csh users should source ~/Growl/setup.csh UK e-Science All Hands Meeting

  8. GROWL Scripts: Contents • Certificate helper scripts • mk-cert • growl-info, growl-login, growl-logout • VDT client installation of globus and MyProxy • grid-proxy-init, grid-proxy-info • globus-job-submit, globus-job-run • gsissh, gsiscp, openssl • myproxy-init, myproxy-info, myproxy-logon • GROWL wrapper scripts • growl-submit, growl-status, growl-get-output, • growl-sh, growl-cp, growl-mkdir, growl-rm, growl-mv, • growl-pwd, growl-which, growl-get-jobmanager, growl-queue UK e-Science All Hands Meeting

  9. Certificate Manipulation • Hard to remember openssl commands are wrapped for you • Fewer passwords need to be entered • Correct file and directory permissions are applied UK e-Science All Hands Meeting

  10. $ openssl pkcs12 –in mykey.p12 \ -clcerts –nokeys -out usercert.pem <Pass1> $ openssl pkcs12 –in mykey.p12 \ –nocerts -out userkey.pem <Pass1> <Pass2> <Pass2> [confirm] $ chmod 444 usercert.pem $ chmod 400 userkey.pem $ mv userkey.pem ~/.globus $ mv usercert.pem ~/.globus $ chmod 700 ~/.globus $ mk-cert mykey.p12 <Pass1> [<Pass2>] mk-cert UK e-Science All Hands Meeting

  11. growl-login • If you need to upload your certificate to MyProxy and generate a local proxy, growl-login is provided: $ grid-proxy-init ...<grid-password> $ myproxy-init Your identity: /C=UK/O=eScience/OU=CLRC/L=DL/CN=john kewley Enter GRID pass phrase for this identity: <grid-password> ... Enter MyProxy pass phrase:<myproxy-pass> Verifying - Enter MyProxy pass phrase:<myproxy-pass> $ growl-login Password to protect MyProxy credential:<myproxy-pass> Enter GRID pass phrase for this id:<grid-password> UK e-Science All Hands Meeting

  12. growl-info • A wrapper for grid-cert-info, grid-proxy-info and myproxy-info $ growl-info Certificate Information (including validity) -------------------------------------------- subject= /C=UK/O=eScience/OU=CLRC/L=DL/CN=john kewley notBefore=Jun 15 16:10:35 2006 GMT notAfter=Jun 15 16:10:35 2007 GMT Local proxy certificate(s) -------------------------- subject : /C=UK/O=eScience/OU=CLRC/L=DL/CN=john kewley issuer : /C=UK/O=eScience/OU=CLRC/L=DL/CN=john kewley identity : /C=UK/O=eScience/OU=CLRC/L=DL/CN=john kewley type : Proxy draft (pre-RFC) compliant impersonation proxy strength : 512 bits path : /tmp/x509up_u13445 timeleft : 11:57:19 UK e-Science All Hands Meeting

  13. GROWL job submission • Help with transparency - user shouldn't really need to know • Machine's jobmanager • Home directory location • Location in your path of executable • Firewall problems minimised UK e-Science All Hands Meeting

  14. Running a grid job (1) $ growl-submit dl1.nw-grid.ac.ukhostname https://dl1.nw-grid.ac.uk:64010/792/116475/ $ growl-status https://dl1.nw-grid.ac.uk:64010/792/116475/ PENDING $ growl-status https://dl1.nw-grid.ac.uk:64010/792/116475/ DONE $ growl-get-output https://dl1.nw-grid.ac.uk:64010/792/116475/ comp023.nw-grid.ac.uk $ growl-submit -c dl1.nw-grid.ac.uk hostname $ growl-status PENDING $ growl-status DONE $ growl-get-output comp021.nw-grid.ac.uk UK e-Science All Hands Meeting

  15. Globus + Firewalls Grid Resource Client globus-job-submit jobmanager globus-job-get_result Results gsiscp sshd gsissh /GSI-SSHTerm UK e-Science All Hands Meeting

  16. GROWL + Firewalls Client Grid Resource growl-submit jobmanager globus-job-get-output growl-get-output (using gsissh) sshd UK e-Science All Hands Meeting

  17. Advantages • growl-submit: • uses growl-get-jobmanager to obtain default parallel queue, rather than defaulting to jobmanager-fork • uses growl-which to get full path of executable, ensuring it is in your path • growl-get-output: • uses gsissh to do remote retrieval, avoiding client firewall problem UK e-Science All Hands Meeting

  18. Remote filestore manipulation • Equivalents of many of the standard unix command tools are provided for remote filestore manipulation. growl-ls: contents of directory growl-mkdir: (sub)directory creation growl-rm: file removal growl-mv: renaming/moving files growl-which: finds executable in your path growl-pwd: prints your home directory on the grid resource growl-sh :gsissh wrapper (using default ports) growl-cp: remote file copying, including "3rd party" • An additional parameter (the grid resource) is required UK e-Science All Hands Meeting

  19. Remote file copying using growl-cp • growl-cp can be used to stage and retrieve files. The syntax follows that of scp. It can also be used for "3rd party" file transfers • For 3rd party transfers to work, there has to be a route through all firewalls between the 2 remote resources in one direction or the other $ growl-cp my_input_file.txt dl1.nw-grid.ac.uk:. $ growl-cp dl1.nw-grid.ac.uk:my_output.txt . $ growl-cp lv1.nw-grid.ac.uk:my_file.txt dl1.nw-grid.ac.uk:. UK e-Science All Hands Meeting

  20. growl-cp (1) Client Grid Resources B A UK e-Science All Hands Meeting

  21. growl-cp (2) Client Grid Resources B A UK e-Science All Hands Meeting

  22. growl-cp (3) Client Grid Resources B A UK e-Science All Hands Meeting

  23. Usage patterns • Easy way to build VDT • As above + certificate scripts • As above + use of job submission features UK e-Science All Hands Meeting

  24. Current/future work • Scripting help for a simplistic DIY meta-scheduler • Use of Java CoG-kit + GSI-SSHTERM on Windows UK e-Science All Hands Meeting

  25. Summary • Useful as an easy way to build VDT • Simpler job submission: • less need be known about Grid resources • less firewall pain for retrieving data • Firewall-aware 3rd party file copying • http://www.growl.org.uk/ UK e-Science All Hands Meeting

More Related