1 / 40

Hands-On Session: Using Grid

Hands-On Session: Using Grid. Regional SEE-GRID-SCI Meteo Training Faculty of Electrical Engineering Banja Luka July 2-3, 2009. Vladimir Slavnic Institute of Physics, Belgrade Serbia slavnic@scl.rs.

korene
Download Presentation

Hands-On Session: Using Grid

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Hands-On Session:Using Grid Regional SEE-GRID-SCI Meteo Training Faculty of Electrical Engineering Banja Luka July 2-3, 2009 Vladimir Slavnic Institute of Physics, Belgrade Serbia slavnic@scl.rs The SEE-GRID-SCI initiative is co-funded by the European Commission under the FP7 Research Infrastructures contract no. 211338

  2. Setting up the user account

  3. Setting up the user accountUser Interface (UI)‏ • Access point to Grid • User must have a local account on machine • Basic grid operations: • list resources suitable for job • submit jobs • cancel jobs • retrieve job output • show job status • file operations • …

  4. Setting up the user account Certificates Grid passport X.509 Subject Name (SN)‏ /C=RS/O=AEGIS/OU=Institute of Physics Belgrade/CN=Vladimir Slavnic Certification Authority (CA)‏ Obtaining a certificate $ grid-cert-request userkey.pem usercert_request.pem usercert.pem Virtual Organisation (VO)‏

  5. Setting up the user accountSetting up .globus Right permissions: – 444 usercert.pem – 400 userkey.pem Checking a certificate $ grid-cert-info $ grid-cert-info -subject (-enddate) (-issuer)‏ $ openssl verify -CApath \ /etc/grid-security/certificates/ ~/.globus/usercert.pem $ grid-proxy-init -verify

  6. Setting up the user account Proxies Standard proxies $ grid-proxy-init $ grid-proxy-info $ grid-proxy-destroy VOMS proxies – Group – Role $ voms-proxy-init -voms <vo> $ voms-proxy-init –voms <alias>:<group name> [Role=<role name>] $ voms-proxy-info (-all)‏

  7. Setting up the user accountProxy renewal MyProxy server $ echo $MYPROXY_SERVER $myproxy-init -s <myproxy_server> -d -n $myproxy-info -s <myproxy_server> -d $myproxy-destroy -s <myproxy_server> -d

  8. Information service

  9. INFORMATION SERVICElcg-infosites & lcg-info $lcg-infosites --vo <vo> <option> -v <verbosity> -f <site> --is <bdii> $ lcg-infosites --vo seegrid ce $ lcg-infosites --vo seegrid se $ lcg-infosites --vo seegrid closeSE $ lcg-info –list-attrs $ lcg-info --list-se --query 'SE=se.csk.kg.ac.yu' --attrs CloseCE

  10. INFORMATION SERVICEldap ldap search examples # RESOURCE ldapsearch -x -H ldap://ce64.phy.bg.ac.yu:2170/ -b “Mdsvo-name=resource,o=Grid” # sBDII ldapsearch -x -H ldap://ce64.phy.bg.ac.yu:2170 -b mds-voname=AEGIS01-PHY-SCL,o=grid # top-level BDII ldapsearch -x -H ldap://bdii.phy.bg.ac.yu:2170/ -b 'Mds-voname=local,o=Grid'

  11. WatG Browser Web based Grid Information System browser Visualization of BDII information Highly responsive tool because it supports partial refreshes and desynchronization of a web page. Developed with Google Web Toolkit (GWT)‏ open source Java software development framework. '

  12. WatG Browser http://watgbrowser.scl.rs:8080/

  13. Data management

  14. DATA MANAGEMENT • Storage element • Classic SE • LCG Disk pool manager • CASTOR • Storage Resource Manager (SRM)‏ • LFC Concept • lfc and lcg commands

  15. DATA MANAGEMENTGrid file LFN – Logical File Name GUID - Grid Unique Identifier SURL – Storage URL

  16. DATA MANAGEMENTLFC scheme

  17. DATA MANAGEMENTLFC commands • echo $LFC_HOST • lfc-ls - List file/directory entries in a directory • lfc-mkdir - Create directory • lfc-ln - Make a symbolic link to a file/directory • lfc-ls - List file/directory entries in a directory • lfc-chmod- Change access mode of a LFC file/directory • lfc-chown - Change owner and group of a LFC file/directory • lfc-getacl - Get file/directory access control lists • lfc-setacl - Set file/directory access control lists • …

  18. DATA MANAGEMENTLFC commands examples (1)‏ Listing the entries of a LFC directory $ lfc-ls /grid/seegrid/vlada new_file result Creating directories in the LFC $ lfc-mkdir /grid/seegrid/vlada/new_dir $ lfc-ls -l /grid/seegrid/vlada drwxrwxr-x 0 165 101 0 Feb 19 00:30 new_dir -rw-rw-r-- 1 165 101 27 Feb 18 01:08 new_file -rw-rw-r-- 1 165 101 83 Feb 15 00:30 result

  19. DATA MANAGEMENTLFC commands examples (2)‏ Creation of symbolic links $ lfc-ln -s /grid/seegrid/vlada/result \ /grid/seegrid/vlada/result_link $ lfc-ls -l /grid/seegrid/vlada/result_link lrwxrwxrwx 1 165 101 0 Feb 19 00:42 /grid/seegrid/vlada/rezult_link -> /grid/seegrid/vlada/rezult Adding metadata information to LFC entries $ lfc-setcomment /grid/seegrid/vlada/result "best result" $ lfc-ls --comment /grid/seegrid/vlada/result /grid/seegrid/vlada/result best result

  20. DATA MANAGEMENTlcg_utils-Replica Management echo $LCG_GFAL_INFOSYS lcg-cp Copies a Grid file to a local destination (download)‏ lcg-cr Copies a file to a SE and registers the file in the catalogue (upload)‏ lcg-del Deletes one file (either one replica or all replicas)‏ lcg-rep Copies a file from one SE to another SE and registers it in the catalogue (replicate)‏ lcg-gt Gets the TURL for a given SURL and transfer protocol

  21. DATA MANAGEMENTlcg_utils-File Catalogue Interaction lcg-aa Adds an alias in the catalogue for a given GUID lcg-ra Removes an alias in the catalogue for a given GUID lcg-rf Registers in the catalogue a file residing on an SE lcg-uf Unregisters in the the catalogue a file residing on an SE lcg-la Lists the aliases for a given LFN, GUID or SURL lcg-lr Lists the replicas for a given LFN, GUID or SURL lcg-ls Lists file information for given SURLs or LFNs

  22. DATA MANAGEMENTlcg_utils examples(1)‏ Upload a file $ lcg-cr file:/home/vlada/banja_luka/mpi.jdl guid:830e7a82-f9fb-4c04-aafd-e24a5f25acdb specify lfn: $ lcg-cr -l lfn:/grid/seegrid/vlada/mpi \ file:/home/vlada/banja_luka/mpi.jdl guid:25fdcd16-bc76-4458-9c1a-8c597fb33194 Replicate a file (one replica per SE) $ lcg-rep -d grid15.rcub.bg.ac.yu \ lfn:/grid/seegrid/vlada/mpi

  23. DATA MANAGEMENTlcg_utils examples(2)‏ List replicas $ lcg-lr lfn:/grid/seegrid/vlada/mpi srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/ae gis/generated/2009-02-18/file8f99889c-c00a-4b7b- 89d3-d9abf66ba103 srm://se.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/generated/2009-02-18/file576fbae0-ba6a-4b21-a205-3d115b2cf7fb List guid $ lcg-lg lfn:/grid/seegrid/vlada/mpi guid:25fdcd16-bc76-4458-9c1a-8c597fb33194

  24. DATA MANAGEMENTlcg_utils examples(3)‏ Listing files and directories $ lcg-ls -l lfn:/grid/seegrid/vlada/mpi -rw-rw-r-- 1 165 101 349 lfn:/grid/seegrid/vlada/mpi Copying files out of the Grid (download)‏ $ lcg-cp lfn:/grid/seegrid/vlada/mpi \ file:/home/vlada/banja_luka/down/mpi Deleting replicas lcg-del srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/aegis/generated/2009-02-19/file36d25d0d-ee61-477eb667-40b3791786a4 $ lcg-del -a lfn:/grid/aegis/vlada/mpi

  25. Data Management Web Portal • Developed by Belgrade University Computer Centre • Usage • Data Management • Logical Files • Development Environments and Portals • Description • Enables easy interaction with grid file catalogue using simple Web-based user interface. • Based on SEE-GRID File Management Java API. • Can be integrated with other Web applications.

  26. Data Management Web Portal • Available features • Directory browsing • Directory management operations: creating new subdirectories, renaming/moving and deleting directories, replicating directory content, reading directory attributes and permissions. • Copying and registering of new files to the catalog as well as file deletion. • Reading file attributes and permissions. • Replica & alias management operations. • Upload and download of files with possible integration of application-defined upload handlers

  27. Data Management Web Portal • http://grid02.rcub.bg.ac.yu:8080/NewDataManagement/login-REFS.jsp • http://grid02.rcub.bg.ac.yu:8080/NewDataManageme nt/login-WRF-ARW.jsp

  28. Job operations

  29. JDL JDL (Job Description Language) file attribute = expression Executable = "test.sh"; Arguments = "fileA fileB"; StdOutput = "std.out"; StdError = "std.err"; InputSandbox = {"test.sh", "fileA", "fileB"}; OutputSandbox = {"std.out", "std.err"}; Environment = {“AEGIS_PATH=$HOME/AEGIS"}; Requirements = RegExp("ce64.phy.bg.ac.rs*",other.GlueCEUniqueID);

  30. SUBMITING JOBS Generating proxy $ voms-proxy-init -voms <VO_NAME> Myproxy $ myproxy-init -d –n -s <myproxy_server> Delegate proxy to WMS $ glite-wms-job-delegate-proxy -d <delegID> Job matching $ glite-wms-job-list-match -a test.jdl Job submission $ glite-wms-job-submit -d <delegID> -i <job_id_file> test.jdl

  31. MONITORING JOBS View status of job $ glite-wms-job-status <job_ID> $ glite-wms-job-status -i <job_id_file> View logging details $ glite-wms-job-logging-info -v <verbosity_level> <job_ID> $ glite-wms-job-logging-info -v <verbosity_level> -i <job_id_file> Verbosity level can be from 0 to 3.

  32. CANCELING JOBS AND RETRIEVING OUTPUT Cancel a job $ glite-wms-job-cancel <job_ID> $ glite-wms-job-cancel -i <job_id_file> Retrieve files from Output Sandbox $ glite-wms-job-output <job_ID> $ glite-wms-job-output -i <job_id_file> $ glite-wms-job-output --dir <directory_path> <job_ID>

  33. ADVANCED JOB TYPES Job Collection - submission of a set of jobs whose description files are placed in one directory Parametric Jobs - submission of a set of jobs having the same descriptions apart from the values of the parametric attributes. Use when your jobs differs only in argument values or input/output files. DAG jobs (Direct Acyclic Graphs) - submission of a set of jobs where the input, output, or execution of one or more jobs depends on one or more other jobs. Interactive Jobs MPI Jobs

  34. JOB COLLECTION Type = "Collection"; InputSandbox = (“fileA”); Requirements = RegExp("ce64.phy.bg.ac.yu*",other.GlueCEUniqueID); nodes = { [ JobType = "normal"; Executable = "/bin/hostname"; Arguments = "-i"; StdOutput = "out"; InputSandbox = (root.InputSandbox, “fileB”); StdError = "err"; OutputSandbox ={"out","err"}; ], [ JobType = "Normal"; Executable = "/bin/date"; StdOutput = "date"; StdError = "err"; OutputSandbox= {"date"}; ] };

  35. JOB COLLECTIONsubmission • Collection defined in one file • $ glite-wms-job-submit -a -o id collection.jdl • Folder contains jdl files • $ glite-wms-job-submit -a --collection jdl • Collection status view • Collection output • $ glite-wms-job-output -i id --dir kolekcija • $ ls kolekcija • ids_nodes.map Node_0 Node_1

  36. PARAMETRIC JOBS Type = "Job"; JobType = "Parametric"; Executable = "/bin/echo"; Arguments = "_PARAM_"; Parameters = {red, green, blue}; StdOutput = "colour_PARAM_.txt"; StdError = "error_PARAM_.txt"; OutputSandbox = {"colour_PARAM_.txt","error_PARAM_.txt"};

  37. PARAMETRIC JOBS Type = "Job"; JobType = “Parametric"; Executable = "exec"; StdInput = "input_PARAM_.txt"; StdOutput = "output_PARAM_.txt"; Parameters = 10; ParameterStart = 1; ParameterStep = 1; InputSandbox = {"<exec>", "<param_file>"}; OutputSandbox = {"output_PARAM_.txt"};

  38. DAG JOBS [ Type = "dag"; InputSandbox = {"job.sh"}; Nodes = [ nodeA = [ file ="A.jdl" ; ]; nodeB = [ file =“b.jdl" ; ]; nodeC = [ file =“C.jdl" ; ]; nodeD = [ Description = [ JobType = "Normal"; Executable = "job.sh"; StdOutput = "std.out"; StdError = "std.err"; InputSandbox = {root.InputSandbox[0]}; OutputSandbox = {"std.out","std.err"}; ]; ]; ]; Dependencies = { {nodeA,nodeB},{nodeA,nodeC},{{nodeB,nodeC},nodeD} }; ]

  39. MPI JOBS JDL file: Type = "Job"; JobType = "MPICH"; NodeNumber = 5; Executable = "mpi.sh"; Arguments = "mpi"; StdOutput = "mpi.out"; StdError = "mpi.err"; InputSandbox = {"mpi.sh","mpi.c"}; OutputSandbox =\ {"mpi.err","mpi.out","mpiexec.out"};

  40. LINKS http://aegis-ca.rcub.bg.ac.yu/ https://edms.cern.ch/file/722398//gLite-3-UserGuide.pdf http://wiki.egee-see.org/index.php/Quick_User_Guide_for_Submitting_Jobs https://edms.cern.ch/document/590869/1/ http://wiki.egee-see.org/index.php/SEE-GRID_MPI_User_Guide http://wiki.egee-see.org http://wiki.egee-see.org/index.php/Data_Management_Web_Portal_-_User_Guide

More Related