1 / 41

Overview of TeraGrid Resources and Usage

Overview of TeraGrid Resources and Usage. Selim Kalayci Florida International University 07/14/2009. Note: Slides are compiled from various TeraGrid Documentations. What is the TeraGrid?. TGUP (TeraGrid User Portal). Accessing TeraGrid User Portal. Accessing TeraGrid User Portal.

tanner
Download Presentation

Overview of TeraGrid Resources and Usage

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Overview of TeraGrid Resources and Usage Selim Kalayci Florida International University 07/14/2009 Note: Slides are compiled from various TeraGrid Documentations

  2. What is the TeraGrid?

  3. TGUP (TeraGrid User Portal)

  4. Accessing TeraGrid User Portal

  5. Accessing TeraGrid User Portal

  6. Portal Overview

  7. Portal: MyTeraGrid

  8. Portal: Resources

  9. Portal: Documentation

  10. Accessing Resources

  11. Web-based SSO via Portal

  12. Web-based SSO via Portal

  13. SSO (Non-Portal) from a TeraGrid Resource

  14. Example - SSH to tg-login.ncsa.teragrid.org or another resource that you have SSH access. - grid-proxy-info - myproxy-logon -lusername • grid-proxy-info • gsisshtg-login.purdue.teragrid.org

  15. TeraGrid Resources http://www.teragrid.org/userinfo/hardware/ Sorted by site Sorted by machine type http://portal.teragrid.org My TeraGrid -> Accounts Resources http://www.ncsa.uiuc.edu/UserInfo/Resources/ URLs here contain detailed user documentation

  16. Moving data to/from TeraGrid systems Sftp clients from your office to TeraGrid Command line sftp on Linux GUI sftp clients GSI-SSHTerm sftp button http://portal.teragrid.org Resources -> File Manager [beta] High speed gridftp between TeraGrid systems globus-url-copy Uberftp

  17. File Transfers: Small ( <100 MB) Files

  18. Large ( >100 MB) File Transfers: globus-url-copy • The globus-url-copy client program is a GridFTP client for transferring files from the command line. • Usage: • globus-url-copy <source_url> <destination_url> where <source> or <destination> is of the format: if local file, file:<full path> if remote file, gsiftp://<hostname>/<full path>

  19. Example – Two Party Transfer • Logon to NCSA Abe Cluster. • Create a large file on NCSA Abe Cluster: • dd bs=100MB count=1 if=/dev/zero of=testfile • Copy this file to Purdue Steele Cluster: • globus-url-copy –vb file:///u/ac/username/testfile gsiftp://tg-steele.purdue.teragrid.org:2811/autohome/u108/username/

  20. Example – Third Party Transfer • Logon to NCSA Abe Cluster • Copy the testfile at Purdue Steele Cluster to NCAR Frost Cluster • globus-url-copy -vb gsiftp://tg-steele.purdue.teragrid.org:2811/autohome/u108/username/testfile gsiftp://gridftp.frost.ncar.teragrid.org:2811//home/username/ • GridFTP server addresses for each site are listed at: • http://www.teragrid.org/userinfo/data/transfer_location.php#deployment

  21. Optimized Data Transfer with globus-url-copy • Using large TCP windows • globus-url-copy –vb-tcp-bs 1048576 file:///u/ac/username/testfile gsiftp://tg-steele.purdue.teragrid.org:2811/autohome/u108/username/ • Using large memory buffers • globus-url-copy –vb-bs 1048576 file:///u/ac/username/testfile gsiftp://tg-steele.purdue.teragrid.org:2811/autohome/u108/username/ • Using multiple parallel streams • globus-url-copy –vb–p 4 file:///u/ac/username/testfile gsiftp://tg-steele.purdue.teragrid.org:2811/autohome/u108/username/

  22. Large ( >100 MB) File Transfers: UberFTP • UberFTP is an interactive GridFTP file transfer client. • Opens a session with a remote host, within which files may be transfered and directories and files may be manipulated. • Requires GSI authentication. • Hands-On: • Login to NCSA Abe • uberftp • open tg-steele.rcac.purdue.edu • parallel 2 • tcpbuf 8388608 • ls/lls/put/get/…

  23. Data movement tips To move a collection of small files, make an archive and move it instead of moving the files individually tar zip For high bandwidth links and moderate file or archive size, do not compress, it's usually faster to just move the data [compression is a time waster] For low bandwidth links, compression is usually a time saver tar z or j options for compression zip ssh -C, sftp -C Imaginations unbound

  24. Permanent Storage at NCSA The larger TeraGrid sites provide persistent high-capacity storage Details vary by site, consult local site documentation for specifics Refer to: http://www.teragrid.org/userinfo/data/storage.php for detailed information about different quotas, policies, and tools (such as SRB, HPSS) at each site.

  25. Managing Your Environment: Softenv

  26. Softenv

  27. Managing Your Environment: Modules * Try at tg-steele.rcac.purdue.edu

  28. Softenv and Modules: Which do I use? • Rule of thumb: go with the default on a given machine • When you login for the first time issue ‘softenv’ and ‘module list’ commands • In general, only one should be active by default: go with that one • If you have questions or run into any problems contact help@teragrid.org

  29. Grid Job Management using Globus • Common WS interface to schedulers • Unix, Condor, LSF, PBS, SGE, … • More generally: interface for process execution management • Lay down execution environment • Stage data • Monitor & manage lifecycle • Kill it, clean up

  30. Grid Job Management Goals Provide a service to securely: • Create an environment for a job • Stage files to/from environment • Cause execution of job process(es) • Via various local resource managers • Monitor execution • Signal important state changes to client • Enable client access to output files • Streaming access during execution

  31. GRAM • GRAM: Globus Resource Allocation and Management • GRAM is a Globus Toolkit component • For Grid jobmanagement • GRAM is a unifying remote interface to Resource Managers • Yet preserves local site security/control • Remote credential management • File staging via RFT and GridFTP

  32. A Simple Example • First, login to queenbee.loni-lsu.teragrid.org • Command example: % globusrun-ws -submit -c /bin/date Submitting job...Done.Job ID: uuid:002a6ab8-6036-11d9-bae6-0002a5ad41e5Termination time: 01/07/2005 22:55 GMTCurrent job state: ActiveCurrent job state: CleanUpCurrent job state: DoneDestroying job...Done. • A successful submission will create a new ManagedJob resource with its own unique EPR for messaging • Use –o option to create the EPR file % globusrun-ws -submit –o job.epr -c /bin/date

  33. A Simple Example(2) • To see the output, use –s (stream) option % globusrun-ws -submit –s -c /bin/date Termination time: 06/14/2007 18:07 GMT Current job state: Active Current job state: CleanUp-Hold Wed Jun 13 14:07:54 EDT 2007 Current job state: CleanUp Current job state: Done Destroying job...Done. Cleaning up any delegated credentials...Done. • If you want to send the output to a file, use –so option % globusrun-ws -submit –s –so job.out -c /bin/date … % cat job.out Wed Jun 13 14:07:54 EDT 2007

  34. A Simple Example(3) • Submitting your job to different schedulers • Fork % globusrun-ws -submit -Ft Fork -s -c /bin/date (Actually, the default is Fork. So, you can skip it in this case.) • SGE % globusrun-ws -submit -Ft PBS-s -c /bin/date • Submitting to a remote site % globusrun-ws -submit -F tg-login.frost.ncar.teragrid.org -c /bin/date

  35. Batch Job Submissions % globusrun-ws -submit -batch -o job_epr -c /bin/sleep 50Submitting job...Done.Job ID: uuid:f9544174-60c5-11d9-97e3-0002a5ad41e5Termination time: 01/08/2005 16:05 GMT % globusrun-ws -status -j job_eprCurrent job state: Active % globusrun-ws -status -j job_eprCurrent job state: Done % globusrun-ws -kill -j job_eprRequesting original job description...Done.Destroying job...Done.

  36. Resource Specification Language (RSL) • RSL is the language used by the clients to submit a job. • All job submission parameters are described in RSL, including the executable file and arguments. • You can specify the type and capabilities of resources to execute your job. • You can also coordinate Stage-in and Stage-out operations through RSL.

  37. Submitting a job through RSL • Command: % globusrun-ws -submit -f touch.xml • Contents of touch.xml file: <job> <executable>/bin/touch</executable> <argument>touched_it</argument></job>

  38. Security - Basics

  39. How to get Help • First, try searching the Knowledge Base or other Documentation • If that doesn’t help, submit a ticket • Send an email to help@teragrid.org • Use the TeraGrid User Portal ‘Consulting’ tab • Can also call TeraGrid Help Desk 24/7: • 1-866-907-2383

  40. Submitting a Ticket

  41. More Info • TeraGrid Resource User Guides • http:www.teragrid.org/userinfo/hardware/resources.php • File Transfers and Data Management on TeraGrid • http:www.teragrid.org/userinfo/data • More Training • https://portal.teragrid.org/gridsphere/gridsphere?cid=onlinetraining

More Related