1 / 34

Globus Toolkit vs Globus.org

Globus.org: High-Performance, Reliable, File Transfer SaaS for TeraGrid Stuart Martin Software Development Manager University of Chicago / Argonne National Lab. Globus Toolkit vs Globus.org. Globus Toolkit not intended for end-users Assortment of Components for Grid Builders

susane
Download Presentation

Globus Toolkit vs Globus.org

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Globus.org: High-Performance, Reliable, File Transfer SaaS for TeraGridStuart Martin Software Development ManagerUniversity of Chicago / Argonne National Lab

  2. Globus Toolkit vsGlobus.org • Globus Toolkit not intended for end-users • Assortment of Components for Grid Builders • Security, data movement, remote job management, Java SOA tools for Grid applications, resource catalog, etc. • GT components “under the hood” of most the world’s large, production, scientific Grids • TeraGrid, OSG, LHC Computing Grid, ESG, LIGO, SCEC, BestGrid, EGEE, BIRN, caBIG, … • But end-users need better end-to-end systems • Globus.org is a new system with this goal Globus.org

  3. Cloud Taxonomy Software-as-a-Service (SaaS) Platform-as-a-Service (PaaS) Infrastructure-as-a-Service (IaaS) Globus.org

  4. What is Globus.org File Transfer Service? • SaaS providing file transfer capabilities • Think RFT in the cloud • Asynchronous "Fire and Forget" interface • Useful for large amounts of data • Automatically recover from failures • Resume transfers interrupted by failures in the network, end systems, etc. “Just get my files from A to B without bothering me with the details” Globus.org

  5. Applying SaaS Techniques • Service: Built as scale-out web application • Hosted on Amazon Web Services • Client: Minimize software deployment • Web 2.0 • AJAX + REST • Notification via email, IM, SMS, Twitter, etc. • Enable mash-ups • CLI 2.0 • sshcli.globus.org … • Mobile clients Globus.org

  6. Why SaaS? • Requires no user software installation • Minimal IT proficiency required • Efficient software delivery lifecycle • Updates developed, tested and deployed quickly • Consolidated troubleshooting and support • An expert group can proactively detect and correct problems • Partnering with Argonne/UC IT support group who specialize in support and ops Globus.org

  7. Application Grid Architecture “Coordinating multiple resources”: ubiquitous infrastructure services, app-specific distributed services Collective “Sharing single resources”: negotiating access, controlling use Resource “Talking to things”: communication (Internet protocols) and security Connectivity “Controlling things locally”: Access to, and control of resources Fabric “The Anatomy of the Grid: Enabling Scalable Virtual Organizations”, Foster, Kesselman, Tuecke, Intl Journal of High Performance Computing Applications, 15(3), 2001. Globus.org 7

  8. Distributed Data NodesAt The Resource Layer GridFTP GridFTP GridFTP GridFTP GridFTP GridFTP GridFTP Globus.org

  9. Globus.org Operates At The Collective Layer GridFTP GridFTP GridFTP Globus.org GridFTP GridFTP GridFTP GridFTP Globus.org

  10. GridFTP Control GridFTP Control GridFTP Data GridFTP Data Globus.org Manages3rd-Party Transfers Globus.org Globus.org

  11. A Peek Inside Globus.org Datastore GridFTP Worker Worker Consumer Worker Consumer Worker Consumer GridFTP Worker Consumer Request collector Notificationtarget Globus.org

  12. CEDPS Data Challenges • Challenge #3 • Run by Lisa Childers (UC/ANL) • 100k files, 200MB per file • 11 users, 12 sites Globus.org

  13. Globus.org

  14. Globus.org

  15. Globus.org

  16. Globus.org

  17. Globus.org

  18. Challenge 3 Recorded Errors Globus.org

  19. Target Users • Ad-hoc: IT non-expert end users who need to move substantial files around ad-hoc • Simple commands, geared for human interaction, that just work • Scripted: IT proficient end-users who want to automate substantial file movement • More sophisticated commands, geared for automated scripting, that just work • System builders: Programmers who don’t want to re-build this sophisticated functionality • Programmatic APIs, that just work Globus.org

  20. Example User: Ad-hoc • Nuclear Physicists sharing data • Hai Ah Nam, a nuclear physicist from Oak Ridge spoke at GlobusWorld March 2010 about the struggles with moving data • Transferring 1.6TB (86 large files) from Oak Ridge to NERSC. • Changed from using scp to GridFTP to reduce transfer from days to hours • Reduced transferring 137 TB from months to days • But, it was not easy... Globus.org

  21. Example User: Ad-hoc • ENZO Data Visualization • Joe Insley a visualization specialist from Argonne needed to move data from a compute cluster to visualization cluster • 500 256GB files from Kraken to ALCF Intrepid • Ran multiple sets of 5 simultaneous globus-url-copy commands to do the transfer • But, “Very Tedious” it was not easy... Globus.org

  22. Example User: Ad-hoc • NERSC: High Performance Scientific Computing Facility • Has many (100s) scp users that transfer large (> 20 GB) of data to/from NERSC • Want higher performance & reliable transfers • With simplicity of scp command • Next globus.org release will have scp look-alike • Extra benefits • Detached/backgrounded transfer: async, fire and forget • Automatic recovery from network & end-system failures • Load balancing & fail-over • More coming soon: end-to-end verification, sync, … Globus.org

  23. Example User: Scripted • CI for Climate Scientists • Chan Wilson (GFDL) is a sys admin / tools script developer for a group of climate scientists • Creating automation scripts for sustaining 80TB / day of simulation output from remote supercomputers to GFDL • ES-net w/ striped GridFTP endpoints • 12 DTN/GridFTP servers attached to GFDL cluster fs • He is counting on globus.org to help him move that data reliably with the performance required Globus.org

  24. Example User: System Builder • High Throughput Computing Integration • Condor supports file stage-in/out for each job • Condor is adding support for Globus.org for staging files to/from compute hosts to off-site locations • Will be available in version 7.6.0 (~Nov 2010) Globus.org

  25. Example User: System Builder • Science gateways • SCEC, AMP, ESG, … Globus.org

  26. Globus.orgGet Started Now! • Beta version is open for testing • cli.globus.org • Globus.org account request • www.globus.org/service/accounts • Quickstart guide • www.mcs.anl.gov/~childers/quickstart Globus.org

  27. Parts of the CLI Command Or instead of sshprepend with:gsissh –p 2222 –t … Globus.org

  28. CLI: xfer(advanced transfer request) • Example of URL source dest pairs in an input file alcf:/example/file1 NERSC:/dest/file1 alcf:/example/file2 NERSC:/dest/file2 Globus.org

  29. CLI: summary • Overview of the transfer state Globus.org

  30. CLI: scp(simple transfer request) $ sshlcc@globus.org \scp -D -v -r alcf:/example/dir1 \pads:/dest/new-dirCreated group 4ad85284-9ef9-11df-962b-1231390bde61 Globus.org

  31. CLI: activate(credential delegation) Globus.org

  32. CLI: manage-endpoint(optional in next release) Globus.org

  33. Coming Soon • scp look-alike command • Recursive directory transfers • REST API that with similar functionality as CLI • Web App for user profile management Globus.org

  34. Acknowledgements Several people have contributed to this work, including: Bryce Allen, Joshua Boverhof, John Bresnahan, Lisa Childers, Paul Dave’, Fred Dech, Ian Foster, Dan Gunter, GopiKandaswany, Nick Karonis, Raj Kettimuthu, Jack Kordas, Lee Liming, Mike Link, Stu Martin, JP Navarro, Karl Pickett, Mei Hui Su,Steve Tuecke, Vas Vasiliadis Funders • DOE and NSF Globus.org

More Related