1 / 35

The Globus Grid Programming Toolkit: A User-level Tutorial

The Globus Grid Programming Toolkit: A User-level Tutorial. The Globus Project Team ANL and USC/ISI http://www.globus.org. Abstract.

chapa
Download Presentation

The Globus Grid Programming Toolkit: A User-level Tutorial

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Globus Grid Programming Toolkit:A User-level Tutorial The Globus Project Team ANL and USC/ISI http://www.globus.org

  2. Abstract This tutorial is a practical introduction to programming for high-performance distributed computing systems, or "computational grids," and the capabilities of the Globus grid toolkit. Emerging high-performance networks promise to enable a wide range of emerging application concepts such as remote computing, distributed supercomputing, tele-immersion, smart instruments, and data mining. However, the development and use of such applications is in practice very difficult and time consuming, because of the need to deal with complex and highly heterogeneous systems. The Globus grid programming toolkit is designed to help application developers and tool builders overcome these obstacles to the construction of "grid-enabled” scientific and engineering applications. It does this by providing a set of standard services for authentication, resource location, resource allocation, configuration, communication, file access, fault detection, and executable management. These services can be incorporated into applications and/or programming tools in a "mix-and-match" fashion to provide access to needed capabilities. Our goal in this tutorial is both to introduce the capabilities of the Globus toolkit and to show attendees how Globus services can be applied in specific applications. Hence, the tutorial covers a mixture of grid programming principles and detailed case studies of real applications.

  3. Tutorial Goals • Provide an introduction • To the structure of the Globus computational grid • To the capabilities of the Globus toolkit • To pragmatic issues associated with using the toolkit • Enable attendees • To start building & using Globus applications • To utilize Globus services

  4. Overview • Introduction to computational grids • High-level overview of the Globus toolkit • Four components: • Security and remote process creation • Running programs across multiple resources • Information services • Dynamic configuration and resource management • Case studies • Other Globus services, and future directions • Globus installation & administration

  5. Why “The Grid”? • New applications based on high-speed coupling of people, computers, databases, instruments, etc. • Computer-enhanced instruments • Collaborative engineering • Browsing of remote datasets • Use of remote software • Data-intensive computing • Very large-scale simulation • Large-scale parameter studies

  6. E.g.: Computer-Enhanced Instruments for Microtomography • Coupling with supercomputers • Interactive use of beamline • Collaboration on results • Parameter studies for experiment planning • Coupling with mass store systems 50 Mb/s -> 5 Gb/s -> 100 Gb/s APS beamline @ Argonne “100 Gflop/sec, 50 Mb/sec, 5 minutes; rendering, 10 GB storage” 5 Mb/s -> 1 Gb/s -> 10 Gb/s Chicago Los Angeles

  7. Control Text Video Audio Database E.g.: Tele-immersion “5 Gflop/sec, flowspecs, design db” Multiple access modalities Multiple flows • Simulation • Tracking • Haptics • Rendering Leigh et al., UofI, Electronic Visualization Lab.

  8. SF-Express: Distributed Interactive Simulation • Issues: • Resource discovery, scheduling • Configuration • Multiple comm methods • Message passing (MPI) • Scalability • Fault tolerance Caltech Exemplar NCSA Origin Maui SP Argonne SP “200 GB memory, 100 BIPs” P. Messina et al., Caltech

  9. The Grid “Dependable, consistent, pervasive access to [high-end] resources” • Dependable: Can provide performance and functionality guarantees • Consistent: Uniform interfaces to a wide variety of resources • Pervasive: Ability to “plug in” from anywhere

  10. Evolution of a Concept • Metacomputing: late 80s • Focus on distributed computation • Gigabit testbeds: early 90s • Research, primarily on networking • I-WAY: 1995 • Demonstration of application feasibility • PACIs (National Technology Grid): 1998 • NASA Information Power Grid: 1999 • ASCI DISCOM: 1999; SSI: 2000?

  11. I-WAY The Alliance National Technology Grid National and International Grid Testbeds NASA’s Information Power Grid

  12. Technical Challenges • Complex application structures, combining aspects of parallel, multimedia, distributed, collaborative computing • Dynamic varying resource characteristics, in time and space • Need for high & guaranteed “end-to-end” performance, despite heterogeneity and lack of global control • Interdomain issues of security, policy, payment

  13. Issues • Authenticate once • Specify simulation (code, resources, etc.) • Locate resources • Negotiate authorization, acceptable use, etc. • Acquire resources • Initiate computation • Steer computation • Access remote datasets • Collaborate on results • Account for usage Domain 1 Domain 2

  14. Architectural Approaches • Distributed systems: DCE, CORBA, Jini, etc. • Rich functionality eases app. development • Complexity hinders deployment • especially in absence of global control • Performance difficulties • Internet Protocol, Web tools • Simple protocols facilitate deployment • Missing functionality hinders app. development • Performance difficulties

  15. Standards & Commodity Tech • Where appropriate, exploit standards and commodity technology in core infrastructure • LDAP, SSL, X.509, GSS-API, GAA-API, http, ftp, XML, etc. • Provides leverage • Interface with other common standards • CORBA, Java/Jini, DCOM, Web, etc • While our core infrastructure may not be built on one of these distributed architectures, we must cleanly interface with them

  16. The Globus Project • Basic research in grid-related technologies • Resource management, QoS, networking, storage, security, adaptation, policy, etc. • Development of Globus toolkit • Core services for grid-enabled tools & applns • Construction of large grid testbed: GUSTO • Largest grid testbed in terms of sites & apps • Application experiments • Tele-immersion, distributed computing, etc.

  17. Globus Approach • A toolkit and collection of services addressing key technical problems • Bag of services model • Not a vertically integrated solution • Inter-domain issues, rather than clustering • Integration of intra-domain solutions • Distinguish between local and global services • “IP hourglass” model

  18. Technical Focus & Approach • Information-rich environment • Basis for configuration and adaptation • Enable incremental development of grid-enabled tools and applications • Support many programming models, tools, applications • Deploy toolkit on national-scale testbed to allow large-scale applications • Evolve in response to user requirements

  19. Globus Approach • Focus on architecture issues • Propose set of core services as basic infrastructure • Use to construct high-level, domain-specific solutions • Design principles • Keep participation cost low • Enable local control • Support for adaptation A p p l i c a t i o n s Diverse global services Core Globus services Local OS

  20. Local Services Condor MPI TCP UDP LSF Easy NQE AIX Irix Solaris Layered Architecture Applications High-level Services and Tools GlobusView Testbed Status DUROC MPI MPI-IO CC++ Nimrod/G globusrun Core Services Nexus GRAM Metacomputing Directory Service Globus Security Interface Heartbeat Monitor Gloperf GASS

  21. Core Globus Services • Communication infrastructure (Nexus, IO) • Information services (MDS) • Network performance monitoring (Gloperf) • Process monitoring (HBM) • Remote file and executable management (GASS and GEM) • Resource management (GRAM) • Security (GSI)

  22. Sample of High-Level Services I • Communication & I/O libraries • MPICH, PAWS, RIO (MPI-IO), PPFS, MOL • Parallel languages • CC++, HPC++ • Collaborative environments • CAVERNsoft, ManyWorlds • Others • MetaNEOS, NetSolve, LSA, AutoPilot, WebFlow

  23. Sample High-Level Services II • Resource brokers and co-allocators • DUROC: co-allocation of multiple systems • Nimrod: high-throughput computing • Graphical system status display elements • GlobusView • MDS Browsers • Health & Status Monitors (HBM) • Network Monitors (Gloperf)

  24. “GUSTO”Globus Ubiquitous Supercomputing Testbed Organization • A collection of organizations committed to creating a persistent computational grid infrastructure • As of November 1998, 70 organizations in 3 continents and 8 countries

  25. 16 sites, 330 computers, 3600 nodes, 2 Teraflop/s, 10 application partners

  26. GUSTO Testbed During SC’97

  27. GUSTO Computational Grid Testbed: November 1998

  28. Where We Are (November 1998) • New results in security, resource management, tools, fault detection, etc. • Globus v1.0 completed • All core services complete, relatively robust, and documented • Available on most Unix platforms • Many tool projects are leveraging this considerable investment in infrastructure • Interesting applications are emerging, although mostly still in “demo” mode

  29. Where We Are (June 1999) • New results in QoS, security, resource management, data management, tools, etc. • Globus v1.1 nearing completion • Available on most Unix platforms and Win32 • Many tool projects are leveraging this considerable investment in infrastructure • Documentation and deployment underway at NCSA and NASA IPG • Always looking for interesting applications

  30. Changes from 1.0 to 1.1 • Tutorial changes for 1.1 are denoted by • Name changes from Globus to Grid • Security and Information Service adopted as core Grid infrastructure by several organizations • Globus Security Infrastructure -> Grid Security Infrastructure • Metacomputing Directory Service -> Grid Information Service • Affects naming of APIs and tools • Numerous small API fixes, additions, changes • Cleanup of programs/tools • A few new modules (I/O, error objects)

  31. Example Application Projects • Computed microtomography (ANL, ISI) • Real-time, collaborative analysis of data from X-Ray source (and electron microscope) • Hydrology (ISI, UMD, UT; also NCSA, Wisc.) • Interactive modeling and data analysis • Collaborative engineering (“tele-immersion”) • CAVERNsoft @ EVL, Metro @ ANL • X-Ray crystallography (ANL, SUNY) • High-throughput computing for Shake ‘n Bake

  32. Example Application Expts (contd) • Distributed interactive simulation (CIT, ISI) • Record-setting SF-Express simulation • Remote visualization and steering for astrophysics • Including trans-Atlantic experiments • Data-intensive computing experiments (with LBNL and SLAC: “Clipper” project)

  33. For More Information on Globus http://www.globus.org • Papers on all components • Tutorial and documents • Software • Application descriptions

  34. The Grid:Blueprint for a New Computing InfrastructureI. Foster, C. Kesselman (Eds),Morgan Kaufmann, 1999 • Available July 1998; ISBN 1-55860-475-8 • 22 chapters by expert authors including Andrew Chien, Jack Dongarra, Tom DeFanti, Andrew Grimshaw, Roch Guerin, Ken Kennedy, Paul Messina, Cliff Neuman, Jon Postel, Larry Smarr, Rick Stevens, and many others “A source book for the history of the future” -- Vint Cerf http://www.mkp.com/grids

  35. Tutorial Approach • Four sections, each illustrates a basic Globus technique • Laboratory material is available to allow practice with the use of each technique • See http://www.globus.org/tutorial

More Related