1 / 76

Porting applications to Globus Toolkit 3.0 and designing an OGSI-based architecture

Porting applications to Globus Toolkit 3.0 and designing an OGSI-based architecture. Pawel Plaszczak and Rich Wellner. Perfringo Grid Solutions http://perfringo.com. Tutorial materials: http://perfringo.com/events. About the authors. From Foster’s Globus team in Argonne

megan-kerr
Download Presentation

Porting applications to Globus Toolkit 3.0 and designing an OGSI-based architecture

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Porting applications to Globus Toolkit 3.0 and designing an OGSI-based architecture Pawel Plaszczak and Rich Wellner Perfringo Grid Solutions http://perfringo.com Tutorial materials: http://perfringo.com/events

  2. About the authors • From Foster’s Globus team in Argonne • Co-founded Perfringo Grid Solutions • Focus: industry level Grid applications • Currently: NeesGRID - distributed earthquake simulations • pawel@perfringo.com, rw2@perfringo.com Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  3. Prerequisities • Necessary: distributed systems & software design, purpose of Grid • Useful: Java, Web services, XML, Grid services (will be briefly explained) • Not necessary: coding experience • We will speak on decision making and design. No hands-on programming Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  4. Plan • Part I: Grid services - why and when? • Part II: OGSI and OGSA technology review • Part III: Use case: NTCP • Part IV: Porting to Grid services and GT3 Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  5. Plan of Part I: Grid services - why and when? • Overview: Globus Alliance and Toolkit • Overview: GT2 vs GT3, OGSA • Why go Grid? • Why comply with OGSA? • Why port to GT3? Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  6. The Globus Alliance • Globus Project ™, since 1996 • Ian Foster (Argonne National Lab), • Carl Kesselman (University of Southern California’s Information Science Institute) • Develop protocols, middleware and tools for Grid computing • Globus Alliance, since Sept 2003 • International scope • University of Edinburgh’s EPCC • Swedish Center for Parallel Computers (PDC) • Advisory council of Academic Affiliates from Asia-Pacific, Europe, US Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  7. Globus Toolkit • GT2 (2.4 released in 2002): reference implementation of Grid fabric protocols • GRAM for job submissions • MDS for resource discovery • GridFTP for data transfer • GSI security • GT3 (3.0 released July 2003): redesign • OGSI based • Grid services, built on SOAP and XML Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  8. OGSI and OGSA • OGSI = Open Grid Service Infrastructure • Specs from GGF OGSI working group • Defines what makes a Grid service • Based on Web service • Naming, life cycle, state, notification • portTypes definitions, WSDL 1.2 draft • OGSA = Open Grid Service Architecture • Specs from GGF OGSA working group • Defines a list of fundamental Grid services, and how they cooperate • Work in progress Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  9. What’s new in GT3.0 • Emphasis on OGSI compliant Grid services • OGSA implemented • Uniform SOAP/XML communication layer • Focus on Java implementation • Old GT2 services redesigned • Backward compatibility where possible Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  10. Meaning of GT3 in the community • Most commonly referred project • Traditionally “de facto standard” • Acknowledged leadership by academia and industry (IBM,…) • BSD style license allows for commercial usage • However, it is only a reference implementation. Now standards = GGF • GT undergoes constant changes • With business entering grid, commercial implementations may soon catch up Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  11. Alternatives to GT3 • Protocol level interoperability: • “Grid compliant” >= “implements OGSI” • Other OGSI implementations • OGSI.NET (U.Virginia) • pyGlobus (LBNL) • .NET (U.Edinburgh) • PERL (U. Manchester) • UNICORE (Fujitsu) • Commercial OGSI compliant products by: • Avaki, Platform, Data Synapse, … • Web service alternative: Grid App Framework http://www.neresc.ac.uk/ws-gaf Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  12. Part I contd. • Overview: Globus Alliance and Toolkit • Overview: GT2 vs GT3, OGSA Why? • Why go Grid? • Why comply with OGSA? • Why port to GT3? • upgrading from GT2 Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  13. Why go Grid? • Hot subject • Try it, experience it to learn the potential • Will enable true ubiquitous computing in future • Today, proven in some areas: intraGrids • But still long way to World Wide Grid • State of art techniques, tools are difficult • Short term goals? Use another technology • Does your system have Grid characteristics? • Distributed users, large scale and heterogeneous resources, across domains Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  14. OGSI architecture: why and when? • Commonly accepted Grid standard • But there are alternatives: WS-GAF, GT2,… • Extends Service-Oriented Architecture • Modular, scalable, maintainable, interoperable • Extensions: use only when needed • Uniform communication layer • Be compatible and inter-operable with other grid systems (that’s what grids are about!) Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  15. Porting to GT3: why and when? • Implementing OGSI is plenty of work! • Piggyback on GT3 for OGSI/A compliance (watch commercial products) • Most serious middleware to consider • Language bindings: traditionally Unix/C, now Java leads • But: unstable, new versions bring changes • Traditionally, for hard core hackers • Traditionally, documentation was lacking • recent signs of improvement, also IBM redbook Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  16. GT3 stability • GT3 took only 1.5 year to develop. It will change many times! • Anticipated or considered changes: • Refactor GridFTP control channel into XML, DAI • Security: add CASS • GRAM: ongoing improvements • OGSA-C hosting environment to come • OGSA specs is draft • OGSI specs are completed (v. 1.0) Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  17. GT3.2 • Expected January 2004 • Features: • CAS (Community Authorization Service) • Index Service reimplemented • GridFTP server reimplemented • GridFTP java client: connection reuse, MLST • GRAM performance improvements • CRL support (certificate revocation list) • several bug fixes in core Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  18. Java CoG Kit status • Before: Java client for GT2 • Now integrated with GT3 • Provides low level API for GT3 services: security, GridFTP • Experimental features to come soon (not part of GT3): • Workflow management, compliant with GT2 and GT3 • Comes with example Jetspeed portlets Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  19. roadmap • Part I: Grid services - why and when? • Part II: OGSI and OGSA technology review • Part III: Use case: NTCP • Part IV: Porting to Grid services and GT3 Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  20. Plan of Part II: OGSI and OGSA technology review • Web service • XML and SOAP • Service Oriented Architecture • Grid service (OGSI) • OGSA • How does GT3 implement OGSA • Examples Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  21. What is a Web service • Not much to do with HTML • Interface to a well-explained, self-contained function (module, component) • Usu. small grained • Defined in WSDL • Communication: XML and SOAP Web service client SOAP/XML OnlineStore Web service queryItems() orderItem() Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  22. More on Web services • Specs • W3 Consortium www.w3.org, IBM, Microsoft • OASIS (WS security) • For developers: “Building Web services with Java” by Steve Graham et al • For managers: “Web services and Service-Oriented Architectures” by Doug Barry (the savvy manager’s guide) • Stay tuned: IBM developerWorks articles • http://www-106.ibm.com/developerworks/webservices/ Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  23. Service-Oriented Architecture • System composed of several components exhibiting WS interfaces • Discovery through a well-known directory, such as UDDI • External services use internal ones • Modular, scalable, maintainable, interoperable Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  24. Service-Oriented Architecture in practice OnlineStore WS Web service client queryItems() orderItem() query order StockMgr WS OrderMgr WS query() ship() ship order() bill() warehouse Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  25. What is a Grid service • Defined by OGSI (GGF working group) • Is a Web service with additions, which are: • Name (handle GSH, reference GSR) • Lifetime management (factories, persistent and transient services) • State (Service Data) • Notification as well as querying • WSDL 1.2 draft (gwsdl: namespace) • Definitions of portTypes Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  26. Grid services in practice: notifications and service data Notification source Subscribe: “tell me when you have it on stock” OnlineStore GS Grid service client query() subscribe() Notify: “it just arrived.” Notification sink StockItem SDE Service data element Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  27. Grid services in practice: naming and life cycle OnlineStore Persistent GS Grid service client 1. query queryItems() 4. query 2. Get reference StockItemAF45X Transient GS 3. create StockMgr Persistent GS getItem() Item SDEs: dimension, price Factory Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  28. What is OGSA • Open Grid Service Architecture, being defined by GGF OGSA working group • In ubiquitous Grid platform, there is common need for some basic services • OGSA defines this set of base services • OGSA builds on OGSI Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  29. How does GT3 implement OGSA • Java implementation, C to follow • Container approach: host small-grained services • Hosting environment takes care of basic GS functions • GT3 provides core and base services, users to implement their own • Transient and persistent GS API • Tools to generate Java <-> GWSDL • Axis engine used for SOAP handling • Run with Tomcat or standalone Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  30. roadmap • Part I: Grid services - why and when? • Part II: OGSI and OGSA technology review • Part III: Use case: NTCP • Part IV: Porting to Grid services and GT3 Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  31. Plan of part III: NTCP • NeesGRID project overview • NTCP requirements & high level design • Choosing Grid services for NTCP subsystem • Choosing Grid services for Matlab subsystem • Not choosing Grid services for ShoreWestern subsystem • Deployment • On-going work Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  32. NeesGRID • Make collective use of geographically distributed experiment facilities, heterogeneous hardware • Users dislocated from facilities • Applying Grid to earthquake science • 3 year project involving several US institutes • Distributed simulation of a building in an earthquake • Scale: models costs millions of $ and several months to build. Some experiments are not repeatable (model used once) Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  33. The MOST experiment UIUC Experimental Model U. Colorado Experimental Model NCSA Computational Model m1 f2 f1 f2 f1 m1 Slide courtesy of Bill Spencer, UIUC Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  34. MOST requirements • Distributed simulation of a building in an earthquake • 3 experimental sites • Hybrid experiment: 2 physical and 1 computational simulations • Results at one time-step used to calculate inputs at the next time-step • Pseudodynamics: order of 10 ms per time-step • Successfully run on 30th July 2003 Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  35. NTCP: NEESGrid Teleoperation Control Protocol • Vital component of NeesGRID and MOST • Telecontrol system, used to coordinate the experiment with commands • Output data collected with other NEESGrid systems like NSDS Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  36. NTCP system architecture Coordinator (NTCP client) NTCP subsystem NTCP NTCP NTCP NTCP server NTCP server NTCP server Local protocol Local protocol Local protocol local subsystems Backend ShoreWestern Backend Matlab sim. Backend Matlab + XPC UIUC NCSA U Colorado Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  37. NTCP protocol • Coordinator follows the script with recorded earthquake data • Coordinator orders servers to simulate one time-step at a time (pseudodynamics) • Servers can refuse if parameters out of range • Core commands • Propose() negotiates an action • Execute() starts an action • Cancel() issued when negotiation failed • Other: get/setParameter, open/closeSession… Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  38. Sequence diagram (simplified) NTCP Server A NTCP Server B coordinator openSession openSession propose (accepted) execute 1 time step (started) execution results propose Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  39. NTCP subsystem: choosing Grid services and GT3 • Long term goals • System characteristics: distributed, heterogeneous resources across administrative domains • Interoperability, manageability, scalability • Risk: GT3 released early July, MOST scheduled end of July Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  40. NTCP server simplified design Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  41. NTCP server as a Grid service • Front end: Permanent Grid service • Operates on real hardware; only admin can restart • methods • Open/closeSession, propose, execute, cancel, get/setParameter, getControlPoint • Service data: Transaction state • Notification: Execution results • Back end: LocalControlPlugin interface for implementing adapters to the local non-OGSI technologies Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  42. Sequence diagram revisited in Grid service terms NTCP Grid service Coordinator (GS client) Propose transaction Transaction Service Data create (accepted) Subscribe for transaction status 1 time step Execute transaction execution (started) update Transaction results notification results Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  43. Matlab subsystem: plugin and backend Coordinator (NTCP client) NTCP NTCP NTCP NTCP server NTCP server NTCP server Local protocol Local protocol Local protocol Backend ShoreWestern Backend Matlab sim. Backend Matlab + XPC Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  44. Matlab subsystem: characteristics • Command flow: • Coordinator • NTCP server frontend • NTCP server Matlab control plugin (Mplugin) • Backend machine running Matlab simulation • Matlab single threaded, not event driven • Information pull model enforced • Matlab has Java interface, fully compliant with Java 1.3. Can use Java class library, third party jar files, multiple threads Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  45. Matlab subsystem: choosing Grid services • Choose network communication technology between Mplugin and backend • TCP/IP, RMI, Corba, Web services… • Although no Grid characteristics… why not Grid services? • Pulling scenario -> “server” on the same machine as the NTCP server • OGSA container is already there • Ockham razor: one technology instead of many Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  46. Typical Service-Oriented Arch compared to NTCP + Matlab Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  47. Client, server and backendin Grid service model • Methods: • hasCommand, pullCommand, pushResult • No service data, notification • Could as well be a Web service • Reversed information flow, as compared to typical Service-Oriented • Both NTCP and commandBuffer services are persistent; must match at start-up Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  48. Testing the NTCP and Matlab subsystem • Junit and TestServer from GT3 • 3 test suites, 4 test units • Some units implement same interfaces • 3 simulators, some deployed inside gar Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  49. ShoreWestern subsystem Coordinator (NTCP client) NTCP NTCP NTCP NTCP server NTCP server NTCP server Local protocol Local protocol Local protocol Backend ShoreWestern Backend Matlab sim. Backend Matlab + XPC Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

  50. ShoreWestern subsystem:plugin and backend • ShoreWestern hardware driver to operate physical experiment facility • Push communication OK • ShoreWestern already speaks a proprietary protocol based on TCP/IP, equivalent in functionality to NTCP Porting to GT3 & designing OGSI - Perfringo Grid Solutions http://perfringo.com

More Related