1 / 41

The Open Grid Computing Environments Collaboratory

The Open Grid Computing Environments Collaboratory. Marlon Pierce Community Grids Lab Indiana University www.collab-ogce.org. NSF NMI Project for Reusable Portal Components: Who We Are. University of Chicago/ANL Gregor von Laszewski Indiana University

judson
Download Presentation

The Open Grid Computing Environments Collaboratory

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Open Grid Computing Environments Collaboratory Marlon Pierce Community Grids Lab Indiana University www.collab-ogce.org

  2. NSF NMI Project for Reusable Portal Components: Who We Are • University of Chicago/ANL • Gregor von Laszewski • Indiana University • Marlon Pierce, Dennis Gannon, Geoffrey Fox, and Beth Plale • University of Michigan • Charles Severance, Joseph Hardin • NCSA/UIUC • Jay Alameda, Joe Futrelle • Texas Advanced Computing Center • Jay Boisseau • San Diego State University • Mary Thomas

  3. Presentation Outline • OGCE Overview • Portlet Standards for OGCE • New JSR 168 compatible Grid portlets • Demonstrate compatibility with uPortal and GridSphere containers • Grid Portlet Programming • Sample Application: TeraGrid User Portal • Sample Application: LEAD Portal

  4. What a Grid Portal Is/Is Not • It is • A tool for aggregating and managing web content • A user customizable view of these Web content pieces. • You see what you want/can see. • But you must log in. • Implemented on top of standard services • Like login, authorization, customization. • May include collaboration, etc, that depend on login. • A way to accomplish Grid tasks through browsers: • Launch, monitor jobs • Move files • Run science applications based on these services. • Compatible with emerging standards and best practices (such as portlets, JSR 168 and WSRP). • It is not (just) • A web page • A collection of links • An applet

  5. Three-tiered architecture is accepted standard for accessing Grid and other services Three-Tiered Architecture Grid and Web Protocols JDBC, Local, or Remote Connection Portal Client Stub Database Service Database Portal User Interface Portal Client Stub Grid Resource Broker Service HPC or Compute Cluster Portal Client Stub Information and Data Services Grid Information Services, SRB

  6. Building Portals from Reusable Components • The component architecture of choice for the Portal community is the one based on portlets • (Java) components that generate content, make local and remote connections to services. • Portlet containers manage portlet lifecycles • Standardized now by JSR 168 • A portlet is a piece of Java code similar to a servlet that does two things: • Creates a fragment of HTML display that becomes part of a web page • Handles any link clicks and HTML <Form> actions. • May involve accessing local and remote services • There are now many, many portlet components. • So don’t start from scratch. • JSR 168 is the Java standard for portlets.

  7. OGCE’s Release 1 • The OGCE Portal release is based on CHEF/Jetspeed 1.4 • Available for download and installation from http://www.collab-ogce.org. • It comes with many pre-configured capabilities if you want a grid portal “out of the box”. • Except for the mysql jar. • You must still set up Grid services (MyProxy servers, Globus, etc). • Globus version compatibility through the Java CoG. • Apache Ant-based installation procedure: • Edit one properties file, run ant, and away you go.

  8. User Portlets

  9. More User Portlets and Services

  10. What’s New for the OGCE2? • JSR 168 Compatible Grid portlet suite • Basic capabilities: MyProxy, GridFTP, GRAM, GPIR. • Working in uPortal, GridSphere containers • Container independent services for sharing data between portlet applications. • GSSCredential objects, global session data. • Limitation in JSR 168 • Support for Velocity development • Velocity is the Apache web application development tool of choice for Jetspeed1. • Provide backward compatibility with OGCE1 • Maven-based build and deploy system • Choose either uPortal or GridSphere • One-command install, but you must still install a Grid toolkit. • NMI GRID Center: www.grids-center.org • Modular portlet extensions • Collaboration tools, etc.

  11. Portlet Demo

  12. Grid Portlet Programming Marlon Pierce, Gregor von Laszewski, Eric Roberts

  13. Grid Programming Interfaces for Portlets • Portlet form actions may result in remote calls to Grid or Web Services • A Portlet is just java code • We use two programming APIs you can choose from • CoG4 API provides abstraction of Grid tasks to hide Grid toolkit version differences. • GridPort provides integrated information, CSF-based job submission and sequencing, and file manipulation services.

  14. Java CoG Kit and Portals • The Java CoG Kit is a bridge between Grids and Portals as it provides an abstraction layer that is supportive for portals developers. • The Goals of the Java CoG Kit include reuse of a variety of commodity tools, protocols, approaches, methodologies, while integrating Grid software to enable • Easier development of advanced Grid services • Easier and more rapid application development • Easier deployment of Grid services • Code reuse and use of component repositories • Use of Web services as part of the Grids • Widespread use of the Grid • Use of commodity technology is not limited to the client. • As a result we make Portal development and Grid computing easier

  15. Java CoG Kit Abstractions • We provide a number of abstractions that build the foundation why we make Grid programming easier: • File transfer, job submission, authentication • We provide a workflow abstraction that makes the specification of Grid workflows possible helping those with complex job management scenarios. • We provide a software methodology that makes adoption to new standards easier • We defined so called providers that allow reusing various Grid and commodity services including • GT2, GT3, GT4 (under development), SSH, Unicore and Condor will be provided by the community. This feature is unique amongst the Grid toolkits.

  16. Development Support Concept of the Java CoG Kit Architecture Nano materials Bio- Informatics Disaster Management Portals Applications Gridfaces Layer (portals, Swing, SWF) Data and Task Management Layer (workflow) GridIDE CoG Abstraction Layer (job submission, file transfer, authentication) CoG CoG CoG CoG CoG CoG CoG GT2 GT3 OGSA/ WS-RF SSH Condor Unicore Others Avaki Others SETI

  17. Java CoG Kit information • More information on the Java CoG Kit can be found at http://www.cogkit.org or visit the posters at SC2004 • P09: Karajan: A Grid Orchestration Framework:  Mihael Hategan, Gregor von Laszewski, Kaizar Amin • P11: The Next Generation of the Java CoG Kit (version 4)  Gregor von Laszewski, Kaizar Amin, Matt Bone, Mike Hategan, Pankaj Sahasrabudhe, Mike Sosonkin, Robert Winch,Nithya Vijayakumar, David Angulo

  18. What is GridPort? • High-level middleware that aids grid portal developers by easing use of low level grid tools • GridPort’s Role in Grid Computing • Aggregate services from grid software packages • Globus Toolkit • Community Scheduling Framework (CSF) • Storage Resource Broker (SRB) • Simple, consistent API • Custom Web services • GPIR • Job Sequencer • Advanced File Transfer • GridPort portlets are compatible with, part of OGCE releases.

  19. GPIR Admin Client GPIR Job Sequencer File Transfer Job Submission Data Access Adv. File Transfer GPIR Adv. File Transfer Job Sequencer J2EE Web Server J2EE App Server Contains: Spring, Hibernate Grid Data Portlets Web Services PostgreSQL GridPort Client API GridPort Service API JDBC HTTP/SOAP Grid Services Web Client GRAM HTTP / SSL OGSI WS-GRAM CSF GSI Security GridFTP SRB JSP/Servlet GridPort’s 3-tier Architecture

  20. GPIR Web Services Resources Information Providers DB Clients Portals Perl Client edu.tacc. gridport.gpir Portlets Query WS Ingester WS Java Client MDS GPIR PostgreSQL Other Middleware Web Scraping OGSA (Future) SOAP-XML HTTP JDBC Other

  21. GridPort Job Sequencer • Web service for simple job workflow composition • Portlet interface available for specifying sequences • Specify job submissions and file transfers • More information available at http://www.gridport.net

  22. CSF Integration • Community Scheduling Framework (CSF) • Meta-scheduler for submitting jobs to a grid • No need to specify a resource • CSF will schedule jobs to appropriate resource for you • OGSI grid service • CSF is an open source project developed by Platform computing • Now available for download from Sourceforge.net http://sourceforge.net/projects/gcsf/

  23. More GridPort Information GridPort Demo in TACC Booth 120 2:00-2:20 PM Thursday, Nov 11

  24. TeraGrid User Portal Eric Roberts Texas Advanced Computing Center

  25. Motivation • Make joining the TeraGrid easier for users • Single place for users to find user information and get user support • Certain information can be displayed better in a web page than in a command shell • Allow novice users to start using grid resources securely through a Web interface • Increase productivity of TeraGrid researchers – do more science!

  26. TeraGrid User PortalService Aggregation User Support Consulting Notification User News Collaborative Calendar Chat Documentation User Guides Information Resource Grid Interactive Job Submission File Transfer HTTP/SSL/SOAP TeraGrid User Portal HTTP/SSL Client Browser

  27. Current TeraGrid User Portal Capabilities • User Services • User Information (User Guides, Support,etc.) • Information Services • System • Grid • Network • Interactive • GSI Authentication • Remote Command Execution • Job Submission • File Management

  28. Future Directions • Central gateway for TeraGrid services • TeraGrid allocations and account creation/management through portal • Streamline the process • Application portals • Science gateways that expose scientific applications through interfaces • We will be contacting users for “friendly-user” testing (2005)

  29. LEAD Application Portal Marcus Christie Indiana University

  30. The LEAD Goal Provide the IT necessary to allow People (scientists, students, operational practitioners) and Technologies (models, sensors, data mining) TO INTERACT WITH WEATHER

  31. Traditional Methodology STATIC OBSERVATIONS Radar Data Mobile Mesonets Surface Observations Upper-Air Balloons Commercial Aircraft Geostationary and Polar Orbiting Satellite Wind Profilers GPS Satellites • Product Generation, • Display, • Dissemination Prediction/Detection PCs to Teraflop Systems • Analysis/Assimilation • Quality Control • Retrieval of Unobserved • Quantities • Creation of Gridded Fields The Process is Entirely Serial and Static (Pre-Scheduled): No Response to the Weather! • End Users • NWS • Private Companies • Students

  32. The Consequence: Model Grids Fixed in Time – No Adaptivity

  33. The LEAD Vision: No Longer Serial or Static STATIC OBSERVATIONS Radar Data Mobile Mesonets Surface Observations Upper-Air Balloons Commercial Aircraft Geostationary and Polar Orbiting Satellite Wind Profilers GPS Satellites • Product Generation, • Display, • Dissemination Prediction/Detection PCs to Teraflop Systems • Analysis/Assimilation • Quality Control • Retrieval of Unobserved • Quantities • Creation of Gridded Fields Models Responding to Observations • End Users • NWS • Private Companies • Students

  34. In LEAD, Everything is a Service • Finite number of services – they’re the “low-level” elements but consist of lots of hidden pieces…services within services. Service B (WRF) Service A (ADAS) Service C (NEXRAD Stream) Service F (IDV) Service E (VO Catalog) Service D (MyLEAD) Service I (ESML) Service H (Scheduling) Service G (Monitoring) Service L (Decoder) Service K (Ontology) Service J (Repository) Many others…

  35. Start by Building Simple Prototypes to Establish the Services/Other Capabilities… Service C (NEXRAD Stream) Service F (IDV) Service L (Decoder) Prototype X

  36. Start by Building Simple Prototypes to Establish the Services/Other Capabilities… Service C (NEXRAD Stream) Service F (IDV) Service E (VO Catalog) Service D (MyLEAD) Service L (Decoder) Prototype Y

  37. Start by Building Simple Prototypes to Establish the Services/Other Capabilities… Service A (ADAS) Service C (NEXRAD Stream) Service F (IDV) Service E (VO Catalog) Service D (MyLEAD) Service I (ESML) Service L (Decoder) Service J (Repository) Prototype Z

  38. Dynamic Workflow – Changes in Response To Observed Events, Model Output, User Input, new load saved

  39. Using the portal to launch A Distributed Grid Workflow The Demo

  40. The Big Picture The user’s browser 1. User buildsexperiment OpenDap Server In Alabama 4. decoderpushesdata toserver Portal Server 2. portrallaunches workflow 6. pushes metadatato opendap Data Decoder Indiana Threddscatalog Generator At NCSA Thredds Server At Unidata 5. tells cat.Generatorabout data 3. pulls datafrom server and decodes

More Related