GridLab 2003/4„Steady leadership in changing times!” Jarek Nabrzyski Project Coordinator Poznan Supercomputing and Networking Center Meeting with Sun Microsystems at PSNC: Exploitation, 13 May 2004
GridLab Project • Funded by the EU (5+ M€), January 2002 – December 2004 • Application and Testbed oriented • Cactus Code, Triana Workflow, all the other applications that want to be Grid-enabled • Main goal: to develop a Grid Application Toolkit (GAT) and set of grid services and tools: • Resource management (GRMS), • Data management (GDMS), • Monitoring (Mercury) and information services, • Adaptive components (Pythia), • Mobile user support and remote visualization, • Security services (GAS), • Portals (GridSphere), • ... and test them on a real testbed with real applications
GridLab Members • PSNC (Poznan) - coordination • AEI (Potsdam) • ZIB (Berlin) • Univ. of Lecce • Cardiff University • Vrije Univ. (Amsterdam) • SZTAKI (Budapest) • Masaryk Univ. (Brno) • NTUA (Athens) • Sun Microsystems GmbH • HP France • ANL (Chicago, I. Foster) • ISI (LA, C.Kesselman) • UoWisconsin (M. Livny) • collaborating with: • Users! • EU Astrophysics Network, • DFN TiKSL/GriKSL • NSF ASC Project • other Grid projects • Globus, Condor, • GrADS, • PROGRESS, Clusterix • GriPhyn/iVDGL, • European Grid Projects (GRIDSTART) • KISTI, Canadian Grid, LSU…
Review Issues and Concerns • It is GridLab’s first direct contact with the new PO and 2 out of 3 reviewers are new to the project, • Difficult to review a project of this size and complexity in several hours, • Decision to present the project from theintegration angle, • Problems with synchronization of deliverables with the last-minute amendment (details in the WP presentations) • Presentations: Short time per workpackage, short time for the demo, • Innovative GAT, the portal and services presented in a practical demo during SC2003 in Phoenix • The more features we add (to every single WP out of 12) the more complex the integration gets
GridLab Aims • Get Computational Scientists using the “Grid” and Grid services for real, everyday, production work (AEI Relativists, EU Network, Grav Wave Data Analysis, Cactus User Community, ...), • Make it easier for applications to make flexible, efficient, robust, use of the resources available to their virtual organizations • Dream up, prototype, and test new application scenarios which make adaptive, dynamic, wild, and futuristic uses of resources.
What GridLab isn’t • We are not developing low level Grid infrastructure - we do more actually, we want to be infrastructure independent, • We do not want to repeat work which has already been done (want to incorporate and assimilate it …) • Globus APIs, • OGSA/OGSI/…, • ASC Portal (GridSphere/Orbiter), • GPDK, • GridPort, • DataGrid, • GriPhyn, • ...
Project Goals for 2003 • Integration of all the GridLab software, • “Zakopane Scenario” driving all the developments and integration, • First release of GridLab software scheduled for December 2003, • Make the testbed more stable and ready for any demo action and/or development, • SC2003 was chosen to demo the GridLab software, • Strong dissemination of the project in order to get attention of interesting Grid applications, • Start the GAT and other GridLab standardization work under GGF, • Strengthen management of the project, with clear responsibilities to ensure better control of resources, fast identification of the risks, ensure progress, • GridLab Open Source License, • Prepare the plan for exploitation of the results, • Adjust the Technical Description of Work to the current needs of the project
What have we done during last 8 months? • Response to the reviewers report sent to all • Open Source License • GT2 vs. GT3 issues (finally it was wise decision) • Strategy for dissemination in 2004 • Deliverables, some delays here (see details in the WP presentations) • Contract amendment is on the way • GridLab meeting in Eger (April) and Olomouc (October) • More developments and integration • More and more WPs brought to the scenario • Today most of the WPs were integrated with each other • SC2003 in Phoenix (2 big „demos”, many „small” demos) • Painful demo, but successful: GRMS + portal + GAT application + Testbed + WP5 information service + Adaptive with some last minute problems with other services • „The devil is in the details” - the most true sentence.
Major achievements • Integration efforts and devotion of project staff, • Meeting in Eger in March/April, meeting in Olomouc in October, • Many small integration meetings (Poznan-Berlin-Potsdam, Potsdam-Budapest, Potsdam-Prague, Cardiff, GGF meetings and at other confs.), • Getting a major attention towards the GAT of many application communities, • GRIDSTART, GGF, application communities and 6FPP (EGEE, DEISA, HPC-Europa, …) • Expectations are high! • Many successful implementations of services • Single GridLab services being used by other projects, • Most of the services are fully integrated with each other, but can be configured as stand alone • GAT can connect to most of the services. All will be available in a month.
Major achievement (cont.) • GridLab middleware being exploited by other projects and production testbeds • GridSphere: tens of projects (Grid Infrastructure Project, GEON, HPC-Europa, Physics Portal Development Project, …) • GRMS (Clusterix, SGI Grid, Progress, Canadian Grid) • Mercury Monitor (DataGrid, P-GRADE) • WP8 Visualization - GriKSL • Open Source License • GridSphere has it’s own Open Source license
Major achievementsStandards, collaboration • Global Grid Forum • Positions in GGF • Alexander Reinefeld - GGF Advisory Board • Jarek Nabrzyski – step down from the SC and AD for Apps and now take care of the GSA RG), • Ed Seidel, Thilo Kielman chairing the Apps RG • Thilo Kileman chairing Grid CPR WG • Andre Merzky chairing the Persistent Archive WG and secretary of Data Area and Persistent Environments • GAT architecture put at the Apps and APM groups • Many WG talks about GridLab • Putting the GAT to the BOF in October 2003 • Very successful • Participation in Gridstart! • GridLab chairs at least half of the TWGs • Other • Projects: GriPhyN, GRIKSL, CrossGrid, Reality Grid (and some others) have a certain level of collaboration with GridLab • Companies: Grid Systems, Platform Computing, IBM, HP, Sun
Major achievementsDissemination • Improved the www.gridlab.org • Several new papers accepted for books, journals and conferences • Many talks have been given at major conferences and events • Events organized • GridLab Conference and Workshop in Eger, March-April 2003 • Grid Portals Conference, August 2003, Edinburgh • Grid Application Workshop in Czestochowa, September 2003 • GridLab Conference and Workshop in Olomouc, October 2003 • Grid Resource Management Workshop, Poznan, October 2003 • Workshops at international conferences and symposia • Tutorials on major conferences, also via AccessGrid • 2 GridLab brochures • In 2004 we will produce 14 different GridLab brochures featuring all the services, tools and advanced scenarios (see first three issues)
Book on Grid Resource Management • 4 GridLab chapters: • Application requirements for resource brokering • Multicriteria resource management (GRMS) • Triana peer-to-peer RM • Workflow management • ISI/Argonne/Wicsonsin contributions • Published in November by Kluwer Academic Publishers
Technical Progress • GAT API specification and GAT adaptor developer’s guide • GAT tested by Cactus and a GAT subset implementation for Triana (GAP) • Strenghten the collaboration with ISI, ANL, Wisconsin • New Portal Framework (GridSphere) - 1st release in 2003 (2nd in 2004) - being used by tens of projects! • Participation in the Grid Technology Repository (Jarek Nabrzyski serves as the Int. Advisory Board member) • Strong participation in the GRIDSTART project (GridLab was one of the main contributors to the Technical Roadmap) • Services ready and interoperable in spite of changing requirements • GAT ready for standardization process, already under the community comment process
Technical Progress (cont.) • Extending the GridLab scenario • Successful meeting in Eger (new application users involved, new developments) and in Olomouc (integration) • Problems/Issues • GAT development delayed - underestimated efforts needed for the GAT development • Should be fully ready in a month • Will be revisited and refactored! User’s input is needed. Good docs is needed. • Hiring a new person, experienced with Cactus development to speed up the CGAT integration. • GAT documentation is behind! We are shifting efforts a bit to speed up the process. • Some requirements and scenarios have changed dramatically: new ones appeared (WP7) • Some unexpected problems with hackers on one of the test machines
Issues • GAT development delayed - underestimated efforts needed for the GAT development • Should be fully ready in a month • Will be revisited and refactored! User’s input is needed. Good docs is needed. • Hiring a new person, experienced with Cactus development to speed up the CGAT integration. • GAT documentation is behind! We are shifting efforts a bit to speed up the process. • Some deliverables are delayed • New Annex1 caused some problems • GAT development delayed (CGAT, GAT documentation) • Some requirements and scenarios have changed dramatically: new ones appeared (WP7) • Some unexpected problems with hackers on one of the test machines
Plans for 2004 • Continue the development of GAT and services • more info in the following presentations • More complicated scenarios • collaborative environments, • submitting, controling and steering jobs from mobile devices, • more dynamic behavior of applications • Two more GridLab Workshop Meetings (Lecce: 16-22 May, Zakopane: early December) • Organizing the GridLab/GT3.2/WSRF integration meeting withUS partners (to ensure GAT compatibility) • Preparing for the Supercomputing demos • GGF BOF (GAT) and finally GGF WG • GGF Scheduling Architecture Working Group now started (GRMS) • Exploitation of the project results • Close work with GridLab’s commercial partners • Global Grid Application Alliance with GridLab’s leadership • GGF activities (long term) • GridSuit (based on GridLab plus Gridstart Open Source, PSNC plus partners) • Open Source + commercial support • European Grid Support Centre (PSNC)
Summary • GAT is important • We will put yet more effort into the GAT • GAT will be revisited after finalizing the 1st release (in one month) • Refactoring • Lecce Meeting to bring application people, TWG8, tutorial, exploitation • Documentation! • GGF/Gridstart • There was a great interest and we are talking to many groups • The project works better than ever! • Project is unique in this sense (this might be an impression of ours only) • Integration was much more difficult than expected • We had to deal with many (too many) unexpected problems in 3rd party software.
Collaboration with Argonne • OGSA C bindings • GridLab/GT3.2 Inegration Workshop (4 Argonne GT3 developers/experts are coming to help us with the integration) • Help with developing the GT3.2 adaptors • GAT review
Exploitation Plan Jarek Nabrzyski Meeting with Sun Microsystems at PSNC: Exploitation, 13 May 2004
Pillars of exploitation • Close work with GridLab’s commercial partners • Global Grid Application Alliance with GridLab’s leadership • GGF activities (long term) • GridSuite (based on GridLab plus Gridstart Open Source, PSNC plus partners) • European Grid Support Centre (PSNC, HP) • European Grid Technology Broker
Exploitation of the results through commercial partners • HP • Grid Resource Toolkit. HP investigates which GL software can be adopted. • Distribution through server/software distribution • Sun • This meeting! • Actions to be taken: • GridLab will work closely with the companies to ensure that the companies evaluate the software and make right decisions on which software is supposed to be part of their distributions • Responsible: WP managers will provide software plus documentation to a contact person at the vendor side.
Global Grid Application Alliance • Global • Europe - GridLab • US - Ed Seidel • Asia Pacific - both Satoshis • Korea - KISTI • Tasks • Bring the GAT and services to the community • Train people to make then use the GAT • People will write adaptors for their favorite services • Work on the new scenarios that the Alliance will bring • Work on portable applications on a global scale using the GridLab technologies • Actions: • The initial meeting of the founders took already place • SC will be responsible for the actions: meetings soon to be scheduled
GGF, GRIDSTART activities • SAGA • GSA • and more will be continued beyond the timeframe of the project • Gridstart TWGs will be used to work with other Grid projects in order to deploy the GAT and GridLab tools in these projects • TWG chairs (from GridLab) are responsible to make it happen (GridLab WP managers, SC members)
GridSuite at PSNC • GridSuite (based on GridLab plus Gridstart open source, PSNC plus partners) - Grid out-of-the-box complete solution • Open Source + commercial support • PSNC has declared to hire a team of 6-8 persons to build the GridSuite this year • PSNC will work with GridLab partners to provide an expertise to build the GridSuite and to design a business plan for the effort. • Release roadmap is under way. • This might be a background for the commercial company selling the Grid services for European industry • Actions to be taken: • PSNC will discuss this with the GridLab Steering Committee soon, • This will be more open to other GridLab partners (not limited to PSNC) • Strategy is being developed by PSNC now • Initial money from the European infrastructure funds (under way) and PSNC’s own funds.
European Grid Support Centre • European Grid Support Centre (HP, PSNC plus UK eScience, CERN, PDC) • Institutions’ own funds • The task is to support different user groups (currently DataGrid, EGEE, GridLab, NorduGrid) • The tools and the procedures are now being established. • The ground for GridLab’s support for its users and infrastructure builders, • Currently PSNC and HP from GridLab only. More will be involved with increase of the number of users. • Actions: • Build the procedures and assure that GridLab user’s know about the Center and use its services, • To be discussed by the SC in more details in 6 weeks time, • Extension of the group with a time and load (currently one person half-time at PSNC, similarly at HP, 3-4 persons at PSNC soon).
Exploitation by partners - general strategy • Every GL partner will work with the users that the SC agrees to work with closely, • In such a case particular institution will make sure that appropriate resources are assigned to work with the user in order to deploy GridLab software in the user’s environment, • The appropriate resolutions will be worked out in details and decided by the Steering Committee by the existence of the GAT release, no later than by the end of April. • At the beginning we will focus on strategic users to ensure the added value to the project. • We will work closely with the GridLab application communities first of all (Cactus and Triana) on the GAT usage.
Beyond the project • Seek for local funding by every partner • Survive the gap after the GridLab • Continue work, bring more users, get new requirements, analyze scenarios, work with the GGAA (Global Grid Application Alliance)