Delivering Grid Interoperability

1. Delivering Grid Interoperability Dr Alistair Dunlop, Project Manager, OMII-Europe a.dunlop@ecs.soton.ac.uk

2. Outline • Macro view – What is the project trying to achieve? (4) • The project organisation and structure (3) • The role of standards in grid interoperability (3) • Micro view – Standardised Components from OMII-Europe (11) • Putting OMII-Europe components together to deliver real grid interoperability (5) • Summary (3)

3. The big picture • A definition of e-Science… • “e-Science is about global collaboration in key areas of science and the next generation infrastructure that will enable it” • ‘Next generation infrastructures’ represented by Grids • Foundation for seamless and secure collaborations over networks • National Grid infrastructures: UK NGI, D-Grid, others • New science will require the combined resources of all existing infrastructures

4. Problem is e-Infrastructure Islands • E.g., DEISA Grid (Supercomputing/HPC community) • Non WS-based UNICORE 5: Proprietary jobs (AJO/UPL) • No Virtual Organization Membership (VOMS), Full X.509 • Suitable for parallel scientific jobs (MPI, many interactions) • E.g., EGEE Grid (mainly HEP community + others) • Non WS-based gLite: Proprietary jobs (JDL) • Proxy-based X.509 security, but proprietary VOMS support • Suitable for embarrassingly parallel jobs (few interactions) • Scientists cannot use one middleware to access both (Little adoption of standards)

5. OMII-Europe Vision: Interoperability Highway End-users via clients& portals GOAL: Transparencyof Grids for end-users Emerging OpenStandards „Interoperability highway“based on open standards others Grid Middlewares others Grid Resources

6. OMII-Europe in context • A guide to flagship e-infrastructure projects... • Connectivity (Geant) • Cluster grids (EGEE) • Supercomputer Grid (DEISA) • Middleware (OMII-Europe) • “Software that enables you to use and easily access these above distributed EGEE and DEISA distributed grid infrastructures” • “Together these elements form the e-infrastructures that create “Global Virtual Research Communities”...”

7. The OMII-Europe Project • OMII-Europe stands for • Open Middleware Infrastructure Institute for Europe • It is an EU-funded project: FP6, RI • It has an initial duration of 2 years • May 2006 -> April 2008 • It has been granted a contribution of 8M € • It involves 16 partners • 8 EU • 4 USA • 4 China

8. Partners

9. Project Structure and Effort Allocation • Networking activities • Management, Outreach, Training • 8% Person Effort • Service Activities • Repository, QA, Support • 25% Person Effort • Joint Research Activities • Re-engineering, new services, integration, benchmarking • 67% Person Effort

10. OMII-Europe Standards-based approach to Grid Interoperability • Adapters-based: • The ability of Grid middleware to interact via adapters that translate the specific design aspects from one domain to another • Standard-based: • the native ability of Grid middleware to interact directly via well-defined interfaces and common open standards * definition inspired by OGF GIN CG

11. Participation in Middleware Standardisation • Most project participants involved as member/observer in many OGF WG • 11 project participant hold senior positions in • OGSA DAIS WG (Database Access and Integration Services) • OGSA RUS WG (Resource Usage Server) • OGSA BES WG (Basic Execution Service) • OGSA JSDL WG (Job Submission Description Language) • GIN CG (Grid Interoperability Now) • OGSA-AuthZ-WG (Authorization) • GLUE WG • GFSG WG (Grid File System) • RM WG (Reference Model) • OGSA Naming WG • Technical Standards Committee • GSA RG (Grid Scheduling Architecture) • GRAAP WG (Grid Research Agreement Allocation Protocol) • OGSA BYTE IO WG • OGSA D WG (Data) • OGSA DMI WG (Data Movement Interface)

12. JRA4 SA3 SA1 JRA3 SA2 The Virtuous Cycle – Technology transfer with Grid projects and standards organisations Standards Compliance Testing and QA JRA2 New Components Standards Implementation Components JRA1 IN Globus Benchmarking Repository OUT OMII-UK Components CROWN Supported Components on Eval. Infrastructure Integrated Components

13. Standards Components from OMII-Europe • Initial focus on providing common interfaces and integration of major Grid software infrastructures • Common interoperable services: • Database Access • Virtual Organisation Management • Accounting • Job Submission and Job Monitoring • Information modelling • Infrastructure integration • Initial gLite/UNICORE/Globus interoperability • Interoperable security framework • Access these infrastructure services through a portal

14. Job Submission • Unify Job Submisson and Monitoring interface • Adoption of emerging OGSA-BES and JSDL standards • Alpha BES and JSDL implementations for • UNICORE 6, gLite 3.1, Globus 4, OMII-UK, CROWNgrid • Interoperability demonstrated through use of a BES compliant meta-scheduler

15. VO Management • To provide a common Virtual Organisation (VO) management solution across middleware distributions • Extend VOMS Interface to support emerging AuthZ standard • compliance with SAML Authorisation model • Extension, not a replacement interface • Public release of VOMS integrated with UNICORE

16. Accounting • Unify accounting information across middleware distributions • Provide standardized interfaces for accessing that information • Information standard: • Usage Record Format (URF) • Service interface standard: • Resource Usage Service (OGSA-RUS) • Alpha versions RUS • gLite (DGAS) • Globus (SGAS) • UNICORE

17. Data Access • Port OGSA-DAI 3.0 from Globus to other middleware distributions available throughout Europe and China • UNICORE • gLite • CROWN

18. Information Modelling • There is a lack of a common description of Grid resources suitable for discovery, monitoring and scheduling • Many descriptions exist • e.g.: GLUE Schema, NorduGrid Schema • Working on the definition of next-generation GLUE Information Model in the context of OGF GLUE WG and its implementation

19. Portal • Deliver tools for developing Grid portals and support for key Web and Grid standards and technologies • Objectives: • Develop gateway to OMII Evaluation Infrastructure • Develop tools for portal and grid software training • Explore new approaches for grid portal development

20. Repository of Open-Source Software • Make available software reengineered within OMII-Europe and contributed by third parties • Single services/tools & complete distributions • Provide an interface to select software from the repository based on user requirements • By capability/standards/provider/… • Support the upload, download and installation of the software • Document platform portability & pre-requisites • Verify the software through compliance & metrics tests

21. Behind the Repository • Leverage existing infrastructure & projects • ETICS • Capture build & test configuration data for repeatability • NMI Build & Test Framework • Manage cross-platform environment for build & tests • Condor • Underlying execution infrastructure • Provides reports to be displayed within the portal • Builds: Pre-requisites & platforms • Testing: Conformance & Interoperability

22. Tests For Standards Conformance • Job Submission and Job Monitoring • Job Submission Description Language (JSDL) • Basic Execution Service (BES) • Accounting • Usage Record (UR) • Resource Usage Service (RUS) • Database Access • WS-DAI, WS-DAIX, WS-DAIR (OGSA-DAI) • Virtual Organisation Management • Move towards SAML2

23. Standards status: March 2008 • Accounting • SGAS/DGAS/UNICORE-RUS • OGSA-Resource Usage Service (RUS – OGF draft specification) - implies: • Usage Record Format (UR) (OGF recommendation status awaiting implementation) • Job Submission and Job monitoring • CREAM-BES, GLOBUS-BES, UNICORE-BES • OGSA-Basic Execution Service (BES) version 1.0 (OGF final specification) – implies: • Job Submission Description Language (JSDL) version 1.0 (OGF final specification) • Database Access • OGSA-DAI • WS-DAIX and WS-DAIR are being implemented in OGSA-DAI (both currently OGF candidate standards at 1.0 awaiting implementation) • Virtual Organisation Membership Service • VOMS implements the following: • Security Assertion Markup Language (SAML) Version 2.0 OASIS (March ’05) • Extensible Access Control Markup Language (XACML) version 2.0 OASIS (February ’05) • Information modelling • GLUE 2.0 • standard is currently draft, expected to go out for public comments by the end of April 2008, final version by September 2008

24. Using OMII-Europe components to deliver grid interoperability – WISDOM scenario • Wide In Silicio Docking On Malaria (WISDOM) projects • Developing new drugs for neglected and emerging diseases with a particular focus on malaria • Accelerated Research & Development for emerging and neglected diseases • Massive computation reduces R&D costs

25. Dr. Nicolas Jacq [16]

26. WISDOM in the context of DEISA & EGEE • WISDOM uses EGEE for large scale in-silicio docking • Comp. method for prediction of whether one molecule will bind to another • Using AutoDock and FlexX software provided via gLite in EGEE (FlexX – licensed software on EGEE with 6k licenses) • Output is a list of best chemical compounds (potential drugs – NOT final solution) • DEISA to find best compound from potential drugs list • Fast molecular dynamics computations • Using highly scalable AMBER (Assisted Model Building with Energy Refinement) in DEISA • Goal: Accelerate drug discovery using EGEE + DEISA

27. Interoperability Scenario: WISDOM SAML- based Attribute Authority (AA) VOMS getscentralrole & middlewareindependent

28. Work in progress Further Interoperability Scenarios: EU - IndiaGrid

29. What can you do Now… and Later… • Now • Most products at Beta stage becoming publicly available • They provide basic interoperability of multiple grid middleware systems focusing on job execution • Available to early adopters working with OMII-Europe partners • Spring 2008 (end of current project) • Further security integration work between different middleware platforms (SAML-VOMS, TLS (Transport level security)) • Completed QA’d services and demonstrated end-to-end solutions • Availability of GLUE 2 information model service implementations

30. Caveat... • So you can use EGEE and DEISA seamlessly? Not quite... • OMII-Europe has provided much of the required technical interoperability, but • Needs EGEE and DEISA sites to deploy either the latest versions of the grid middleware, or to deploy the OMII-Europe services available from the repository • AND... • Currently need to apply for resources independently. Intention is that EGI will do this in the future

31. Summary (1/2) • OMII-Europe is a 24 Month EU funded project with 16 partners to establish grid infrastructure interoperability through implementing a set of agreed open standards on all middleware platforms • OMII-Europe is implementing a number of components that will allow identically specified jobs to be run, managed and migrated to different middleware platforms • Initial versions of BES, VOMS/SAML and security service have already enabled UNICORE and gLite managed resources to be used by the same job • Users can try interoperability on the OMII-Europe evaluation infrastructure, or obtain services for installation on their own resources from the OMII-Europe repository

32. Summary (2/2) • We anticipate OMII-Europe services to be integrated into standard middleware distributions as well as deployed on large scale e-infrastructures such as EGEE and DEISA • OMII-Europe requested continuing funding in the September EU call to support the existing services and provide further services in the areas of data and Grid management • We are interested in working with projects that have real grid interoperability issues to ensure our solutions match your needs.

33. Further Information General information: http://omii-europe.org Repository: http://repository.omii-europe.org Support: http://support.omii-europe.org • With special thanks to the following project members for diagrams and slides: • Morris Riedel • Stephen Brewer