1 / 30

EGEE is a project funded by the European Union under contract IST-2003-508833

EGEE First Conference, Cork, April 19, 2004. JRA1 Overview: Introduction Frédéric Hemmer EGEE Middleware Manager. EGEE is a project funded by the European Union under contract IST-2003-508833. Contents. EGEE Middleware activities Objectives, Partners, Milestones, WBS EGEE Middleware and LCG

mcorey
Download Presentation

EGEE is a project funded by the European Union under contract IST-2003-508833

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EGEE First Conference, Cork, April 19, 2004 JRA1 Overview:IntroductionFrédéric HemmerEGEE Middleware Manager EGEE is a project funded by the European Union under contract IST-2003-508833

  2. Contents • EGEE Middleware activities • Objectives, Partners, Milestones, WBS • EGEE Middleware and LCG • EGEE Middleware and ARDA • Progress so Far • Summary EGEE 1st Conference, Cork, April 19, 2004 - 2

  3. Objectives of the EGEE Middleware activity • Provide robust, supportable middleware components • Select, re-engineer, integrate identified Grid Services • Evolve towards Services Oriented Architecture • Adopt emerging OGSI standards* • Multiple platforms • Selection of Middleware based on requirements of • The Applications (Bio & HEP) • In particular requirements are expected from LCG’s ARDA & HepCALII • The Operations • E.g. deployment, updates, packaging, etc.. • Support and evolve of the middleware components • Evolution towards OGSI* • Define a re-engineering process • Address multiplatform, multiple implementations and interoperability issues • Define defect handling processes and responsibilities *: Now questioned given the WSRF announcement on January 20, 2004. The strategy is to use plain Web Services and review the situation towards the end of the year (GT4). EGEE 1st Conference, Cork, April 19, 2004 - 3

  4. EGEE Middleware Partners Issue: US involvement to be further detailed (and needed for reporting) EGEE 1st Conference, Cork, April 19, 2004 - 4

  5. EGEE Middleware Activity • Hardening and re-engineering of existing middleware functionality, leveraging the experience of partners • Activity concentrated in few major centers and organized in “Software clusters” • Key services: • Data Management (CERN) • Information Collection (UK) • Resource Brokering, Accounting (Italy-Czech Republic) • Quality Assurance (France) • Grid Security (Northern Europe) • Middleware Integration (CERN) • Middleware Testing (CERN) EGEE 1st Conference, Cork, April 19, 2004 - 5

  6. EGEE Milestones and Deliverables for the first year EGEE 1st Conference, Cork, April 19, 2004 - 6

  7. EGEE Middleware Work Breakdown Structure • Main components: • Middleware Re-engineering • Workload Management, CE • Data Management • Information Services • Authentication/Authorization • Accounting • Integration • Testing EGEE 1st Conference, Cork, April 19, 2004 - 7

  8. EGEE Middleware – Other Components • A few more components are being worked at, such as: • Access Services • Authentication/Authorization • Involvement of the Security cluster • Common Services • Messaging • Error Handling • Logging • WS Containers • Some of these components do not have a clear mapping in the current EGEE middleware software cluster organization EGEE 1st Conference, Cork, April 19, 2004 - 8

  9. EGEE Middleware and LCG EGEE 1st Conference, Cork, April 19, 2004 - 9

  10. LCG New Middleware Development • Exploiting and integrating experience, expertise and technology from DataGrid (EU), the Virtual Data Toolkit (US), AliEn (ALICE), NorduGrid • Joint EGEE-VDT design team • Focus on HEP requirements + bio-medical • Strongly coupled to ARDA - a new LHC distributed analysis project • We need to see an early prototype soon, involving HEP applications and users and a “usable system” within a year - stability, performance as important as functionality • By this time next year we will have to start making decisions about the middleware to be used in 2007 EGEE 1st Conference, Cork, April 19, 2004 - 10 Les Robertson – “LCG Middleware”

  11. 2004 2005 LCG and Next Generation Middleware • LCG-2 will be the main service for the 2004 data challenges • This will provide essential experience on operating and managing a global grid service – and will be supported and developed • Target is to establish a base (fallback) solution for early LHC years • LCG-2 will be maintained until the new generation has proven itself LCG-2 Next Generation prototype product development mainline service EGEE 1st Conference, Cork, April 19, 2004 - 11 Les Robertson – “LCG Middleware”

  12. Aiming at the right Goal • Our goal is straightforward - to set up a computing environment for LHC • The grid is only a means to that end • We have to set our priorities by the practical needs of the experiments • Focus on data challenges • Evolve in stages a workable computing model • With the experience of this year’s data challenges we must set realistic goals for 2007 That is what the middleware must address EGEE 1st Conference, Cork, April 19, 2004 - 13 Les Robertson – “LCG Middleware”

  13. EGEE Middleware and ARDA EGEE 1st Conference, Cork, April 19, 2004 - 14

  14. Middleware & ARDA • ARDA RTAG has influenced considerably the EGEE Middleware activity • Reference included in the Technical Annex • Group of Middleware providers met as of December 2003 • Monthly meetings (design & implementation) • Goal to define and provide Middleware components as described in the ARDA RTAG • Participants from AliEn, EDG, VDT • ARDA Project has been established • It is a distinct project, focused on the usage of the Middleware within the experiments • Providing resources to HEP to help delivering end to end analysis prototypes • Providing an organization to discuss and agree on Middleware components EGEE 1st Conference, Cork, April 19, 2004 - 15

  15. ARDA Working Group Recommendations • New service decomposition • Strong influence of Alien system • Role of experience, existing technology… • Web service framework • Interfacing to existing middleware to enable their use in the experiment frameworks • Early deployment of (a series of) prototypes to ensure functionality and coherence EGEE Middleware ARDA project EGEE 1st Conference, Cork, April 19, 2004 - 16 Massimo Lamanna – “The ARDA Project”

  16. ARDA End-to-end prototypes • Provide a fast feedback to the EGEE MW development team • Avoid uncoordinated evolution of the middleware • Coherence between users expectations and final product • Guarantee the experiments are ready to benefit from the new Middleware as soon it becomes available • Expose the experiments (and the community in charge of the deployment) to the current evolution of the whole system, to be prepared to use it in the best and quickest way • Move forward towards new-generation real systems • Prototypes should be exercised with realistic workload and conditions (experiments absolutely required for that!) • No academic exercises or synthetic demonstrations • A lot of work (and useful software) is involved in current experiments data challenges: this will be used as a starting point • Adapt/complete/refactorise the existing: we do not need another system! EGEE 1st Conference, Cork, April 19, 2004 - 17 Massimo Lamanna – “The ARDA Project”

  17. ARDA End to End PrototypesImplementation • Every experiment has already at least one system • Analysis/Production typically distinct entities • Using a variety of back-ends (Batch systems, different grid systems) • ARDA will put its effort on the experiment (sub)system the experiment chooses • EGEE Middleware as foundation layer • Multi-grid interfaces outside our scope • Experiments do know how to deal with this • By default, we expect 4 systems • There is nothing like an ARDA prototype • Adapt/complete/refactorise the existing (sub)system! • Collaborative effort (not a parallel development) • Commonality is not ruled out, but it should emerge and become attractive for the experiments. Anyway not imposed “from outside” EGEE 1st Conference, Cork, April 19, 2004 - 18 Massimo Lamanna – “The ARDA Project”

  18. ARDA End to End PrototypesImplementation (II) • The initial prototype will have a reduced scope • The component should be select to deliver the first prototype • Experiments components not in use for the first prototype are not ruled out (and used/selected ones might be replaced later on) • Not all use cases/operation modes will be supported • Attract and involve users • Many users are absolutely required EGEE 1st Conference, Cork, April 19, 2004 - 19 Massimo Lamanna – “The ARDA Project”

  19. ALICE Distr. analysis ATLAS Distr. analysis CMS Distr. analysis LHCb Distr. analysis …….. …….. GAE SEAL PROOF POOL The ARDA Project ARDA Project Collaboration Coordination Integration Specifications Priorities Planning GAG Specifications Experience Requirements Guidelines Resource Providers – Regional CentresGDB Security Group Generic Middleware Project EGEE/VDT/.. EGEE 1st Conference, Cork, April 19, 2004 - 20 Massimo Lamanna – “The ARDA Project”

  20. ARDA Project plan • One prototype per experiment • Formally, the project starts April the 1st • Preparation phase already started • Same pattern being proposed to each experiments • Interfacing to EGEE MW • Direct contribution into experiment-specific “Upper Middleware” • Focused dedicated effort to be added to the experiment system • Not a demonstration system to be added to the experiments plans • Mainstream activity EGEE 1st Conference, Cork, April 19, 2004 - 21 Massimo Lamanna – “The ARDA Project”

  21. EGEE, LCG & ARDAHigh-Level Strategy for Middleware • LCG-2 middleware package strongly supported and evolved • Demonstrating a base solution for LHC start-up • Supported until overtaken by EGEE Middleware • EGEE Middleware – • Re-engineered generic middleware package • Incorporating experience from AliEn, EDG, …., VDT • Architected for scale and performance requirements of LCG • “batch” and “analysis” • Fast prototyping approach – with clear end-to-end goals • Short update cycles to give LHC experiments the chance to influence and give feedback • Important Note: we do not have an “ARDA” Mechanism for BioMedical & Other applications • We do not need other projects, but the need same spirit EGEE 1st Conference, Cork, April 19, 2004 - 22

  22. JRA1 Progress so far EGEE 1st Conference, Cork, April 19, 2004 - 23

  23. Work done so far • Middleware Engineering • Gathered a set of Middleware providers • AliEn, EDG, VDT, … • Meetings so far • December 3-4, 2003 • Workload Management System, CE • ARDA Workshop January 21-22, 2004 • Setting up ARDA project • February 24-27, 2004 • File catalogs, replica management, SE • March 24-April 1, 2004 • Information system • Security • Interfacing with Ganga • A working document • Overall design & API’s • https://edms.cern.ch/document/458972 • Real prototype being implemented • Aim at end of April 2004 for a first (incomplete) version • Solid steps towards Deliverables in June and August: • DJRA1.1 (Document) Architecture and Planning (Release 1) • DJRA1.2 (Document) Design of grid services (Release 1) EGEE 1st Conference, Cork, April 19, 2004 - 24

  24. Work Done so far (II) • Middleware Integration and Testing • Continued support for software currently deployed on LCG service organized and the groups responsible for each module identified. Document available. • LCG Grid Deployment Group (CERN) ensures 1st level support • VDT ensures 2nd level support for Globus/Condor • Named individuals ensure 2nd level support for Workload and Data Management • EMT has been established • Software Clusters (CERN, Italy, UK) have clarified plans for their development, integration and testing testbeds in terms of scale, sites and support. • Infrastructure for development, integration, and testing almost finalized. • Most clusters will take-over existing EDG infrastructure for development. • CERN, NIKHEF and RAL will be the main testing sites. • Tools and mechanisms will be used for software packaging and distribution are being clarified with SA1; a document is in preparation • Software Configuration Management document has been produced and being discussed with the partners. • Initial platforms have been proposed based on discussions with SA1 • Solid steps towards Milestones in in June and August: • MJRA1.1 Tools for middleware engineering and integration deployed • MJRA1.2 Software cluster development and testing infrastructure available • MJRA1.3 Integration and testing infrastructure in place including test plans (Release 1) EGEE 1st Conference, Cork, April 19, 2004 - 25

  25. Effort Summary from Execution Plan* *Note: numbers need to be finalized EGEE 1st Conference, Cork, April 19, 2004 - 30

  26. Resources indicators EGEE 1st Conference, Cork, April 19, 2004 - 31

  27. Next steps • Deliver first version of the prototype • Find a new middleware name • In order not to confuse Generic Middleware and ARDA project • Suggestions welcomed • Consolidate (working) Interface document • Architecture & Design • API’s • Services Interfaces • Exercise interface with the ARDA project • Interface experiments frameworks • Agree on interfaces/API’s • And iterate through prototype versions • Get documented requirements from Deployment • Implement in prototype • Use the prototype to validate Integration & Testing plans • Nightly builds • Savannah Portal/CVS repositories • Software Configuration Management plans • SPI tools • Complete Execution Plan EGEE 1st Conference, Cork, April 19, 2004 - 32

  28. Issues related to other activities • NA4 • Lack of a formal structure to get requirements from non-HEP applications • Such as it exists with ARDA for HEP • Security JRA3 • Development must be part of JRA1 structure • Security is not an orthogonal activity • Security Group still needs to be formed • Same for JRA4 • Interaction with SA1 • Support • Requirements and acceptance criteria gathering • Definition of responsibilities • Testing • Support • Packaging/Configuration/Distribution Mechanisms • JRA2 • Documents (standards, templates) need to be available early • And relevant to the complexity of the activity • Quality group still needs to be formed • Project • “Architecture Group/Oversight Board” now being discussed } All being discussed EGEE 1st Conference, Cork, April 19, 2004 - 33

  29. Summary • EGEE Middleware Engineering effort is being used to provide next generation Middleware for LCG and others • For “batch” and “analysis” • According to the ARDA RTAG • Leveraging experience from AliEn, EDG & VDT • Complying with the requested • Quality from both EGEE & LCG point of view • Deployment requirements gathered through LCG-{1,2} experiences • Defining API’s and WS Interfaces • Allowing for alternative implementations • Ensuring LHC Experiments (and other sciences) requirements are met • Through rapid prototyping and short release cycles • Through Analysis prototypes built from the ARDA project • Providing the expected robustness and quality • Through the Integration & Testing processes EGEE 1st Conference, Cork, April 19, 2004 - 34

  30. Plan for the rest of the week • Today • 14:50-15:30 Middleware Implementation • 16:00-17:00 Configuration Management Plan • 17:00-17:30 Testing • Tomorrow • 9:00-9:20 UK Cluster • 9:20-9:40 IT Cluster • 9:40-10:00 CERN Cluster • 11:00-11:20 Nordic Cluster • 11:20-12:30 Security Issues in Middleware • 14:00-14:45 JRA1/SA1 Issues • 14:45-16:00 JRA1/JRA3/SA1 Issues • 16:00-17:30 Wrap-up • Goals • Status of the activities (what has changed in Cork) • Steps until November 2004 incl. • Deliverable Planning • Relations with other activities • Amendment to TA EGEE 1st Conference, Cork, April 19, 2004 - 35

More Related