EU DataGrid progress and EGEE plans

1. EU DataGrid progress and EGEE plans Fabrizio Gagliardi EDG Project Leader - EGEE Proposal Coordinator Fabrizio.Gagliardi@cern.ch

2. The EU DataGrid Project • 9.8 M Euros EU funding over 3 years, twice as much from partners • 90% for middleware and applications (HEP, Earth Obs. and Bio Med.) • Three year phased developments & demos (2001-2003) • 2nd annual project review successfully passed in Feb 2003! • Total of 21 partners • Research and Academic institutes as well as industrial companies • Related projects and activities: • DataTAG (2002-2003) • CrossGrid (2002-2004) • GRIDSTART (2002-2004) • Grace (2002-2004) EDG6 Barcelona

3. Applications • Genomic Exploration • Earth Observation • High Energy Physics more and more scientists begin to use the EDG middleware and testbed,relying on Grid technology to solve huge data challenges EDG6 Barcelona

4. Project priorities refocused at EDG5 After initial middleware development and testbed deployment, effort has been refocused on quality and stability • Quality Policy Statement published http://eu-datagrid.web.cern.ch/eu-datagrid/WP12/default.htm • List of priorities defined at a project retreat http://documents.cern.ch/age?a021130 • Followed-up at the Budapest project conference http://www.tomiexpress.hu/datagrid/ • Show-stoppers found by users on the application testbed were the highest priority • Incremental improvements driven by the needs of the applications (HEPCAL, Bio and EO use cases) EDG6 Barcelona

5. Atlas & CMS Evaluations • RESULTS • Could distribute and run CMS s/w in EDG environment • Generated ~250K events for physics with ~10,000 jobs in 3 week period • OBSERVATIONS • Were able to quickly add new sites to provide extra resources • Fast turnaround in bug fixing and installing new software • Test was labour intensive (since software was developing and the overall system was fragile) • EDG 2.0 should fix the major problems providing a system suitable for full integration in distributed production • RESULTS • Atlas software was used in the EDG Grid environment • Several hundred simulation jobs of length 4-24 hours were executed , data was replicated using grid tools • Results of simulation agreed with ‘non-Grid’ runs • OBSERVATIONS • Good interaction with EDG middleware providers and with WP6/8 • With a substantial effort it was possible to perform the jobs • Showed up bugs and performance limitations (fixed or to be fixed in EDG 2.0) • We need EDG 2.0 release for use in large scale data challenges EDG6 Barcelona

6. Since Last Year: Improved software (EDG 1.4.3). Doubled sites. More waiting… Australia, Taiwan, USA (U. Wisc.), UK Sites, INFN, French sites, CrossGrid, … Significantly more CPU/Storage. Hidden Infrastructure MDS Hierarchy, Resource Brokers, User Interfaces, VO Replica Catalogs, VO Membership Servers, Certificate Authorities EDG Application Testbed for the EU review EDG6 Barcelona

7. EDG & GEANT • Connections of the different nodes of the EDG testbed are made possible by the EU-funded GEANT project • connecting more than 30 countries across Europe • speeds of up to 10 Gbit/s • high data throughput • Quality of Service • EDG and GEANT: the first major production quality tests of the network • speed • reliability • monitoring capabilities EDG6 Barcelona

8. GriPhyN PPDG iVDGL Related Grid Projects Through links with sister projects, there is the potential for a truely global scientific applications grid EDG6 Barcelona

9. Second project review feedback Feb ‘03 • Congratulations for a good review. • Good presentations and no "Murphy's law“ for the demos. An impressive job. • This success reflects the interest of all the partners involved. • Congratulates the project management for taking the risk of concentrating on production quality. • Would like to see the promise fulfilled of no relevant loss of functionality by the end of the project. EDG6 Barcelona

10. Current Plans • Second EU review successfully passed, needs to finish well • Major re-orientation of the project towards production accepted • Need to develop further plans in view of future project EGEE • Continue collaboration with US and across Atlantic testbeds (with DataTAG) • Need to accommodate other applications • Plan long term support of EDG developments • Major proposal for further EU funding (EGEE) EDG6 Barcelona

11. EGEE vision:Enabling Grids for E-science in Europe • Goal • Create a wide European Grid production quality infrastructure on top of present and future EU RN infrastructure • Build on • EU and EU member states major investments in Grid Technology • International connections (US and AP) • Several pioneering prototype results • Larg Grid development team (>60 people) • Requires major EU funding effort • Approach • Leverage current and planned national and regional Grid programmes (e.g. LCG) • Work closely with relevant industrial Grid developers, NRENs and US-AP projects Applications EGEE Geant network EDG6 Barcelona

12. Why EGEE? The Historical Analogy • Prior to the EU Geant program ,there was in Europe a multitude of exploratory projects in networking technology. Geant was truly production oriented, and brought European telecom operators actively into the picture • In a similar way, EGEE can ensure preservation of current investments in European Grid R&D, extending the present infrastructure and focussing all activities towards establishing a production quality Grid EDG6 Barcelona

13. Why EGEE? The Societal Impact Access to a production quality GRID will change the way science and much else is done in Europe An international network of scientist will be able to model a new flood of the Danube in real time, using meteorological and geological data from several centers across Europe. A team of engineering students will be able to run the latest 3D rendering programs from their laptops using the Grid. A geneticist at a conference, inspired by a talk she hears, will be able to launch a complex biomolecular simulation from her mobile phone. EDG6 Barcelona

14. Why EGEE? The Political Context • Current Grid R&D projects run out within 18 months • The EGEE partners have already made major progress in aligning national and regional Grid R&D efforts, in preparation for EGEE • Launching EGEE now will preserve the current strong momentum of the European Grid community, and the enthusiasm of the hundreds of young European researchers already involved in EU Grid projects (>150 in EDG only) EDG6 Barcelona

15. EGEE Approach • Most of partners built national and regional Grid federations to participate in EGEE • Condition to participate in EGEE is to have already an established Grid activity or be an established Grid technology centre • EGEE overall project funding needs will require staggered proposals to respond to several separate EU calls • EU FP6 Communication Network Development Call opened on December 17th, proposal submitted on May 6th EDG6 Barcelona

16. The EGEE consortium of Grid Federations Total of 70 full partners covering entire EU and beyond EDG6 Barcelona

17. Integrated Infrastructure Initiative (I3) Response to FP6 call “Communication Network Development – Grids” Three lines of EU funding (with current funding breakdown): • Networking Activities: • Management & coordination (7% of total funding) • Dissemination and outreach (7% of total funding) • Application support (10% of total funding) • Specific Service Activities: • Provision and procurement of Grid services (52% of total funding) • Joint Research Activity • Engineering development to improve the services provided by the Grid infrastructure (24% of total funding) EDG6 Barcelona

18. Networking activities General management and coordination of the project User groups, Industry Forum Dissemination, outreach, training and pilot applications EDG6 Barcelona

19. Specific service activities Integration of national and international Grid infrastructures EDG6 Barcelona

20. Joint research activity Hardening and re-engineering of Middleware EDG6 Barcelona

21. Summary of budget requirements for EGEE • EGEE will integrate existing computer resources, no hardware requests. The bulk of the budget is for human resources • Building strong teams where already existing expertise, to progress as quickly as possible. Host partners for these teams will provide matching funding and demonstrate adequate expertise for the job • The partners estimated a total funding need of about 100 M € corresponding to about 33 M € EU funding. Not included in this are US and Russian contributions (4 M €), and overall project management, corresponding to 7% of the EU funding over two years EDG6 Barcelona

22. EGEE and Industry • Industrial participation encouraged both as potential end-users and IT technology and service suppliers • Normally through national and regional Grid EGEE federations • EGEE will maintain an Industry Forum to keep selected Industrial and Commercial interested parties in close contact • Services developed in first EGEE 2 years phase (2004-5) might be tendered to Industry in second phase (2006-7) EDG6 Barcelona

23. Conclusions • The EU DataGrid project has successfully fulfilled its role of EU Grid flagship project in collaboration with several other EU and international projects • Essential to keep the momentum and the current lead in production Grids in Europe • Important to build an international cooperation between European and US/AP Grid infrastructure projects • The scientific user communities are already international (HEP is an excellent example) and so the computing resources and most of the experimental instruments • EGEE proposes the right framework and plans to accomplish the above objectives • EGEE executive summary document linked to this conference agenda (http://agenda.cern.ch/fullAgenda.php?ida=a03816) EDG6 Barcelona