1 / 60

Experiences integrating new applications in EGEE

Experiences integrating new applications in EGEE. Roberto Barbera University of Catania and INFN First EGEE User Forum CERN, 01-03.03.2006. Outline. The mission The present results The future. Goals of EGEE Applications Sector.

renata
Download Presentation

Experiences integrating new applications in EGEE

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Experiences integrating new applications in EGEE Roberto Barbera University of Catania and INFN First EGEE User Forum CERN, 01-03.03.2006

  2. Outline • The mission • The present results • The future First EGEE User Forum, CERN, 01-03.03.2006

  3. Goals of EGEE Applications Sector • Drive the evolution of the grid technology through specific, challenging applications. • pilot applications (HEP and Biomed) committed to use the large distributed infrastructure EGEE to achieve their scientific goals. • Demonstrate that EGEE provides a viable computing infrastructure for research to several scientific communities. • EGEE hosts a number of scientifically diverse applications with the help of teams of engineers funded by EGEE and the scientific communities concerned. First EGEE User Forum, CERN, 01-03.03.2006

  4. Pilot Applications: HEP First EGEE User Forum, CERN, 01-03.03.2006

  5. How to prepare for LHC: LCG Service Challenges • LHC starts in 2007 • Ramp-up with series of service challenges to ensure key services & infrastructure in place for the experiments computing systems • Extremely aggressive timescale • Emphasis on providing a service • Data movement • Data handling First EGEE User Forum, CERN, 01-03.03.2006

  6. HEP success stories LHCb • Fundamental activity in preparation of LHC start up • Physics • Computing systems • Examples: • LHCb: ~700 CPU/years in 2005 on the EGEE infrastructure • ATLAS: over 10,000 jobs per day • Comprehensive analysis: see S.Campana et al., “Analysis of the ATLAS Rome Production experience on the EGEE Computing Grid“, e-Science 2005, Melbourne, Australia • A lot of activity in all involved applications (including as usual a lot of activity within non-LHC experiments like BaBar, CDF and D0) ATLAS First EGEE User Forum, CERN, 01-03.03.2006

  7. 10,000 jobs/day ! From Accounting data: • ~3 million jobs in 2005 so far • Sustained daily rates (per month Jan – Nov 2005): [2185, 2796, 7617, 10312, 11151, 9247, 9218, 11445, 10079, 11124, 9491] • ~8.2 M kSI2K.cpu.hours  >1000 cpu years • Real usage is higher as accounting data was not published from all sites until recently First EGEE User Forum, CERN, 01-03.03.2006

  8. Pilot Applications: Biomed First EGEE User Forum, CERN, 01-03.03.2006

  9. Medical image processing • GATE: Radiotherapy planning • CNRS • Monte Carlo simulation • Parallel execution on different seeds • Pharmacokinetics: contrast agent diffusion study • UPV • Medical images registration • Distribution of registration pairs First EGEE User Forum, CERN, 01-03.03.2006

  10. Bioinformatics • GPS@: bioinformatics portal • http://gpsa.ibcp.fr/ web portal • Existing (but overloaded NPSA portal) • Tens of bioinformatics legacy code • Thousands of potential users • Large input databases • Electron-microscopic image reconstruction • Image filtering and noise reduction • 3D structure analysis First EGEE User Forum, CERN, 01-03.03.2006

  11. First biomedical data challenge: World-wide In Silico Docking On Malaria (WISDOM) • Significant biological parameters • two different molecular docking applications (Autodock and FlexX) • about one million virtual ligands selected • target proteins from the parasite responsible for malaria • Significant numbers • Total of about 46 million ligands docked in 6 weeks • 1TB of data produced • Up to 1000 computers in 15 countries used simultaneously for a total of about 80 CPU years • Significant results • Best hits to be re-ranked using Molecular Dynamics New data challenge in the fall of 2006 New malaria targets Focus on other neglected diseases Enlarged collaboration (possibly including related projects) First EGEE User Forum, CERN, 01-03.03.2006

  12. Generic Applications First EGEE User Forum, CERN, 01-03.03.2006

  13. The EGEE Virtuous Cycle NA2, NA3, N4 JRA1 NA3, NA4, SA1 SA1 First EGEE User Forum, CERN, 01-03.03.2006

  14. The birth of a new VO in EGEE New community Deployment & configuration Gen. Apps. Quest. & Prop. EGAAP Resource allocation proposal Recommended VO candidate Ask for change MoU NA3/NA4/SA1 (OAG) VO requirements First EGEE User Forum, CERN, 01-03.03.2006

  15. The MoU’s First EGEE User Forum, CERN, 01-03.03.2006

  16. The status of Generic Applications deployment • Applications accepted before Pisa conference • Earth Science Research (Earth Observation, Hydrology, Climate) • Geophysics (Industry) • Computational Chemistry • Astro(particle)-physics (MAGIC and Planck collaborations) • Finance (EGRID) • Applications approved at last EGEE conference in Pisa (October 2005) • Fusion (ITER) • Archaeology • EC Projects (EELA, EUMEDGRID, EUCHINAGRID, BIOINFOGRID) First EGEE User Forum, CERN, 01-03.03.2006

  17. SimGate EGEE Grid MPI libraries Apache server Computational Chemistry GEMS Programs Client side HTTP Server side First EGEE User Forum, CERN, 01-03.03.2006

  18. Computational Chemistry Venus: QCT for many body systems First EGEE User Forum, CERN, 01-03.03.2006

  19. Earth Science Research Earthquakes’ epicenter determination Ozone maps Climate First EGEE User Forum, CERN, 01-03.03.2006

  20. FUSION: reactor confinement optimization First EGEE User Forum, CERN, 01-03.03.2006

  21. FUSION: reactor confinement optimization First EGEE User Forum, CERN, 01-03.03.2006

  22. MAGIC • Build a Grid system with • FZK (Germany) • CNAF(Italy) • PIC (Spain) • MAGIC applied as a generic application for EGEE • MAGIC got accepted with the air shower Monte Carlo simulation based on CORSIKA First EGEE User Forum, CERN, 01-03.03.2006

  23. MAGIC • Stability is improving • Number of failed jobs is decreasing: • March 2005 - 10% • June 2005 - 3.9% • September 2005 - 3.4% • Around 18000 jobs submitted in three data challenges • A system with >90% quality level is there! First EGEE User Forum, CERN, 01-03.03.2006

  24. Planck on Grid PLANCK • LevelS basic gridification: • UI env; • WN env; • Data handling. • Basic tests: • INFN production grid; • LFI (22 channels); Development of G-DSE in collaboration with INFN First EGEE User Forum, CERN, 01-03.03.2006

  25. EGEE application sector: other numbers • Total effort (funded + unfunded) : 100 FTE’s for 2 years • Total EC funding: 3600k€, corresponding to about 30 FTE’s • 20 partners • More than 20 applications deployed on the production infrastructure • The number of users in application related VO’s has doubled from ~500 in December 2004 to ~1000 in September 2005 First EGEE User Forum, CERN, 01-03.03.2006

  26. EU Projects Applications First EGEE User Forum, CERN, 01-03.03.2006

  27. Related EU grid projects First EGEE User Forum, CERN, 01-03.03.2006

  28. EELA • Biomed • GATE • WISDOM • CECALC web portal (ULA - http://www.cecalc.ula.ve/) • Protein Dynamics (UFRJ) • HEP • LHC Exps. • Additional (EELA specific) • Climate in the Grid Environment • Education in the Grid Environment (“gridification” started in Trieste, see further on) • New (not foreseen in the T.A.) • Questionnaire finalized First EGEE User Forum, CERN, 01-03.03.2006

  29. The ARGO–YBJ Experiment EUCHINAGRID Unique High Altitude Cosmic Ray Laboratory (4300 m) Tibet, 90 km North to Lhasa. Chinese-Italian collaboration. The Experiment data rate to be transferred is 250 TB/Year requiring a steady transfer rate of the order of 100 Mbps to Beijing and from there to Italy. First EGEE User Forum, CERN, 01-03.03.2006

  30. EUMEDGRID • Aim: estimate sustainable extraction scheme - improve management • CODESA-3D:Density-dependent 3D coupled groundwater flow and transport simulations • Data requirement • Geology • Topography • Meteorology • Water extraction by the farmer • Aquifer properties • Soil maps • Land use One simulated map of water levels First EGEE User Forum, CERN, 01-03.03.2006

  31. GILDA Applications First EGEE User Forum, CERN, 01-03.03.2006

  32. The GILDA t-Infrastructure(https://gilda.ct.infn.it) 19 sites in 3 continents > 3000 certificates issued, >15% renewed at least once > 100 tutorials and demos perfor-med in 23 months > 1,000,000 hits (> 50,000 unique visits) on (of) the web site from 10’s of different countries > 0.6 TB of training material downloaded from the web site First EGEE User Forum, CERN, 01-03.03.2006

  33. Number of trained users vs. time First EGEE User Forum, CERN, 01-03.03.2006

  34. Early Diagnosis of Alzheimer Disease First EGEE User Forum, CERN, 01-03.03.2006

  35. Early Diagnosis of Alzheimer Disease First EGEE User Forum, CERN, 01-03.03.2006

  36. Laboratory Measurements DB GeoArchaeology DB Archaeo Climatology DB Archaeo Zoology/Botanic DB Archaeological bibliography DB Simulation/VR DB Archaeological Objects DB Archaeology Media Archaeological GIS Tourism Cultural Heritage Images DB Land Management TextFile DB ArchaeoNet ArchaeoGrid USERS First EGEE User Forum, CERN, 01-03.03.2006

  37. Archaeological Digital Archives/GIS Earth Science GRID Web Service Interface Web Service Interface Digital Library DILIGENT UI I Visualization Narration gLite Distributed Resources ArchaeoGrid First EGEE User Forum, CERN, 01-03.03.2006

  38. Music for Science: Data Sonification on Grid • Data sonification is the representation of data by sound signals. It can be considered as the acoustic counterpart of data graphic visualization, a mathema-tical mapping of information from data sets to sounds. • Data sonification is currently used in several fields, for different purposes: science and engineering, education and training, mainly as data analysis and interpretation tool. • Data sonification has been applied to geophysical data collected by a digital seismograph placed on the Etna volcano in Italy (courtesy of INGV). • The hope is to learn more about eruption dynamics from patterns in sonograms. First EGEE User Forum, CERN, 01-03.03.2006

  39. Seismograms Sonification(15’ of data = 20h of CPU!) Example of audible seismogram (just at the start of the eruption) Corresponding sonogram Spectral lines ↔ Regular patterns First EGEE User Forum, CERN, 01-03.03.2006

  40. Seismograms Melodisation

  41. Seismograms Melodisation

  42. Seismograms Melodisation

  43. Seismograms Melodisation … have you ever heard a volcano playing on a piano ? Melodies can then be analized with mathematical tools to look for patterns and self-similarities

  44. Grid for Art: 3D Data Visualization • Letters in a text are associated to graphical objects such as spheres. • Radii, indexes of reflection and refraction, light diffusion properties, positions in the space are associated to the position of the letters in the english alphabet. NA4 Generic Applications First EGEE User Forum, CERN, 01-03.03.2006

  45. A new kind of cryptography is born! First EGEE User Forum, CERN, 01-03.03.2006

  46. CE WN WN WN Metadata Catalog FiReMan Catalog gMOD(a grid for entertainment) VOMS Storage Elements GENIUS Portal AMGA get Role User Workload Management System First EGEE User Forum, CERN, 01-03.03.2006

  47. gMOD(a grid for entertainment) First EGEE User Forum, CERN, 01-03.03.2006

  48. LFC (or Fireman) Catalog gLibrary(multimedia contents management on the grid) VOMS VOMS Proxy with Group & Role Information VOMS Proxy w/Role & Group Authenticate with X509 Certificate PostGreSQL (gLibraryManager gLibrarySubmitter VO user) AMGA Server UI VOMS Proxy VOMS Proxy SE SE SE First EGEE User Forum, CERN, 01-03.03.2006

  49. gLibrary(multimedia contents management on the grid) To be soon integrated in GENIUS and ap- plied to the NA3 collection of presentations First EGEE User Forum, CERN, 01-03.03.2006

  50. Lessons learned • The first three magic words: training, training, training! • Identify the VO experts from the beginning so that they can set a bridge between EGEE and their community. • Show to the users what is possible but also what is not so that they can understand the current limits of the middleware. • For things not working look around very quickly for replacements (remember the metadata crisis in June ’05 and the adoption of AMGA). • Provide users not only with the information on the single services but with small use cases that can be expanded (e.g., gMOD, gLibrary, etc.) • Try to find occasions to organize dedicated meetings/ workshops to port the new applications on grid • The last three magic words: support, support, support! First EGEE User Forum, CERN, 01-03.03.2006

More Related