1 / 27

Presenter: Christian Neissner (PIC)

Presenter: Christian Neissner (PIC) R. Firpo (PIC), J. Rico (IFAE), I. Reichardt (IFAE), M. Delfino (PIC), A. Moralejo (IFAE). The MAGIC Data Center storage and computing infrastructures in Grid. Contents. MAGIC The MAGIC Data Center Storage and Computing in Grid Conclusions. MAGIC.

ettathomas
Download Presentation

Presenter: Christian Neissner (PIC)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Presenter: Christian Neissner (PIC) R. Firpo (PIC), J. Rico (IFAE), I. Reichardt (IFAE), M. Delfino (PIC), A. Moralejo (IFAE) The MAGIC Data Center storage and computing infrastructures in Grid

  2. Contents • MAGIC • The MAGIC Data Center • Storage and Computing in Grid • Conclusions

  3. MAGIC

  4. What is MAGIC? MAGICis a Cherenkov telescope system for g-ray astronomy in the very high energy range (VHE, E > 25 GeV) Scientific targets • Cosmic Accelerators • Active Galactic Nuclei, Pulsare Wind Nebula, SN Remnants, Gamma Ray Bursts, … • Fundamental Questions • Dark Matter, Cosmic Rays, Quantum Gravity, Cosmology…

  5. The MAGIC Collaboration • The MAGIC Collaboration: • 21 institutes (mostly in Europe) • ~ 200 members • Telescope site in Canary Islands • Observatorio Roque de los Muchachos (ORM) • MAGIC-I in operation since 2004 • MAGIC-II in operation since 2009 • Future detector enhancements • Equip MAGIC-I with same camera and readout as MAGIC-II

  6. Scientific Highlights • Discovery of 14 new VHE g-ray sources • 8 extragalactic + 4 galactic • New populations unveiled • Radio-quasar & Micro-quasar • Detection of distant VHE g-rays • z = 0.54, farthest up to now • Detection of pulsated VHE g-rays • Originated in the Crab pulsar • Test on Lorentz Invariance (QG effects) • Using big emission flares • >40 papers in high impact journals • including 4 in Science

  7. MAGIC Data MAGIC records Cherenkov light flashes from g-ray induced atmospheric particle showers • Major issue: Background rejection • Separate g-rays from hadrons • Based on image parameters • Monte Carlo simulations required • No VHE “test beam” available

  8. The MAGIC Data Center

  9. MAGIC Data Center @ PIC • MAGIC produces ~300 TB of raw data per year • And up to 400 TB in the final configuration • The MAGIC Data Center at PIC provides: • Data transfer from ORM and storage • Official data reprocessing • Computing resources and tools • User access and support • PIC data center operating since 2006 • 2009: Upgraded for the 2nd telescope • Challenge: scalable infrastructure

  10. Data volume • Increase in data volume after the upgrade • In ~3 years data volume will be increased 4-fold

  11. MAGIC in Grid

  12. Grid as a solution • Challenges: • Large increase in data volume • Need scalable Storage solution • Maintenance of old infrastructure prevented Innovation • Need to improve data access • Robust infrastructure for many concurrent accesses • Data catalog with metadata • Open Computing: • Accessible to all collaborators • Simple and easy to use for standard analysis

  13. Opportunities for MAGIC in GRID • Why GRID? • Data reduction and analysis require lots of computing resources • Must distribute data to all collaborators across Europe • User access to shared resources and standardized analysis tools • Better and easier data management • Increased technical support, benefit from community • The MAGIC Data Center @ PIC • Experience and knowledge on using Grid from LHC projects • Manpower partially funded by EGEE • Storage based on PIC SEs • Computing @PIC and other MAGIC sites • Other sites currently devoted to MC production

  14. The MAGIC VO • MAGIC VO exists since 2004 • Initiative by H. Kornmayer et al. • Hiatus 2005-2007 • No manpower • 2007-08: New crew taking over grid operations • UCM (Madrid) and Dortmund, in collaboration with INSA (MC) • IFAE and PIC (Data Center) • 2009-10: Wide adoption • Now Grid is widely used in MAGIC

  15. MAGIC in Grid (2004) • Initial project for MAGIC (H. Kornmayer) • Involve 3 national centers • CNAF (Bologna) • PIC (Barcelona) • GridKA (Karlsruhe) • Interconnect MAGIC resources • 2 subsystems: • Monte Carlo • Storage & Analysis • Evolved separately

  16. MAGIC DC migration to Grid • Migration of services while in production (in progress) • Migration of Storage: • Move existing data to Storage Elements in Grid • Use FTS for the data transfer from the observatory • Adapt administration tools to new infrastructure • Create user-friendly interfaces to access data • Migration of Computing: • Port existing analysis tools to Grid • Develop library of standard tools for the user community • User-friendly interface to monitor and submit jobs

  17. MAGIC users and Grid • MAGIC users were reluctant to use Grid • Grid has a steep learning curve • Used to the old storage and computing • Lack of a ‘killer application’ • Conquer your user community • Good documentation and user support • Training sessions • User-friendly tools • Work with users: feedback • Highlight the hidden benefits of the new infrastructure • Less maintenance -> Better support

  18. Storage Data Transfer, Storage and Access

  19. Storage • Current storage system requires too much maintenance • Non-existent file catalog, fragmented disk space, … • Solution: adopt Tier1-grade Grid-based storage system • Standard tools + supported service @ PIC • LFC: Easier data management and monitoring

  20. Data transfer • Suboptimal network data transfer (SSH-based) • Insufficient bandwidth • RAW data stored in LTO tapes and sent by air mail • Poor control over network transfers • Bad integration with Grid (intermediate disk) • Integration to Grid Infrastructure • Data cluster in observation site uses GFS • Not supported by SRM • Using BeStMan to create GridFTP + SRM server • Data transfer managed by FTS + manager • Pending administrative actions to set it up • Aim to deprecate air mail transfers

  21. Data access • Data access requirements: • Access data anytime from anywhere • Two approaches: • Data access using Grid tools (GridFTP, SRM or equivalent) • Robust transfers, not easy file browsing • Not all institutes support Grid • Web access • Easy file browsing, not that easy transfers • Security concerns • Solution: • Build web-based service to interface to GridFTP + LFC • Use dCache httpDoor for “Grid-handicapped” users • Access only to local SE

  22. Data flow

  23. Computing

  24. Data Center: Computing • Traditional computing at MAGIC • Each institute uses its own computing resources (CPU + Storage) • Only few users have access to a computing farm • Data center CPUs exclusive to “official” analysis • We recently opened the computing to all users • Grid-based computing • Additional resources for users: CPU and Disk • In development: library of standard analysis tools • PIC data center will still play a central role • Data management, manpower, … • + resources & efficiency: more and better scientific outcome

  25. Standard analysis tools • Looking to create a ‘Killer Application’ • Aim: cover all steps in the analysis chain • A tool for everybody: Simple yet flexible • Based on existing tools and years of experience • Library of high level functions • Shield from Grid complexity • One central tool is easier to develop • Better user support • Most user support is for buggy user-created software • Future: Interface to submit and monitor jobs

  26. Summary

  27. Summary • MAGIC has adopted Grid as a computing model. • The use of Grid in the Data Center was key to the successful upgrade process. • A WLCG Tier-1 site is now also the Tier-0 for MAGIC reusing methodologies and personnel. • It is necessary to create customized applications for an easy access to the data and computing.

More Related