1 / 20

Experiment support at IN2P3

Experiment support at IN2P3. Artem Trunov CC-IN2P3 artem.trunov@cc.in2p3.fr. Experiment support at IN2P3. Site introduction System support Grid support Experiment support. Introduction: Centre de Calcul of IN2P3. Serving 40+ experiments HEP Astrophysics Biomedical

craig
Download Presentation

Experiment support at IN2P3

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Experiment support at IN2P3 Artem Trunov CC-IN2P3 artem.trunov@cc.in2p3.fr

  2. Experiment support at IN2P3 • Site introduction • System support • Grid support • Experiment support

  3. Introduction: Centre de Calcul of IN2P3 • Serving 40+ experiments • HEP • Astrophysics • Biomedical • 20 years of history • 40+ employees http://cc.in2p3.fr/

  4. CC-IN2P3 - Stats • ~900 batch workers - 2200 CPUs (cores) – 3400 jobs • Dual CPU P-IV 2.4 GHz, 2GB RAM • Dual CPU Xeon 2.8GHz, 2GB RAM • Opteron Dual CPU Dual core 2.2Ghz, 8GB RAM. • Mass Storage – HPSS • 1.6 PB total volume stored • Daily average transfer volume ~10TB • Rfio access to disk cache, xrootd, dcache • Network • 10 Gb link to CERN since January 2006 • 1 core router (catalyst 6500 series) • 1 Gb uplink per 24 WN (upgradable to 2x1GB)

  5. LCG Tier 1 Center in France LCG MoU Jan 31 2006

  6. Systems support • A guard on duty 24 hours • Smoke etc alarm sensors, service provider visits • Telecom equipment sends an sms on failure • Storage robotic equipment sends signals to the service provider who acts according to the service agreement. • Unix administrator is on shift during evening hours. He can call an expert at home, but the expert is not obliged to work odd hours • Weekly reviews of problems during shifts • Nights and weekends not covered, overtime not paid. • But in case of major accidents like power failure system group is making all efforts to bring the site back as soon as possible. • During the day routine monitoring of • Slow or hang jobs, jobs that end to quick, jobs exceeding requested resources – a message is sent to submitter • Storage services for problems • Some service to end users is delegated to experiments representatives • Password reset, manipulation with job resources

  7. Grid and admin support at IN2P3 • Dedicated Grid team • People have mixed responsibilities • 2.5 – all middleware: CEs, LFC etc • + operations • 1.5 – dcache + FTS • 1.5 – HPSS administrator • 1 – Grid jobs support • Few people doing CIC, GOC development/support • + many people not working on grid • Unix, network, DB, web • Not actually providing 24/7 support • Few people do too much work. • Weekly meetings with “Tour of VOs” are very usefull storage GRID team production system

  8. Grid and admin support at IN2P3 • But Grid people not enough to make grid working • Didn’t Grid mean to help experiments? - yes • Are experiments happy? - no • Are Grid people happy with experiments? – no • “Atlas is the most ‘grid-compatible’, but it gives us a lot of problems”. • “Alice is the least ‘grid-compatible’, but it gives us a lot of problems”. • One of the answers: it’s necessary to work closely with experiments. • Lyon: a part of user support group is oriented to LHC experiment support

  9. Experiment support at CC-IN2P3 • Experiment support group - 7 person • 1 – BaBar + astrophysics • 1 – all Biomed • 1 – Alice and CMS • 1 – Atlas • 1 – D0 • 1 – CDF • 1 – general support (sw etc) • All – general support, GGUS (near future) • Plan is to have one person for each LHC experiment • All are former physicists • The director of the CC is also physicist

  10. Having physicists on site helps • Usually had in-depth experience with at least one experiment • Understand experiment’s computing models and requirements • Proactive in problem solving, have broad view on experiments computing. • Work to make physicists happy • Bring mutual benefit to the site and the experiment • Current level of Grid and experiments’ middleware really requires a lot of efforts • And grid is not getting easier, but more complex.

  11. CMS Tier1 people – who are they? • Little survey on CMS computing support at sites • Spain (PIC) – Have dedicated person • Italy (CNAF) – Have dedicated person • France (IN2P3) - yes • Germany (FZK) – no • Support is by physics community • SARA/NIKEF – no • Nordic countries – terra incognita • US (FNAL) - yes • UK (RAL) – yes, but virtually no experimental support at UK Tier2 sites

  12. What are they doing at their sites? • Making sure site setup works • integration work • optimization • Site admins usually can’t check whether their setup works and asks VO to test it. • Reducing “round trip time” between experiment and site admins • Talking is much better for understanding than email exchange. • Sometime it’s just necessary to sit together in order to resolve problems • Helping site admins to understand experiment’s use of the site • Especially “exotic” cases, like Alice (asks for xrootd) • Also requires lot of iterations • Testing new solutions • Then deploying and supporting • Administration • At Lyon, all xrootd servers are managed by user support • Grid expertise, managing VO Boxes, services (FTS)

  13. VO support scenarios • Better “range” when experiment expert is on site • Could interact directly with more people and systems • Integration is a keyword Grid admins DB systems Siteadmins storage Experiment community Grid a) Grid admins DB systems Siteadmins storage Grid b)

  14. At Lyon • Some extra privileges that on-site people use • Installing and supporting xrootd servers (BaBar, Alice, CMS, other experiments), root • Managing VO boxes, root • Installing all VO software at site - special • Co-managing FTS channels - special • Developing SW (Biomed) • Debugging transfers, jobs – root • Transfer and storage really needs attention!

  15. Grid support problems • SFT not objective • Grid service depends on some external components • E.g. foreign RB or BDII may be at fault • Too many components are unreliable • Impossible to keep a site up according to MoU • GGUS support not 24 h either. • Select some support areas and focus on them

  16. Focus points • In short, what does an experiment want from a site? • Keep the data • To ingest a data stream and archive it on tape • Access the data • Grid jobs must be able to read data back • When experiments are not happy? • Transfers are failing • Jobs are failing • What causes failures? • ► Network is stable • ► Local batch is stable • ►Grid middleware • ►Storage

  17. Debugging grid jobs • Difficult • Log file not available even for superuser – they are on a Resource Broker • Whether it’s user application failing or site’s infrastructure is not clear without VO expertise • At Lyon, production people monitor slow jobs, jobs that end too quick, jobs killed • sending messages to users – not scalable.

  18. Storage failures • Storage is another key component (transfer, data access) • Transfer made too complex • SRM • Non-interoperable storage solutions • Exotic use cases • Site storage experts can’t always debug transfers themselves • Don’t have VO certificates/roles, • Don’t have access to VO middleware that initiates transfers • Storage classes WG (T1 storage admins) is trying to • reduce complexity, • bring the terminology to the common ground, • make developers understand the real use cases, • make experiments aware of the real life solutions, • make site storage experts fully comfortable with experiments’ demands • This is all for the mutual interest • File loss is inevitable • An exchange mechanism for site-VO interactions is being developed by Storage Classes WG initiative. • But detection of the file loss is still a site’s responsibility!

  19. Storage • Needs serious monitoring • Pools down • Staging fails • Servers overloaded • Make storage a part of 24h support? Train operators to debug storage problems and localize the damage to an experiment (e.g. by notification to experiments) • Is this possible to develop such monitoring and support scenario, where problems are fixed before users complain? • Shame, when a user tells an admin “You have a problem”. Better be vice-versa. • At Lyon will try to expand storage expertise • Experiment support people are already involved. • VO will obviously have to train data taking crew on shift to recognize storage problems at sites (since exporting data from CERN is a part of initial reconstruction workflow).

  20. Little summary about what should work in Grid support • Dedicated grid team • Deep interaction with experiments, regular meetings on progress and issues • Storage/transfers monitoring • Human factor Everything will be fine!

More Related