1 / 31

The German Tier 1 LHCC Review, 19/20-nov-2007, stream B, part 2

Holger Marten Forschungszentrum Karlsruhe GmbH Institut for Scientific Computing, IWR Postfach 3640 D-76021 Karlsruhe. The German Tier 1 LHCC Review, 19/20-nov-2007, stream B, part 2. 0. Content. GridKa location & organization - skipped - but included in the slides

atira
Download Presentation

The German Tier 1 LHCC Review, 19/20-nov-2007, stream B, part 2

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Holger Marten Forschungszentrum Karlsruhe GmbH Institut for Scientific Computing, IWR Postfach 3640 D-76021 Karlsruhe The German Tier 1LHCC Review, 19/20-nov-2007, stream B, part 2 LHCC Review, November 19-20, 2007

  2. 0. Content GridKa location & organization - skipped - but included in the slides Resources and networks Mass storage & SRM Grid Services Reliability & 24x7 operations Plans for 2008

  3. 2. Resources and networks

  4. Current Resources in Production LCG non-LCG HEPothers CPU [kSI2k] 1864 (55%) 1270 (37%) 264 (8%) Disk [TB] 878 443 60 Tape [TB] 1007 585 120 October 2007 accounting (example): • CPUs provided through fair share • 1.6 Mio. hours wall time by 300k jobs on 2514 CPU cores • 55% LCG, 45% non-LCG HEP

  5. Installation of MoU Resources 2007 (from WLCG accounting spread sheets) installed WLCG milestone

  6. GridKa WAN connections internal network

  7. CERN redundancy GridKa WAN connections internal network

  8. The benefit of network redundancy April 26, 2007: failure of DFN router of CERN-GridKa OPN  Automatic (!) re-routing through our backup link via CNAF; this was not a test !

  9. Summary of GridKa networks • LAN • Full internal redundancy (of one router) • Additional layer-3 BelWue backup link (to be realized in 2008) • WAN • multiple 10 Gbps available to CERN, Tier-1s, Tier-2s • Sara/Nikhef: will be in production (end of Q4/2007) • additional CERN independent Tier-1 transatlantic link(s) would be highly desirable

  10. 3. Mass storage & SRM

  11. dCache & MSS at GridKa Long time instabilities with SRM and gridFTP implementation • reduced availability because SAM critical tests fail; many patches since Dual effort for complex and labour intensive software (data management) • running instable dCache SRM in production • running next SRM 2.2 release in pre-production • in the end SRM 2.2 was tested formally with F.Donnos S2 test suite, but only very limited by the experiments Read-only disk storage (T0D1) is administrative difficulty • full disks imply stopping experiment’s work => experiments ask for “temporary ad-hoc” conversions into T1D1 • no failover or maintenance (reboot) is possible, otherwise jobs will crash

  12. dCache & MSS at GridKa Migrated to dCache 1.8 with SRM 2.2 on Nov 6/7 • very fruitful collaboration with dCache/SRM developers in situ • bug fix for globus-url-copy in combination with space reservation “on-the-fly” during migration process => many thanks to Timur Perelmutov and Tigran Mkrtchyan for support Stability has to be verified during the coming months. Connection to tape (MSS) is fully functional and scalable for writes • read tests by experiments have only started recently • difficult to estimate tape resources to reach required read throughput • workgroup with local experiment representatives to provide access patterns, tape classes and recall optimisation proposals

  13. 4. Grid Services

  14. Installed WLCG middleware services* * In a wide sense, i.e. incl. physics DBs and dCache pools with grdiFTP; only production listed  FTS 2.0 deployment example

  15. FTS 2.0 [+LFC] deployment at GridKa Setup to ensure high availability. • Three nodes hosting web services. • VO- and channel agents are distributed on the three nodes. • Nodes located in 2 different cabinets to have at least one node working in case of a cabinet power failure or network switch failure. • 3 nodes RAC on Oracle 10.2.0.3, 64 bit • RAC will be shared with LFC database. • Two nodes preferred for FTS, one node preferred for LFC. • Distributed over several cabinets. • Mirrored disks in the SAN.

  16. FTS/LFC DB: One 3-node Cluster on Oracle 10.2.0.3, 64bit FTS (LFC) SAN PubIP1VirIP1 eth 1 internal network PrivIP1a external network PrivIP1b eth 2 node 1 .53 .52 192.x.x.x public VLAN VLAN RAID1 142 GB RAID1 142 GB RAID1 142 GB RAID1 142 GB 10.x.x.x PrivIP Switch1 PrivIP2a LFCREC1 LFC (FTS) PrivIP2b 192.168.52 PubP2 VirIP2 LFCDATA1 FTSDATA1 FTSREC1 Voting OCR node 2 ASMSpfile VirIP, PubIP ExtIP PrivIP3a FTS (LFC) PrivIP3b PubIP3 VirIP3 node 3

  17. Tested FTS channels GridKa ⇔ Tier-0 / 1 / 2 (likely incomplete list) Tier-0  FZK CERN - FZK FZK  Tier-2 (cont.) FZK - TROITSKINR FZK - UNIFREIBURG FZK - UNIWUPPERTAL FZK - WARSAW FZK  Tier-1 IN2P3 - FZK PIC - FZK RAL - FZK SARA - FZK TAIWAN - FZK TRIUMF - FZK BNL - FZK FNAL - FZK INFNT1 - FZK NDGFT1 - FZK FZK  Tier-2 FZK - CSCS FZK - CYFRONET FZK - DESY FZK - DESYZN FZK - FZU FZK - GSI FZK - ITEP FZK - IHEP FZK - JINR FZK - PNPI FZK - POZNAN FZK - PRAGUE FZK - RRCKI FZK - RWTHAACHEN FZK - SINP FZK - SPBSU

  18. FTS 2.0 deployment experience WLCG milestone – as a member of MB I accepted it ToDo’s @ GridKa after experience with FTS 1.5 • Migrate FTS to 3 new redundant servers => buy, install LAN, OS, … in advance • Set up new Oracle RAC (new version) on 64 bit • Migrate DB to redundant disks => new SAN configurations required • Set up and test all existing transfer channels (by all experiments) And the migration experience • learning curve for new 64-bit Oracle version • fighting esp. with changes in behaviour with two networks (internal + external) • setting up and testing channels needs people, sometimes on both ends (vacation time, workshops, local admins communicate with 3 experiments – sometimes with different views – in parallel) For sites, upgrading also means time consuming service hardening and optimization, and is not just “pushing the update button.”

  19. 5. Reliability & 24x7 operations

  20. SAM reliability (from WLCG report)

  21. SAM reliability Some examples with zero severity for experiments • config. changes of local or central services that result in failures for OPS-VO only • missing rpm ‘lcg-version’ in new WN distribution • SAM tests CA-certificates that already became officially obsolete More severe examples • pure local hardware / software failures (redundancy required…) • scalability of services after resource upgrades or during heavy load • stability of “MSS-related” software pieces (SRM, gridFTP) Overall very complex hierarchy of dependencies • esp. transient scalability and stability issues are difficult to analyse • but this is necessary:analyse + fix instead of reboot ! (sometimes at the expense of availability though)

  22. Site availability – OPS vs. CMS view To be further analysed: Do we have the correct (customers) view?

  23. To be further analysed…

  24. Preparations for 24x7 support Currently • site admins (experts) during normal working hours • experiment admins with special admin rights for VO-specific services • operators (not always “experts”) watch the system and intervene during weekends and public holidays on a voluntary basis Needs for and permanently working on • redundancy, redundancy, redundancy • multiple experts 24h x 7d x 52w on site is out of discussion • hardening / optimization of services • the more scalability tests in production, the better (even if it hurts) • but we depend on robust software • documentation of service components and procedures for operators • service dashboard for operators

  25. GridKa service dashboard for operators See A. Heiss et al., CHEP 2007

  26. 6. Plans for 2008

  27. C-RRB on 23-oct-2007: LCG status report Concern: Are sites aware of the ramp-up (incl. power & cooling)?

  28. Electricity and cooling at GridKa Planning & upgrades done during the last 3 years • second (redundant) main power line available since 2007 • 3(+1; redundancy) x 600 kW new chillers available • 1 MW of cooling (water cooling) capacity ready for 2008 Capacity not an issue, but concerned about running cost • started benchmarking of compute and el. power in 2002 • efficiency (ratio of SPECint / power consumption) enters into call for tenders since 2004 (“penalty” of 4 €/W at selection) • many discussions with providers (Intel, AMD, IBM,…) • contributing to HEPiX benchmarking group and publishing results

  29. Intel Xeon E5345Intel Xeon 5160Intel Pentium M 760AMD Opteron 270AMD Opteron 246 (b)AMD Opteron 246 (a) Intel Xeon 3.06 GHzIntel Xeon 2.66 GHzIntel Xeon 2.20 GHzIntel Pent. 3 1.26 GHz 0 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 Efficiency (SPECint_rate_base2000 per W) 2005-2007: much more promising 2001-2004: very alarming Based on own benchmarks and measurements with GridKa hardware.

  30. Extensions for 04/2008: everything is bought ! Oct’07 • 40 new cabinets delivered and installed • 1/3 of CPUs (~130 machines) delivered Nov’07: arrival and base installation of • all new networking components (incl. cabling) • remaining 2/3 of CPUs • tape cartridges & drives Nov/Dec’07: • arrival of 2.3 PB disks (incl. non-LHC) + servers Jan-Mar’08: installations, tests, acceptance, bug fixes, …

  31. Summary • GridKa contributes with full MoU 2007 resources • we are ready for the April’08 ramp-up • Good collaboration with • sites, developers and experiments (e.g. local / remote VO admins) • Much effort spent into • service hardening (redundancy …) • tools and procedures for operations • scalability and stability analysis • access performance optimization (e.g. tape reads) • This is still a necessity which requires • time of admins • patience and understanding by customers • …sometimes at the expense of reliability measures

More Related