uki southgrid overview gridpp30 n.
Skip this Video
Loading SlideShow in 5 Seconds..
UKI-SouthGrid Overview GridPP30 PowerPoint Presentation
Download Presentation
UKI-SouthGrid Overview GridPP30

Loading in 2 Seconds...

play fullscreen
1 / 23

UKI-SouthGrid Overview GridPP30 - PowerPoint PPT Presentation

  • Uploaded on

UKI-SouthGrid Overview GridPP30. Pete Gronbech SouthGrid Technical Coordinator and GridPP Project Manager Glasgow - March 2012. UK Tier 2 reported CPU – Historical View to present. Last reported stats at CERN in Sept 2011, so data since then. SouthGrid Sites Accounting as reported by APEL.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'UKI-SouthGrid Overview GridPP30' - winchell-vance

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
uki southgrid overview gridpp30

UKI-SouthGrid OverviewGridPP30

Pete Gronbech

SouthGrid Technical Coordinator and GridPP Project Manager

Glasgow - March 2012


UK Tier 2 reported CPU

– Historical View to present

  • Last reported stats at CERN in Sept 2011, so data since then.

SouthGrid March 2013

vo usage
VO Usage

Usage dominated by LHC VOs.

6% Non LHC

SouthGrid March 2013

non lhc vos

A wide range of ‘Other VOs’

gridpp4 h w generated mou for 2012 14 from steve lloyd
Gridpp4 h/w generated MoU for 2012-14 from Steve Lloyd

Q412 Resources

SouthGrid December 2012

question of mou s
Question of MoU’s
  • New experimental requirements in Sept 2012 generated new increased shares.

SouthGrid March 2013

dave britton generated mou shown at gridpp29
Dave Britton generated MoU shown at GridPP29

Q412 Resources

Up 151TB

Up 2933 HS06

SouthGrid December 2012

  • The site has been under used for the last 6 months.
  • This is a non Particle Physics site so all LHC work is a bonus.
  • Essentially a pure CPU site
    • 1772 HepSPEC06
    • 10.5 Tb of storage
  • All service nodes have been upgraded to EMI2
  • CVMFS has been setup and configured for LHCb and Atlas.
  • Could be utilised much more!!
  • Active non LHC VOs : Biomed, esr, fusion and Pheno

SouthGrid March 2013


Birmingham Tier 2 Site

  • Most active Non LHC VOs : Biomed and ILC.
  • MS has helped a local Neuroscience group set up a VO, grid work to follow. (Some involvement with CERN@school)
  • Mark Slater is the LHCb UK Operations rep/ shifter
  • Major VO’s are Atlas , ALICE and LHCb.
  • Middleware:
    • DPM, WN: EMI 2, Everything else: UMD 1
  • Complete overhaul of aircon in the last 12 months. One unit (14kW) left to be installed in the next couple of weeks (hopefully!)
  • CVMFS fully installed.
  • Now providing 110TB space for ALICE in their own xrootd area. No other xrootd/webDAV updates

SouthGrid March 2013



  • StoRM SE upgraded to SL6 Storm 1.10. problematic at first but StoRM developers helped with modify from default config. Helped debug why it was publishing 0 used, apparently known bug.
  • Upgrades to EMI middleware has improved CMS site readiness
  • CVMFS setup for CMS, Atlas & LHCb
  • Onoing development of Hadoop SE: gridFTP + SRM server ready, set-up of PhEDEx(Debug) in progress
  • Active non LHC Vos : ILC
  • Landslides VO work currently on hold.
  • Working with CMS to plan the best way forward for Bristol

SouthGrid March 2013

  • Status
    • CPU : 140 job slots, 1657 HS06
    • Storage : 277TB [si]
    • Most active non LHC Vos: Camont, but almost exclusively and Atlas /LHCb site.
  • The Camtology/Imense work at Cambridge has essentially finished (we still host some of their kit)
  • We have an involvement in a couple of Computational Radiotherapy projects (VoxTox and AccelRT) where they may possibly be some interest in using the Grid

SouthGrid March 2013

  • SouthGrid’s biggest site.
  • Major VO’s CMS, Atlas and LHCb
  • Non LHC VOs – Biomed, ILC and esr.
  • Planned migration to a new computer room this year, with six water cooled racks. Will try to minimise downtime. Other racks will move to the Atlas building.
  • SE is dCache 1.9.12 – planning to upgrade to 2.2 in the near future.
  • 20Gbit link between the two computer rooms.
  • Rob Harper is on the security team
  • New member of staff (Ian Loader) starting very soon.

SouthGrid March 2013

  • Oxford’s workload is dominated by ATLAS analysis and production
  • Most active non LHC VOs : esr, fusion, hone, pheno, t2k and zeus
  • Recent Upgrades
    • We have a 10Gbit link to the JANET router but is currently rate capped at 5Gbps.
    • Oxford will get a second 10Gbit line enabled soon and then the rate cap can get lifted
  • SouthGrid Support
    • Providing support for Bristol, Sussex and JET
    • The Landslides VO supported at Oxford and Bristol
    • Helped bring Sussex onto the Grid as an Atlas site
  • Oxford Particle Physics Masterclasses with Grid Computing talk.

SouthGrid March 2013

other oxford work
Other Oxford Work
  • CMS Tier 3
    • Supported by RALPPD’s PhEDEx server. Now configured to use CVMFS and xrootd as the local file access protocol
    • Useful for CMS, and for us, keeping the site busy in quiet times
    • However can block Atlas jobs soduring accounting period max running jobs limit applied
    • Largest non CMS Tier-2 site in UK
  • ALICE Support
    • The ALICE computational requirements are shared between Birminghamand Oxford.
  • UK Regional Monitoring
    • Kashif runs the nagios based WLCG monitoring on the servers at Oxford
    • These include the Nagios server itself, and support nodes for it, SE, MyProxy and WMS/LB
    • KM also remotely manages the failover instance at Lancaster
    • There are very regular software updates for the WLCG Nagios monitoring.
  • VOMS server replication at Oxford (and IC)
  • Early Adopters
    • In the past were official early adopters for testing of CREAM, ARGUS and torque_utils. Recently have tested early in a less official way.

SouthGrid March 2013

multi vo nagios monitoring
Multi VO Nagios Monitoring
  • Monitoring three VO’s
  • Customized to suit small VO’s
    • Only using direct job submission
    • Test jobs submitted every 8 hours
    • Test jobs can stay in queue for 6 hours before being cancelled by Nagios
multi vo nagios monitoring1
Multi VO Nagios Monitoring
  • Using VO-feed for topology information
    • VO-feed is also hosted at Oxford
    • Fine grained control of what to monitor
    • But it requires manual changes
  • Probably it is the first production Multi-VO Nagios instance in egi
    • Found many bugs
    • It worked very well with two VO’s but after adding third VO it showing some strange issues.
    • Opened a GGUS ticket and working on it
gridpp cloud work at oxford
GridPP Cloud work at Oxford
  • Working to provide an OpenStack based cloud infrastructure at Oxford
  • OpenStack Folsom release installed with the help of Cobbler and Puppet
    • Using RHEL 6.4
    • Most of installation and configuration is automated
  • Started with three machines
    • One controller node running OpenStack core services
    • One Compute node running Nova-compute and Nova-network
    • One storage node to provide block storage and NFS mount for glance images
  • Plan to add more compute nodes in future
  • We are open to provide our infrastructure for testing
xrootd and webdav
XrootD and WebDAV
  • Xrootd and WebDAV on DPM
  • Completely separate services, but similar minimum requirements, so if you can do XrootD, you can do WebDAV
  • Local XrootD is pretty well organised, federations currently require name lookup libraries which lack a good distribution mechanism.
  • Configuration is mostly boiler plate – copy ours:

# Federated xrootd


# Atlas federated xrootd




# CMS federated xrootd


DPM_XROOTD_FED_CMS_NAMELIB=" file:/etc/xrootd/storage.xml?protocol=xroot"

# General local xrootd


DPM_XROOTD_DISK_MISC="xrootd.monitor all rbuff 32k auth flush 30s window 5s dest files info user ioredir

if exec xrootd every 60s all -buff -poll sync




SouthGrid March 2013

  • The main source of documentation is here:

  • We’ve also switched to using xrootd for ATLAS and CMS local file access (which is a VO side change, not a site one), but this isn’t making use of the federation yet.
  • ATLAS are currently using XrootD based file stager copies, not XrootD direct IO. We do hope to try that too.
  • All the xrootd traffic currently gets reported to the ATLAS FAX monitoring, which made the graphs look a bit odd when we turned it on:

SouthGrid March 2013

  • WebDAV is simpler:

# DPM webdav

DPM_DAV="yes" # Enable DAV access

DPM_DAV_NS_FLAGS="Write" # Allow write access on the NS node

DPM_DAV_DISK_FLAGS="Write" # Allow write access on the disk nodes

DPM_DAV_SECURE_REDIRECT="On" # Enable redirection from head to disk using plain HTTP.

  • Commodity clients work, but nothing supports everything you’d want.
  • More here:

SouthGrid March 2013

  • Sussex has a significant local ATLAS group, their system is designed for the high IO bandwidth patterns that ATLAS analysis can generate.
  • EMI2 middleware installed
  • CVMFS installed and configured
  • Setup as an Atlas production site running jobs in anger since February 2013
  • New 64 core node arriving in the next week and a further 128 cores and 120TB to be added in the summer.
  • SNO+ is expected to start using the site shortly.
  • JANET link scheduled to be upgraded from current 2Gb (plus 1Gb resilient failover) to 10Gb in Autumn 2013.

SouthGrid March 2013

  • SouthGrid seven sites well utilised, but some sites small compared with others.
  • Birmingham supporting Atlas, Alice and LHCb.
  • Bristol; Have upgraded to the latest version of STORM, and EMI middleware and have been available to CMS since mid December. Hope to be better utilised by CMS now. Local funding will be used to enhance the CPU and storage.
  • Cambridge; size was reduce when Condor part of the cluster decommissioned, local funding to be used to boost capacity.
  • JET continue to be available as a CPU site but very little storage available. However CVMFS and all middleware at EMI2. Should be a useful MC site for LHCb
  • Oxford wide involvement in many VOs and areas of development and GridPP infrastructure.
  • RALPPD remain SouthGrid s largest site and a major CMS contributor.
  • Sussex; successfully running as an Atlas site with some local upgrades planned for this year.

SouthGrid March 2013