The fnal cms glideinwms experience at bnl
This presentation is the property of its rightful owner.
Sponsored Links
1 / 15

The FNAL/CMS GlideinWMS: experience at BNL PowerPoint PPT Presentation


  • 50 Views
  • Uploaded on
  • Presentation posted in: General

The FNAL/CMS GlideinWMS: experience at BNL. Maxim Potekhin Panda/DDM Workshop October 4, 2007 BNL. glideinWMS. What will be covered: a very brief overview of a few existing Workload Management Systems general idea of the FNAL/CMS glideinWMS (I.Sfilogoi, FNAL)

Download Presentation

The FNAL/CMS GlideinWMS: experience at BNL

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


The fnal cms glideinwms experience at bnl

The FNAL/CMS GlideinWMS: experience at BNL

Maxim Potekhin

Panda/DDM Workshop

October 4, 2007

BNL


Glideinwms

glideinWMS

  • What will be covered:

    • a very brief overview of a few existing Workload Management Systems

    • general idea of the FNAL/CMS glideinWMS (I.Sfilogoi, FNAL)

    • glideinWMS test bench at BNL

    • strengths and weaknesses of the glideinWMS, and what we can learn from it, in the Panda system context


Workload management systems overview

Workload Management Systems Overview

  • for details, see a talk by Igor Sfiligoi and Burt Holzman at CHEP07:

    • http://indico.cern.ch/contributionDisplay.py?contribId=216&sessionId=26&confId=3580

  • systems considered:

    • Condor-G

    • ReSS

    • gLite WMS

    • glideinWMS – Igor’s effort

  • what was looked at:

    • performance

    • scalability

    • reliability


Workload management systems overview1

Workload Management Systems Overview

  • notes on Condor-G:

    • significantly, it is used as the underlying submission mechanism for most others WMS

    • part of the Condor distribution

    • job submission with Condor-G:

      • scales up to 7k jobs in the queue

      • start-up speed depends on the number of Grid Managers in configuration (from 30 with 30 managers to 60 with a 100)

      • same applies to job-removal speed

      • empirical result: with 100 managers, Condor-G has enough throughput to saturate the batch system

    • with CE crashes, jobs may stay in the queue forever


Workload management systems overview2

Workload Management Systems Overview

  • notes on ReSS:

    • Resource Selection System is a matchmaker for Condor-G, using the information harvested from CEMon on the Grid sites

  • submission is still done via Condor-G, with ReSS responsible for the determination as to where to submit

  • tested with 4x10k jobs queued

  • characteristics similar to plain Condor-G


Workload management systems overview3

Workload Management Systems Overview

  • notes on gLite WMS:

    • relies on BDII fir information on Grid sites

    • has a dedicated cliens

    • uses Condor-G fro submission

  • performance

    • slow, down to 5 submissions per minute

    • however, in collection mode, submission can be very fast with an effective rate of 1000 jobs per minute. Caveat: occasional failures of the collection submissions due to overload of the WMS itself

  • monitoring:

    • no easy way to find IDs of owned jobs


Glideinwms1

glideinWMS

  • For a complete overview of the FNAL/CMS glideinWMS, see http://home.fnal.gov/~sfiligoi/glideinWMS/doc/manual/index.html#overview

    (two following slides are borrowed from that source)

  • in the context of glideinWMS, a glidein is a Condor startd submitted as a Grid job

  • once it starts, it registers with a Condor pool available from the submission node

  • the users then submit their payload jobs to Condor, while being insulated from the Grid implementation details

  • obvious benefits of a familiar environment and monitoring tools that come with it


Glideinwms2

glideinWMS


Glideinwms3

glideinWMS


Glideinwms4

glideinWMS

  • our experience with glideinWMS:

    • the installation script is complex, mostly works and is being improved

    • due to ready available support from the developer (I.Sfiligoi) in the installation stage, we got it up and running after a few initial misconfigurations

    • for testing purposes, the Front End and the Factory were colocated on the submission machine

    • used an instance of the Apache server on a separate node to host the payload

    • we successfully ran a test job on Panda Pilots, which in turn were deployed on the BNL cluster via the glideinWMS mechanism – a nice proof of principle

    • the monitoring tools that come with the product work rather nicely (provide detailed stats of the factory operation in graphic and XML formats)


Glideinwms5

glideinWMS

  • The payload hosted on the Apache server: note the sha1sum file that is used to verify the payload’s authenticity


Glideinwms6

glideinWMS

  • observations:

    • Condor-G is used for glidein submissions; we can, therefore, expect same intrinsic limitations as with other WMS

    • Q: how does glideinWMS apparently do better in certain tests?

    • A: preemptive submission of startd’s, which allows the following:

      • sequential execution of jobs off the same startd

      • effectively, advance reservation on remote sites by occupying batch slots

    • Q:is that compatible with current practices and policies?

    • A:remains to be seen, and probably not... (cf the large number of slots that need to be “hot” for speedy submission)

    • Q:what happens to the unused glideins?

    • A:they die after a predetermined timeout, typically 20 minutes

    • empirically, the latencies behave as expected, i.e. when a sufficeint number of glideins are active, the submissions are indeed speedy; when there is not enough glideins, the submission is effectively throttled


Glideinwms7

glideinWMS

  • observations:

    • Q: how does glideinWMS handle inter-site, inter-process communications despite the presence of firewalls?

    • A: by using the GCB (Generic Connection Broker), which is reached by the two communicating nodes via an outgoing connection on either side. The GCB must reside on the public Internet.

    • Q:Howdoes using the GCB effect the security?

    • A:Most likely slightly adversely, and remains to be seen (see the GCB site for detail). Almost by definition, anything that defeats a firewall can’t possibly enhance security of the system.

    • Q:are there known scalability problems with GCB?

    • A:yes, a GCB instance will keel over if more than approx. 600 connections are made, and will take all the associated Condor jobs with it. Work is being done to rectify that, however currently this is more than a trivial limitation and potential vulnerability of the production system


Glideinwms8

glideinWMS

  • observations:

    • Q: what are other scalability problems with the glideinWMS?

    • A: Due to memory requirements of the current Condor implementations, the submission machine itself can be a bottleneck, however tests were successful with up to 4k running jobs and this problem can be further circumvented by using multiple submission machines; number of queued jobs can be significantly higher.

    • Scalability issues can be addressed with multiple Front Ends and multiple Factories

    • Hardware matters! Dedicated machines needed for critical functions.

  • redundancy?

    • one of the useful features of the glideinWMS is that a loss of a startd process is handled elastically in the system, i.e. not user jobs are lost. However, this overlaps with an identical feature of the Panda Pilot submission... Do we need an extra layer of indirection?


Glideinwms9

glideinWMS

  • Conclusions:

    • we have learned a lot about the glideinWMS, and having installed it locally, demonstrated that it can be used to instantiate Panda Pilots on the Grid

    • we formed an opinion that the glideinWMS is, in principle, subject to the same limitations as any WMS which is using Condor-G as the underlying remote job submission protocol, however it works around them by running payloads serially on the same CE (which is effectively sequestered by the user) and in addition having a configurable number of jobs idling on the remote site, waiting for payload. The latter can be achieved with the existing Panda system, yet it is not likely to be aligned with site policies

    • we are in the process of integrating our experience with the glideinWMS into our work with the Panda Pilot

    • certain features of the glideinWMS are highly practical and can be used in our systems, such as using checksums to validate payloads, thus mitigating security gaps in the Panda system


  • Login