1 / 10

MC Production System & DM catalogue

MC Production System & DM catalogue. A. Fanfani University of Bologna. ProdAgent. ProdRequest. ProdAgent. ProdMgr. ProdAgent. In production since the summer. Under High development Basic chain PR  PM  PA works. Production System Overview.

osanna
Download Presentation

MC Production System & DM catalogue

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MC Production System & DM catalogue A. Fanfani University of Bologna

  2. ProdAgent ProdRequest ProdAgent ProdMgr ProdAgent In production since the summer Under High development Basic chain PR PM PA works Production System Overview Aim at automating as much as possible, easy maintenance ProdAgent LCG/EGEE Resource ProdRequest Jobs User Request • User Interface to create requests OSG Resource Get Work ProdMgr ProdAgent Jobs Report Progress • Manage request • Allocate work to PA when PA request it • Tracks the global completion of the task Resource ProdAgent Jobs • Various Grid/batch Middleware to support • Ask for work • Convert work into processing jobs • Create, submit, track jobs • Manage the merge, failures, resubmit, local cataloguing, etc..

  3. Processing Processing Processing ProdAgent Processing Workflow Local DBS/DLS • Processing jobs sent to sites • Output data left in local SE • Report back to ProdAgent • Data management cataloguing (registration in local DBS/DLS) • Failed jobs handled automatically Tier-1 SE ProdAgent Grid WMS Tier-2 SE Tier-2 Small output file from Processing job SE

  4. Merging Merging Merging ProdAgent Merge Workflow Local DBS/DLS • Merge data at site • Watch DBS/DLS for produced unmerged data • send merge job at sites hosting data • Transfer data • PhEDEx injection Tier-1 ProdAgent SE Grid WMS Tier-2 SE Tier-2 Large output file from Merge job SE PhEDEx PhEDEx transfer invoked by PA

  5. ProdAgent Architecture ProdAgent core • Core MySQL DataBase • Python API Core Services • Work split into atomic Python Components • Asynchronous Publish/Subscribe model for inter-component communications • Simple API to communicate between components  easy to add new functionality and build on existing features • Persistent state recorded in DB

  6. ProdAgent Core DBS/DLS interface Local DBS ProdMgr interface Local DLS PHEDEX PhEDEx interface Merge Sensor Merge Accountant Job Tracking Job Creator ResourceMonitor Job Queue Job Cleanup Job Submitter Error handler BOSS DB BOSS submit LCG RB/gLite WMS Production Agent components workflow Retrieve work watchs watchs

  7. Status of each component Overview of current job status mc-physval-120-ZToMuMu-StartUpLumiPU Monitoring • PA level monitoring for operators (developed by Bari team + Carlos) overview of the current job status status of each component

  8. Prod system Status & Plan • ProdAgent implemented and deployed operationally since summer • CSA06 pre-production of 66Mevents + organized skimming run at Tier1s • PhysVal+HLT samples with CMSSW12x , see Nicola’s talk • Focus is now on automation to reduce manual work for operators • i.e. automatic block management and PhEDEX injections • and on performance to make it more scalable, more robust • Bulk creation & bulk submission with gLite (with LCG RB 2000jobs/day per PA) • Deployment of ProdRequest/ProdMgr/ProdAgent system • The production teams will no longer have to inject workflows taking them from Twiki pages • The production coordinator will assign work to teams with given priority via ProdManager • DBS-2 integration • Alpgen integration True collaborative development effort: • Dave Evans, Frank Van Lingen, Giulio Eulisse (US) • Carlos Kavka, Alessandra Fanfani, William Bacchi,Giuseppe Codispoti, contribution from Bari team (IT)

  9. Data Management catalogues: DBS • The Dataset Bookkeeping System (DBS) provides the means to define, discover and use CMS event data • First version deployed for CSA06, including data discovery browser • Development for 2nd generation (DBS-2) • Prepare the system for describing real data • Added info like run, luminosity sections, primary dataset description • Preliminary support for Analysis dataset • A subset of a Processed Dataset representing a coherent sample for physics analysis • More functionalities for browsing data discovery • Deployable with Oracle at CERN for Global DBS • Deployable with MySQL too  to be used as “local scope” DBS • Under integration with CRAB, ProdAgent, PhEDEx, MTCC data

  10. Data Management catalogues: DLS • The Data Location Service (DLS) provides the means to locate replicas of data in the distributed computing system • The DBS knows how datasets are organized in term of file-blocks • The Data Location Service (DLS) maps file-blocks to storage elements (SE’s) • DLS based on LCG LFC used for CSA06: • some drawbacks (performance issues for reverse lookup, i.e. data discovery), some advantages (production service mantained by LCG, VOMS authentication-authorization, DLI) • No server-side work needed by CMS • Serving us right so far • Evaluating to have DLS on the same server as DBS • Decide based on CMS use case • Still keeping DLS API functionalities • Add support for the Resource Broker to talk to directly (via DLI)

More Related