1 / 22

CMS Monte Carlo Production in LCG

CMS Monte Carlo Production in LCG. J. Caballero, J.M . Hernández, P. García-Abia (CIEMAT) CMS Collaboration. Computing in High Energy and Nuclear Physics, T.I.F.R. Mumbai, India, 13-17 February 2006. Outline. Introduction Monte Carlo production framework:

brandy
Download Presentation

CMS Monte Carlo Production in LCG

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CMS Monte Carlo Production in LCG J. Caballero, J.M. Hernández, P. García-Abia (CIEMAT) CMS Collaboration Computing in High Energy and Nuclear Physics, T.I.F.R. Mumbai, India, 13-17 February 2006

  2. Outline • Introduction • Monte Carlo production framework: • Data tiers, metadata attachment, publication of data • Production workflow • First experiences • Improvements to production: • Output ZIP archives, treatment of pile-up, local software installation • Production operations • Efficiency, problems • Migration to LFC • The new MC production system • Conclusions CMS Monte Carlo Production in LCG

  3. Introduction • Monte Carlo (MC) production is crucial for detector studies and physics analysis • Event simulation and reconstruction typically done in computer farms of CMS institutions • Porting production to LCG allows using a large amount of computing, storage and network resources • MC simulation was previously run in a LCG0-LCG1 dedicated testbed: • Low scale production • Low efficiency: RLS, site configuration • We had introduced novel concepts which have made running the full production chain possible on LCG, from the generation of events to the publication of data for analysis • We had coupled production and the CMS data transfer system (PhEDEx) and made tools more robust • Important implications for the design of the new production framework CMS Monte Carlo Production in LCG

  4. Introduction II • The CMS event data model (EDM) and the MC production framework are somewhat monolithic, not suitable for a Grid environment: • Lack of modularity • Grid provides basic services: • Reliability, stability and flexibility are important issues • We have identified the main weak points of LCG and made the production framework more robust: • Efficient running of production in LCG is manpower intensive • Availability of the resources and responsiveness of the local administrators are crucial • Code development-and-testing and running of production in LCG done by ~1.5=FTE at CIEMAT CMS Monte Carlo Production in LCG

  5. Production framework • The basic unit in MC production is the dataset: • a given physics process with a well defined set of parameters • The production chain is: • generation of events, simulation (hits), digitization (digis) and reconstruction (DST) • these are called data tiers or steps • owner: data tier with defined geometry, SW version and pile-up (PU) sample • Detector and physics groups request events of a specific dataset/owner pair • For practical reasons, requests are split in small assignments composed of a number of runs (~1000 events) • The data relevant to production (requests, owner/dataset, assignments, runs, data attributes) are kept in a global database (RefDB) • The MC production framework is McRunjob, a python application developed at FNAL, used for local farm production since long CMS Monte Carlo Production in LCG

  6. Data tiers • Generation: • no input, small output (10 to 50 MB ntuples) • pure CPU: few minutes, up to few hours if hard filtering present • Simulation (hits): GEANT4 • small input • CPU and memory intensive: 24 to 48 hours • large output: ~500 MB in three files (EVD files), the smallest is ~ 100 KB ! • Digitization: • lower CPU/memory requirements: 5 to 10 hours • I/O intensive: persistent reading of PU through LAN • large output: similar to simulation • Reconstruction: • even less CPU: ~5 hours • smaller output: ~200 MB in two files CMS Monte Carlo Production in LCG

  7. Event metadata attachment • In order to run the digitization step, event metadata have to be generated for the whole collection of simulated events • When running reconstruction the metadata of both the simulated and the digitized events are required • The generation of metadata (metadata attachment) needs direct access to the event files, not suitable for distributed systems: • output of the jobs potentially distributed among several Storage Elements with no POSIX I/O-like access • Metadata attachment was the main show-stopper for porting the MC production system to LCG: • lack of modularity (atomicity) in the old EDM • We introduced the concept of atomic attachment: • Metadata attachment done on the Worker Node for the run to be processed • Negligible overhead: EVD files already in the working area CMS Monte Carlo Production in LCG

  8. Publication of data • We have coupled production in LCG and the CMS data transfer system: • PhEDEx is used to collect event files in the T1s/T2s that host data for analysis • However, data handling for intermediate steps not done by PhEDEx... (this is one of the main problems in production) • For each owner/dataset, a global metadata attachment is performed: • metadata and local XML POOL file catalogs are produced and made public in the data location system (global RefDB and local data location DB -PubDB-) • Analysis tools inspect RefDB/PubDB for data discovery: • Analysis jobs are submitted to the appropriate T1/T2 CMS Monte Carlo Production in LCG

  9. Production workflow • Job preparation: • McRunjob downloads assignment information from RefDB: • List of runs, job templates, application data-cards, input file specification, input/output virgin metadata • Jobs are created for each run using the templates: • Application scripts • JDL file with grid requirements (CPU, memory, SW tags, site...) • Wrapper script: specific stuff to let the job run in a LCG WN • Jobs are submitted to a LCG CE using the JDL • At runtime on the WN: • Input files are downloaded from SE • Metadata are generated for the input event files • After the application runs the output EVDs are copied to the SE • The summary file is returned in the output sandbox and sent to RefDB from the UI for validation of the job • Originally, also the application output/error was returned CMS Monte Carlo Production in LCG

  10. First experiences • The first experiences were disappointing: • Extremely high submission time (very low rate) • Very low job efficiency • Job retrieval time too high: huge output • Failure causes: • Local configuration problems: unavailability of CMS software (installation problems), NFS • Instability of the RLS global catalog • Problems staging in/out files: weak staging procedure, copy from/to the SE unreliable • Poor error report from the application: hard to automatise job resubmission, typically done after visual inspection of the logs • Real time monitoring unavailable CMS Monte Carlo Production in LCG

  11. Improvements to production • We introduced new ideas in the production system in order to make it more robust • Output/error files of the application removed from the output sandbox: • Size largely reduced: significant improvement of the job retrieval rate • Virgin metadata and XML POOL catalog of the job removed from the input sandbox (size reduced to 10 KB): • Stored in several SEs at job submission time (atomic operation) to improve their availability • Significant improvement of the job submission rate • More robust stage in/out procedure: • Failing input/output operations to/from the SE are retried several times (with a delay) to avoid temporary access problems to the SE/RLS • The copy of the job output is tried on several SEs if one fails CMS Monte Carlo Production in LCG

  12. Output ZIP archives • At job completion time, the output EVD files are packed in a ZIP archive (without compression) together with other important files: • Checksum of the EVD files, XML POOL catalog fragment of the output files, summary file, output and error files of the application • Just one big file is copied to the SE, instead of several EVD files • One of the EVDs is only 100 KB in size (very bad for MSS performance) • CMS applications can read files inside uncompressed ZIP archives (without unpacking them) • Zipping had implications for the job preparation of subsequent steps: • We instrumented the job wrapper to deal properly with ZIPs • ... and in the publication of data: • The publication tool, CMSGLIDE (M.A. Afaq, FNAL), was modified to be able to create XML POOL catalogs and attached metadata for production ZIPs • Global metadata attachment done using ZIPs (w/o unzipping) • Zipping of EVDs widely adopted in CMS: EVDs produced in local farmas merged into 2 GB files • Great benefits for PhEDEx and MSS: much less, much larger, files CMS Monte Carlo Production in LCG

  13. Treatment of pile-up • Proper simulation of events requires the superposition of events from inelastic pp interactions on the events of simulated physics processes • Large pile-up (PU) samples prepared at CERN: about 100 GB • Local farm: PU installed locally and made accessible to the jobs at runtime via POSIX I/O-like (rfio, dcap) • LCG: PU sample, EVD (zipped) and metadata transferred with PhEDEx to T1/T2 sites that will run digitization/reconstruction: • The XML POOL catalog of the PU, with site-dependent PFNs/protocol, is placed in a standard location • Publish a LCG software tag for the PU in the grid information system, used as a requirement in the JDL of the jobs • At runtime, the job wrapper merges the PU catalog with that of the job • This simple (novel) implementation has been crucial for running digitization and reconstruction jobs in LCG CMS Monte Carlo Production in LCG

  14. Other interesting ideas • Local CMS software installation at runtime: • We instrumented the job wrapper to install the CMS software in the working area of the job • Little overhead: software downloaded from the SE • Avoid NFS problems, software installation problems, black holes • Suitable for running in sites with little or no support for CMS • Local pile-up installation at runtime (a la ATLAS): • Store and replicate the PU sample in several SEs • Download a (random) fraction of the PU sample • Generate metadata for the PU runs downloaded • An experimental version exists, not used for physics: • Important to determine the number of events required to have minimal or no impact on physics • Need to study tradeoff between local access and downloading of files (LAN) CMS Monte Carlo Production in LCG

  15. Production operations • Production in LCG started slowly one year ago with reduced manpower: • Development/testing of the McRunjob-LCG software and production operations done for a long time by ~1.5 FTE • Other production operators joined the effort few months later • The number of events (in millions) produced in LCG per data tier are: • 13.1 generated, 11.7 simulated, 11.4 digitized, 5.1 reconstructed 14 M 12 M Simulation 11/04 to 02/06 Digitization 06/05 to 02/06 CMS Monte Carlo Production in LCG

  16. Production operations II • We decided to use white lists due to grid/site unreliability: • Sites selected for their size and robustness • Big sites still running production in local farm mode (FZK, IN2P3) • Local administrators providing fast response: fix problems • Availability of PU • This represents a fraction (~30 %) of the production in LCG • No proper bookkeeping in the initial phase of production 9800 digitization and reconstruction jobs 4275 simulation jobs CMS Monte Carlo Production in LCG

  17. Efficiency and failures • Rather low efficiency • Stage in/out and catalog (RLS) related problems • LCG and site problems: RB, CE CMS Monte Carlo Production in LCG

  18. Example of problems jobs • Lack of automatic monitoring/resubmission • Lack of coupling of the CMS data management system (pre-staging) • Temporary grid and site problems: CE, SE, RLS • Lack of manpower • Organization: lack of dedicated resources (PU) • Lack of priorities: competition with CMS analysis and other experiments’ jobs CMS Monte Carlo Production in LCG

  19. Migration to LFC • Recently, CMS has migrated from RLS to LFC, as a global file catalog for LCG (thanks to S. Lemaitre, A. Sciabà , J. Casey) • We adapted McRunjob to use LFC instead of RLS • So far, a small fraction of production in LCG done using LFC • Very satisfactory results as compared to RLS CMS Monte Carlo Production in LCG

  20. Performance of production (LFC) • Significant improvement in performance when using LFC as a global catalog • A bunch of jobs died due to a unscheduled power cut CMS Monte Carlo Production in LCG

  21. New MC production system • Expert system (prodagent) • Automatic data merging step • Job chaining • Coupled to the Data Management System • New EDM (no metadata attachment) • Improved monitoring • Better error handling CMS Monte Carlo Production in LCG

  22. Conclusions • End-to-end production system in LCG • Invaluable experience for the next generation Monte Carlo production system • Robustness is very a important issue given the current unreliability and instability of grid/sites CMS Monte Carlo Production in LCG

More Related