1 / 14

Analysis Framework news

A. Gheata , M. Gheata , C. Grigoras , J.F. Grosse- Oetringhaus ALICE offline week , March 2013. Analysis Framework news. General framework LEGO improvements Central QA and filtering. Outlook. File prefetching. ROOT Users Workshop 2013.

mio
Download Presentation

Analysis Framework news

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A. Gheata, M. Gheata, C. Grigoras, J.F. Grosse-Oetringhaus ALICE offline week, March 2013 Analysis Framework news

  2. General framework LEGO improvements Central QA and filtering Outlook

  3. File prefetching ROOT Users Workshop 2013

  4. Can beused in ROOT togetherwithTTreeCache to enableasynchronous file prefetching • Gain extra speed whenreadingfrom WAN, but alsofrom local storage • To beactivated in the AF • AliAnalysisManager::SetAsyncReading(kTRUE) • Stillundertesting... File prefetching

  5. Currentlywriting outputs to a TList (or other container) • Convenient (easyseparation and retrieval) • Not efficient memory-wise • To get a histogram one needs to load the full list • Merging has to load in memory the full list for the 2 files • Migration to DefineOutput(i,TDirectory::Class()) • Availableverysoon... Output folders

  6. New system allowing to tag jobs by: • Processing type • Simulation, reconstruction, analysis, calibration, merging, filtering … • Data type • ESD, AOD, MC, mixing, ... • Groups • Central production, LEGO production, user, … • Key-value based block in the JDL, understood by alimonitor • Aggregation essential information (efficiency, CPU, memory, time,...) per time slots • Possibly storing some averages per master job • Will allow real profiling of our GRID computing activity • Cross-correlations, understanding patterns, better understanding of the differences per site/job type, ... Job tags & monitoring - PLANS

  7. LEGO improvements By default your wagons are in the group 'Default' whichisshownfolded. Please click on the small + (plus) to show the content of the group Wagons are noworganized in groups which are defined by the train operators

  8. LEGO improvements Administratorscancreare new groups, delete or renameoldones Wagons are noworganized in groups which are defined by the train operators

  9. The enabling of wagons is now per dataset. • Before wagons can be enabled, the operators have to enable datasets available for the train and then as a user you can enable your wagon for the dataset(s) you want to run on. • There is also a button on the right to enable your wagon for all enabled datasets. • If you see no enable button now, please wait for your operator to enable the datasets you need. LEGO improvements

  10. LEGO improvements • The columnsowner, macro path, and dependenciescanbeshown or hidden by clicking on the plus / minus symbols in the table header. By default macro pathishidden and the others are shown

  11. Automatic e-mail when train iscompleted LEGO improvements Dear Train user,You participated in train pA_Express in run 73. The train run is completed and merged. You can view the output directory at /alice/cern.ch/user/a/alitrain/PWGCF/pA_Express/73_20130322-1842/mergeRegards, the LEGO framework on duty • Wagons enabling/disabling Buttons have been added (below the wagon section) to allow to enable or disable a list of wagons or all wagons • Clone & enable wagons button A button was added to clone a train run and enable the wagons that participated in that run at the same time

  12. Merging test • For each wagon the merging functionality is tested by multiplying the output and calling the merging macros • Tests also the Terminate functionality • 2 files per wagon and 10 files per full train test • Shown as new column in the test results • Catch cases were a full train was lost due to a single wagon failure • A train can now be submitted even if the merging test failed • Will be impossible at the next update LEGO improvements

  13. Stable procedure • Mostly run inline with reconstruction passes • No manual QA needed in the last months (except some manual merging resubmissions for very big runs) • Some manual refiltering done on request • Settings well tuned now to minimize failures and inefficiencies • Central trains tested manually with each revision • Manual tunings are needed for train configurations (configuration for delta AODs, new wagons, disabling not needed wagons, splitting parameters) • Refiltering needs special configurations Central QA and filtering trains

  14. Some part of the process could be automated • QA and filtering trains with default settings tested before tagging • Will at least detect automatically basic problems • Can give green light for pending productions • Procedure may depend on the input data • A policy to copy small datasets on the testing cluster (?) • This could aim for fast response • Larger scale tests could start in parallel, but not constraining the beginning of productions. Improvements in the QA and filtering testing procedure

More Related