XML Summary: Implementation

1. XML Summary: Implementation Robert W. Lambert Software Week, 9th Oct 2009

2. Example Summary • The XML summary is a computer and human-readable shorter version of the stdout, with salient information kept <?xml version="1.0" encoding="UTF-8"?> <summary version="1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="XMLSummary.xsd"> <success>true</success> <step>finalize</step> <usage> <stat unit="b" useOf="MemoryMaximum">1024.0</stat> </usage> <input> <file status="full" name="PFN:/path/file.dst" GUID="0000-0000-0000-0000">500</file> </input> <output> <file status="part" name="PFN:/path/file.dst" GUID="">500</file> </output> <counters> <counter name="UserAlg/UserEventCounter">500</counter> <statEntity name="UserAlg/UserStatEntity“ format="Flag Entries Flag2" max="1.0" min="1.0">500.0 500 500.0</statEntity> </counters> <lumiCounters> <counter name=“LumiAlg/EmptyEvents">1000000</counter> <statEntity name="LumiAlg/SomeComplicatedMeasurement" format="Flag Entries Flag2" max="1000.0" min="0.0">40050.0 500 2500001.0</statEntity> </lumiCounters> </summary> Software Week, 9th Oct 2009

3. What’s stored in XML? • Counters defined in tools/algorithms • Simple counters and StatEntities • Configurable for each alg, all as simple counters by default • Lumi counters (always written if declared) • List of files, input and output • Name, GUID • Status: “none” “fail” “part” “full” “mult” • Events processed • Status of Gaudi • Initialize, execute, finalize • “Application manager has terminated successfully” • Usage statistics (Memory usage) Software Week, 9th Oct 2009

4. What’s ready today? • Counters defined in tools/algorithms • Simple counters and StatEntities • Configurable for each alg, all as simple counters by default • Lumi counters (always written if declared) • List of files, input and output • Name, GUID • Status: “none” “fail” “part” “full” “mult” • Events processed • Status of Gaudi • Initialize, execute, finalize • “Application manager has terminated successfully” • Usage statistics (Memory usage) Software Week, 9th Oct 2009

5. How does it work? XML Schema (XSD) Gaudi #include “Python.h” Python Parsing Code DIRAC Ganga XML Summary (XML) Software Week, 9th Oct 2009

6. What’s ready today XML Schema (XSD) Gaudi #include “Python.h” Python Parsing Code DIRAC Ganga XML Summary (XML) Software Week, 9th Oct 2009

7. Why should I use it? • Typical user job: • ~1000 counters in a stripping job • ~2000 counters in an HLT job • ~10,000 error messages • Benchmarks: • Time: 0.5 s to write xml with 2000 counters (supra-linear) • Size: 10-times smaller than stdout (at least) • Readable/useable: • Merging of summaries already implemented • Vastly simplifies the first step in debugging Software Week, 9th Oct 2009

8. How can I get it • It is currently in the lhcb3 nightly, requires: • The next Gaudi release (21r4?) • The next LHCb release (28r1?) • Turn it on with two lines: • (at the moment, SetupProject <project> HEAD --nightly lhcb3) • Get the python code within another cmt project: • project.cmt -- use LHCb v28r1 • requirements -- use XMLSummaryBase v* Kernel from Configurables import LHCbApp LHCbApp().XMLSummary=“summary.xml” Software Week, 9th Oct 2009

9. Where is it documented? • DOxygen (once it is compiled) • This talk, and my talk in the core software meeting: • http://indico.cern.ch/materialDisplay.py?contribId=0&materialId=slides&confId=55771 • A TWiki FAQ (already quite comprehensive) • https://twiki.cern.ch/twiki/bin/view/LHCb/FAQ/XMLSummaryFAQ • A DaVinci Tutorial • (to be written) Software Week, 9th Oct 2009

10. For Gaudi Experts Software Week, 9th Oct 2009

11. The Gaudi side • The objects to write come from different times and places Event Selector (initialize) XMLSummarySvc #include “Python.h” Filenames/GUID, pull XSD GaudiCommon (finalize) Counters, push Python Incident, push Generic, +Luminosity IncidentSvc End of file, push XML IODataStream, Event Selector, PoolDbCnvSvc, RANDOM Failed file, push Written in Initialize, Finalize, and every time an input file is completed. Output Files, push Opened file, push PoolDbCnvSvc, Output Stream RANDOM Software Week, 9th Oct 2009

12. Package organisation Gaudi Core LHCb Kernel / XMLSummaryKernel LHCb Kernel / XMLSummaryBase GaudiUtils GaudiSvc GaudiPoolDb GaudiAlg GaudiKernel use GaudiKernel use XMLSummaryBase use Python use GaudiSvc use GaudiUtils private use GaudiPolicy GaudiCommon XML Schema (XSD) ICounterSummarySvc XMLSummarySvc #include “Python.h” Event Selector, OutputStream, PoolDbCnvSvc, IODataStream Simple Parsing Code tests IncidentSvc tests Software Week, 9th Oct 2009

13. More options • LHCbApp configurable, delegates to XMLSummary • To configure the summary itself: #what filename to write LHCbApp().xmlfile="summary.xml“ #no output file LHCbApp().xmlfile="“ #no output file LHCbApp().xmlfile=None #No service #what filename to write XMLSummary().xmlfile="summary.xml“ #no output file XMLSummary().xmlfile="“ #no output file XMLSummary().xmlfile=None #No service xmlsum= XMLSummarySvc(“CounterSummarySvc”) #what schema to use xmlsum.xsdfile=“$XMLSUMMARYROOT/xml/XMLSummary.xsd“ #output level xmlsum.OutputLevel=1 #what counters to fill, list of regex xmlsum.counter_list=[ “.*” ] xmlsum.stat_entity_list=[ “.*” ] #default saves all declared counters Software Week, 9th Oct 2009

14. Customise the counters • New counter alg with nameable counter: • Configure any algorithm or tool: • Only save my counters: myCount= EventCountAlg(“RWL_Count”) myCount.CounterName=“RWL_Name” #will appear as RWL_Count/RWL_Name mySelectionAlg= CombineParticles(“RWL_Something”) mySelectionAlg.CounterList=[ “.*” ] mySelectionAlg.StatEntityList=[ “.*fficiency.*” ] #add efficiency as a StatEntity, not just an int xmlsum= XMLSummarySvc(“CounterSummarySvc”) xmlsum.counter_list=[ “.*RWL.*” ] xmlsum.stat_entity_list=[ “.*RWL.*” ] #Will save anything with RWL in it Software Week, 9th Oct 2009

15. Luminosity • To determine the luminosity may require: • List of files processed ... exists • number of events per file ...exists • Sum of particular integers or complicated StatEntities • Lumi counters need to be added directly, probably in finalize: if(m_counterSummarySvc != NULL){ //declare your Counter to be stored as an integer m_counterSummarySvc->addCounter(name(), //name of the alg/tool counter_name_string, my_stat_entity_counter, Gaudi::CounterSummary::SaveAlwaysSimpleCounter); //declare your StatEntity to be stored as a StatEntity m_counterSummarySvc->addCounter(name(), //name of the alg/tool counter_name_string_2, my_stat_entity_counter_2, Gaudi::CounterSummary::SaveAlwaysStatEntity); //ensures these counters will be written despite user options } Software Week, 9th Oct 2009

16. Conclusions • The XMLSummary is ready for deployment • Python code can be incorporated into Ganga/Dirac • Documentation already exists, and a tutorial will follow • Luminosity Counters need to be declared separately! Software Week, 9th Oct 2009

17. End Software Week, 9th Oct 2009

18. Ganga Ideas • GangaLHCb will depend on LHCb/XMLSummaryBase • Helper methods will already exist in the “Summary” class • Extra functionality can be added through inheritance XSD Ganga Job j.application.summary() (don’t store in job) Inherited ganga (GPI) object Inherited ganga object Helper methods isFailure() Merge([s,s’]) Python schema (me) Python parsing (xml.etree) XML Gaudi/XMLSummary Software Week, 9th Oct 2009