Detector (Tracker) monitoring: introduction to COSINE and the Filter Farm Giacomo BRUNO

1. Detector (Tracker) monitoring: introduction to COSINE and the Filter Farm Giacomo BRUNO UCL Louvain Tracker Monitoring Workshop – 3 September 2004

2. COSINE links Web : http://cmsevf.web.cern.ch/cmsevf/ Mailing list: cms-daq-eventfilter@cern.ch Responsibles Detector monitoring with COSINE G. Bruno

3. Outline • Introduction to COSINE • Special ORCA features when running online applications • Monitoring scheme foreseen for the filter farm • Prototype monitoring tool in COSINE Detector monitoring with COSINE G. Bruno

4. Filter farm DAQ and Filter Farm Detector readout Event building Event filtering • Being components of the DAQ, the Filter Units (FUs) run a XDAQ application (intra-node communication, control, configuration…) • For the filtering task the FUsuse the same reconstruction software as the off-line analysis: ORCA Detector monitoring with COSINE G. Bruno

5. Bringing Reconstruction online • On-line software (XDAQ) • Basic core executive ( intra-application communication, memory management, control, configuration) • Offline software (ORCA, COBRA) • Reconstruction code • HLT steering algorithms • Data persistency • Integration of 1) and 2) has been accomplished by means of • extensions of the off-line software • new software integration environment Detector monitoring with COSINE G. Bruno

6. Integration software project Offline World OnlineWorld COBRA Daq Prototype XDAQ COBRA ORCA *Daq* ORCA COSINE Dependency Detector monitoring with COSINE G. Bruno

7. COSINE • Consistent Online Software Integration Environment • afs location: /afs/cern.ch/cms/Releases/COSINE • CVS/SCRAM managed • Latest release: 1_0_0_pre5 • External tools: XDAQ, ORCA, COBRA, Geometry.. • Basic software: • FU implementation, • SubFarm Manager implementation: configuration, control, monitoring (first prototype based on CDF model) Detector monitoring with COSINE G. Bruno

8. Modes of operation Object DB Standard Offline application ( Sim/Rec Application) exchange File Object DB 1. New Offline ORCA application: simulated raw data production exchange File 2. New Offline ORCA application: test beam data analysis, data format studies Object DB 3. New Online COSINE application: CMS filter farm, test beams, local daq Event Builder FU Object DB Detector monitoring with COSINE G. Bruno

9. GbE DSN LTC Service PC(s) Event building • The full event information is made available in ORCA as a collection of FED raw data fragments, irrespective of the details of the hardware components involved in the data flow. • The event building software (http://cmsdoc.cern.ch/cms/TRIDAS/DAQkit/doc/html/index.html) is fully configurable. Example: there can be a single FED-builder, a single RU and a single BU application, all running on the same machine (interesting for local DAQ) Detector monitoring with COSINE G. Bruno

10. Sub-detector specific tasks • Setup description (xml geometry file) • Dimensions and location in space of the detectors (DetUnits) • Readout description/partitioning (“DetPartitioner” class) • Connection between Detectors and FEDs (perhaps this information should be in the xml geometry files, in the configuration DB …?) • Grouping of the sub-detector FEDs into “RecoRUs” (logical software construct): subject to optimization • Transformation of FED raw data into DIGIs (optionally also their delivery from the BU to the FU) are “on demand” (reconstruction driven). • Persistent data are organized into collections of DIGIs associated to the RecoRUs • Raw data – DIGI conversion (“DataFormat” package) • Application-specific code in the form of a standard ORCA user analysis class (Observer<G3EventProxy *> …): • HLT event selection code for CMS • Test beam data analysis • Monitoring task: booking and filling of histograms Detector monitoring with COSINE G. Bruno

11. RecoRUs-FEDs-DetUnits association Det group 2 Det group 1 Det subgroup1 Det subgroup2 Det subgroup1 Det subgroup2 FED3 FED4 FED1 FED2 RU1 RU2 creates SubDetectorSetUp Det Partitioner Geometry • Dets-FED association: must reflect the actual hardware connections • FEDs-RecoRU: logical software construct subject to optimization Detector monitoring with COSINE G. Bruno

12. Online reconstruction working scheme ORCA + COBRA Data source/destination Reconstruction algorithm D1 Detectors FED 1 D2 Standard or Daq Readout Unit RecoRU1 FED 2 Formatter String (RU name) FED formatter POOL DB algo OODB FED 3 Raw data source (BU/file) int (FED ID) RecoRU2 FED 4 BU FED formatter Detector monitoring with COSINE G. Bruno

13. Current default (CMS geometry) implementation • RecoRUs • 77 SiStrip, 8 Pixel (determined by a division in phi by groups of 200 DetUnits) • FEDs • 440 SiStrip, 38 Pixel (numbers from the DAQ TDR) • Connection to DetUnits: arbitrary, but balanced repartition. • Association to RecoRUs: balanced repartition. Detector monitoring with COSINE G. Bruno

14. The Silicon Strip case • Silicon Strip FEDs can be readout in zero suppressed (ZS) mode or non-ZS mode, concurrently. ZS FED1 ZS FED2 Non-ZS FED3 Non-ZS RecoRU ZS RecoRU ZS Formatter Non-ZS Formatter creates creates Silicon Strip SetUp Geometry Still to implement: persistency for the NON-ZS DIGIs Detector monitoring with COSINE G. Bruno

15. Data formats • Arbitrary and unrealistic data formats released since a long time for both Pixel and Silicon Strip sub-detectors • Silicon Strip: data – formatter for test beam data analysis from Root files (M. Pearson, I. Reid, I. Tomalin) • Pixel: work started for the final CMS data format (M. Konecki, D. Kotlinsky) Detector monitoring with COSINE G. Bruno

16. Test beams applications The framework allows interchange of geometries and data formats. An obvious advantage of this feature is the immediate extension to test beam applications Silicon modules In 2003 attempt to have an ORCA-based monitoring (when COSINE did not exist..) • Implementation with patched code (test beam “official” FU was pushing events into the ORCA-based FU) • Failed due to different versions of the same library used by ORCA and XDAQ FEDS PC (FEDS in PCI slots) Builder Unit PC Monitoring info OO DB ORCA FU Filter Unit PC Light-weight monitoring Root data file Detector monitoring with COSINE G. Bruno

17. 2004 test beam Silicon modules • Direct use of the COSINE-based FU in the test-beam setup has been proved to be possible (FED emulator tests), though minor modifications to the code were needed • Due to lack of manpower the data formatter is still not ready FEDS PC (FEDS in PCI slots) Builder Unit PC OO DB OO DB ORCA monitor COSINE Filter Unit PC Root data file Online monitor Detector monitoring with COSINE G. Bruno

18. Open issues Short term (being addressed): • Final porting to ORCA/src/CommonDet of the remaining code common to all sub-detectors • Functionality extensions, minor bug fixes • Memory leak when running track reconstruction in the FU • Allow persistency of NON-ZS silicon strip digis • Prove smooth running of a COSINE-based FU within the silicon strip test beam DAQ through the FED emulator (need to complete the data formatter) Longer term: • Documentation • Release of realistic CMS data formats Detector monitoring with COSINE G. Bruno

19. Monitoring The following slides are mostly based on the presentation by E. Meschi at the Event Filter meeting during the May 2004 CPT week (http://agenda.cern.ch/askArchive.php?base=agenda&categ=a041651&id=a041651s6t1/transparencies) Detector monitoring with COSINE G. Bruno

20. Monitoring: Global vs. Local DAQ Global DAQ: • First access to the full event data: in the FU • It should be possible to observe and analyze any given subset of events accepted by the Level 1 • The FU runs an ORCA application where access to geometry, calibrations, conditions is guaranteed The FU is a natural candidate to run monitoring tasks Local DAQ: • Fundamental rate/throughput limitations • In principle lots of freedom as to: • Choice of processing and analysis framework • Feedback scheme and access to central information repositories • Clients Could profit from implementing the monitoring tasks in the software framework of the global DAQ Detector monitoring with COSINE G. Bruno

21. Options for Online Monitors • Every FU is able to run certain monitor tasks: • Monitor "algorithms" need to be maintained on each FU, and handled like an HLT algorithm • Need an API (not necessarily different from the DAQ monitoring) to publish and update complex monitor objects to a central server • Every FU is able to forward specific event types to dedicated processing nodes (either directly or through a concentrator): • One more interface in the FU • More network traffic, more complex topology (but need not be 100% reliable. At worst consumers can be ranked by relevance) • What protocol for event forwarding ? See this in relationship to event storage • Why might BOTH 1 and 2 needed ? • Some monitor process may need to see events at large rate • An obvious example is HLT monitoring • A lot of processing is already performed on the farm Detector monitoring with COSINE G. Bruno

22. Storage Manager and Consumers BU To tier0 (events?/files?) FDN1 ”Quasi-online Monitor” FU Storage manager FCN SM Server FU Reco Server Abstract Layer "Monitor Consumer" "Event Consumer” Registry Publish SM Streaming Communication FCN Detector monitoring with COSINE G. Bruno Server

23. Monitor Task Location Requires > (say) 1% FU resources (CPU/mem) yes yes > (say) ~10 Hz (> 10 MB/s aggregate) ??? no no yes low latency (0.1÷1 min) no Requires > (say) 1% FU resources (CPU/mem) yes On local mass storage yes Requires < ~hours latency no On tier 0 Dedicated processor off online stream On FU no Detector monitoring with COSINE G. Bruno

24. Monitoring tool in COSINE • Prototype ROOT-based tool for monitoring already present in DaqPrototype + COSINE (based on the CDF monitoring model): COSINE_1_0_0_pre5/src/SubFarmManager/ COBRA/DaqPrototype/DaqMonitor • Directly applicable to tasks where histograms are filled by the FUs • A server application collects the histograms from several FUs and presents them • A GUI allows histograms to be visualized, updated, printed…. • No documentation yet, but use is quite straightforward Detector monitoring with COSINE G. Bruno

25. Exercises • Try to follow something like flowchart above (with numbers) for each task to: • Define rate and resource budget for online monitors (similar to HLT) • Define local storage needs (in addition to normal event buffering) • Examine current design of the FU physics monitor tool in light of possible additional uses. • Can we propose a uniform framework for all use cases (maybe even including local DAQ?) • In connection with DAQ monitoring (currently also in the requirement collection phase, see http://cmsdoc.cern.ch/cms/TRIDAS/distribution/Meetings/DAQ.weekly/0DW2004/SRSMON_V1_0.pdf Detector monitoring with COSINE G. Bruno

26. BACKUP SLIDES Detector monitoring with COSINE G. Bruno

27. Global vs. Local DAQ • Global DAQ • Trigger uses GTP: GTP trigger distributed to FEDs via TTC • FED / GTP readout via Slink into FRL • FED builders assemble “superfragments” • RU builders in Surface-Bld: full events assembled in BU from superfragments • EventFilter farm in Surface-bld: complete event record available in FU • Intended for physics data taking • Throughput: 200 MB/s per FED, or 12.5 kHz for 2kB fragments per RU builder, or 200 events/s per BU • Can operate up to 8 partitions in parallel • Local DAQ • Trigger uses LTC: LTC inputs distributed to FEDs via TTC • FED readout via VME into crate controller PC • RU: crate controller PCs. Also BU / FU / monitor if no event building • Event building over DSN (GbE) FU / monitor on network PCs • Throughput: something like 20 MB/s per FED crate, few 100 Hz rates for 2kB fragments (less if events built) • Operation completely independent of central DAQ and central Trigger (apart for central services such as boot, file servers, databases ..) Detector monitoring with COSINE G. Bruno

28. Examples • Farm specific monitoring (HLT performance, rates etc., control plots like geographic distributions of objects etc. for rejected/accepted events): on FU • SM collects, summarizes and serves directly monitor objects (histograms etc.) to consumers • Event display: use event forwarding (?) • Most calibration tasks which can’t run on tier 0 should qualify to run off data buffered locally (provided they don’t require a kHz) • And even then, might be only parts of an event so actual bw limited (how much in total ?) say want to allow another 100 MB/s in addition to data? • These data have a short life cycle, they never make it as such to tier 0 and/or databases, only summary information persists Detector monitoring with COSINE G. Bruno