Maximize your calibration event rate with partial events

1. Maximize your calibration event rate with partial events Ignacio Aracena (SLAC) SLAC ATLAS forum March 18th 2009

2. Outline • Introduction • Motivation • The ATLAS detector • Calibration requirements • The ATLAS TDAQ system • Partial events for calibration • Calibration with partial events • Inner detector alignment • LAr calibration • Tile calibration • Test results • Rate studies with partial events • Open issues • Summary

3. Motivation Physics at the LHC: New physics with (relatively) high pt at few Hz vs SM physics at ~kHz • ATLAS trigger strategy for new physics: • Improve physics object selection at three trigger levels • Region of Interest (RoI) minimizes data requests/shipping through network L1 out 100kHz L2 out 3kHz 5GB/s EF out 200Hz 300MB/s Level 1 : Identifies RoI to seed L2 @ 100kHz Level 2 : Improves RoI eta, phi, pT by using high detector granularity, accept event rate ~3kHz Event Filter : full event data available for further improvement, output rate ~200Hz, i.e. 300MB/s to permanent storage for offline analyses. Trigger architecture driven by physics programme How much trigger rate can we afford for calibration events during physics run?

4. ATLAS subdetectors EM calorimeter: LAr calorimeter Hadronic calorimeter: Tile calorimeter (barrel) LAr calorimeter (endcaps) Tracking : Pixel, SCT, TRT Muon spectrometer TGC/RPC (trigger chambers) CSC/MDT (precision chambers) The ATLAS detector Precise and fast calibration of different subdetectors is crucial for achieving performance goals

5. Event data volume composition • Event data in terms of Readout buffers (ROBs) requested by TDAQ: • LAr 762 ROBs • Tile 64 ROBs • Pixel 132 ROBs • SCT 86 ROBs • TRT 192 ROBs • Muons ~400 ROBs • Total ~1600ROBs • 1ROB ~ 1kB • Full event with data of all the ROBs is ~1.5MB (about 50% from LAr...) How much data is needed for calibration of subdetectors?

6. Calibration requirements • Inner detector alignment • Need 6-8 M isolated tracks in 6 hours for pixel/SCT and in 24 hours for TRT • Requires ~50Hz of isolated tracks • LAr calibration • Study pulse shape of individual cells • Requires ~5 Hz to achieve precision of <1% • Muon spectrometer • Need to collect ~106 muons/day for MDT t0 calibration • ~1kHz of muon tracks • Tile calorimeter • Illuminate full calorimeter with dedicated tile calibration system, run during empty bunch crossings • New calibration types?

7. L1 out 100kHz Muon spectromter calibration ~1kHz ≈ L2 accept rate L2 out 3kHz 5GB/s Inner detector alignment ~50Hz 25% of EF output rate EF out 200Hz 300MB/s LAr calibration ~5Hz 3% of EF output rate Calibration vs Physics Can we take calibration data during physics runs? Calibration rate appear to be not matching the rate limits of TDAQ design scheme at first glance!

8. Level1 : If RoI found, all detector data is kept in the Readout buffers (ROBs) of the Readout system (ROS). Level2 : Requests data from RoI, with full detector granularity. If event is accepted, the event is fully built Event builder : For L2 accepted events the full event data from the ROS is pulled to build the full event. Event Filter : Receives full event, which can be used for further improvements. Sends accepted events to the data logger (SFO) SFO : Writes events into different data streams, sends data to offline mass storage The ATLAS TDAQ system ATLAS TDAQ system is based on: L1 100kHz → L2 3kHz → EF 200Hz full event size ≈ 1.5MB, RoI size ≈ 3% of full event hardware 100 kHz ReadOut Systems (ROS) Readout buffers (ROBs) Event dataflow L1 RoI 3 kHz L2 Event builder (SFI) 3 kHz EF 200 Hz Data logger (SFO) 200 Hz software Tier0

9. The TDAQ system ATLAS TDAQ system is based on: L1 100kHz → L2 3kHz → EF 200Hz full event size ≈ 1.5MB, RoI size ≈ 3% of full event hardware 100 kHz ReadOut Systems (ROS) Readout buffers (ROBs) Network requirements: bandwidth = event rate * event size Max. bandwidth is fixed by the system installed at P1 L1 RoI 3 kHz L2 Event builder (SFI) (5600MB/s) 3 kHz Calibration events should use small fraction of total bandwidth need high trigger rate don’t need the full event! EF (300MB/s) 200 Hz Data logger (SFO) 200 Hz software Can increase calibration event rates for same bandwidth budget by using partial events! Tier0

10. Partial events for calibration • Select calibration event at Level 2: • Fill list of detector (ROB) identifiers needed for a specific calibration purpose. • In the event builder (SFI) use this list to pull out data only from identifiers in this list and buildpartial event • Send partial event to SFO, no need to go through EF. hardware 100 kHz ReadOut Systems (ROS) Readout buffers (ROBs) L1 RoI 3 kHz L2 partial event building Event builder (SFI) • Select calibration event at the Event Filer: • All events that are processed in the EF are fully built • Fill list of detector (ROB) identifiers needed for a specific calibration purpose. • In the SFO use this list to extract only data fragments from listed ROB identfiers • Create a partial event from the full event : event stripping, ship and store only partial event (5GB/s) 3 kHz EF (300MB/s) 200 Hz Data logger (SFO) 200 Hz software event stripping Tier0 For an allocated bandwidth partial events allow higher trigger rates of calibration events, i.e. partial events save bandwidth at the SFI and SFO

11. Calibration + physics events Overlapping events: events that are selected by calibration triggers and physics triggers • Logic to recover the partial event for calibration implemented • If only overlap at L2, apply event stripping in the EFD • Allows further save of bandwidth in the EFD • If overlap at EF, apply event stripping at the SFO EFD = Event Filter data flow program PT = EF processing task

12. Calibration requirements and partial events • Inner detector alignment • Need 6-8 M isolated tracks in 6 hours for pixel/SCT and in 24 hours for TRT • Requires ~50Hz of isolated tracks • partial event build • LAr calibration • Study pulse shape of individual cells • Requires ~5 Hz to achieve precision of <1% • event stripping • Muon spectrometer • Need to collect ~106 muons/day for MDT t0 calibration • ~1kHz of muon tracks • partial event building? Muon group has developed a standalone “TDAQ” system to extract muon tracks. Less flexible than using partial event from TDAQ • Tile calorimeter • Illuminate full calorimeter with dedicated tile calibration system, run during empty bunch crossings • partial event building

13. Inner Detector alignment • Requirements • Get high rate O(50Hz) over wide pT range • Chains in current Lumi1E31 menu (athena 15.0.0): • trk9i_id, prescale 40 • trk16i_id, no prescale • Route to “calibration_IDTracks” stream Fill list of ROBs at L2 use case for partial event building (Anna Sfyrla, Carlos Escobar)

14. LAr calibration • Requirements • Sufficient rate (5Hz) given by photon triggers into EF, no need for additional rate using partial EB • Minimize bandwidth at the EFD/SFO • Chains in current Lumi1E31 menu (athena 15.0.0): • g3_larcalib (PS 10^6) • g10_larcalib (PS 100) • g20_larcalib (PS 1) • g20i_larcalib • j10_larcalib (PS 10^6) • j80_larcalib (PS 10^6) LVL1 EM RoI LVL2 photon selection EF photon selction event stripping use only ROBs inside RoI Fill list of ROBs only at EF selection use case for event stripping (Nicolas Berger, Guillame Unal, Isabelle Wingerter-Seez)

15. LVL1 CALREQ LVL2 fill list with all tile ROBs partial event build Tile calibration • Tile calorimeter has dedicated calibration system: • Laser injection (timing studies) • Charge injection (pulse shape) • which illuminates the full tile calorimeter • Used during emtpy bunch crossing • Chains used in cosmic trigger menu • TileCalib_laser • TileCalib_cis Trivial use case of partial event with a complete subdetector (Andrea Dotti, Oleg Solovyanov)

16. Tests with AtlasHLT-14.5.0 • Setup multihost partition on lxplus with AtlasHTL-14.5.0 (tdaq-02-00-00) • 2 L2 segments (L2SV, L2PUs), • 1 EBEF segments (2 SFIs, 1 SFO, 1 EF segment, 4PTs) • 1 ROS segment (8 ROSs) • Data sample : /pcatr-srv1/data/files/aagaard/14.5.0/enhancedBias10TeV/BS/ (1200 events) • “enhanced bias” = L1 preselection EM3, MU4, TAU5, J18, FJ18, XE25, TE250 • preload the data on ROS (ROS application running on lxplus node) • Lumi1E31 menu • testing the ID and LAr calibration streams • Recorded monitoring histograms after ~1hour run • Monitoring plots from this test on the following slides

17. Partial event building performance – event size Size of built events in the SFI Event size vs streams in the SFI overlapping events (calib + phys) need to be stripped Full event size ~2MB (MC data) Partial event ~50kB (~3% of full event) partial events for ID alignment About 30% of ID alignment events overlap with physics events Overlap fraction can be tuned with the menu Impact on resources? See next slide

18. Partial event building performance – rates Contribution to event rate at the SFI Bandwidth contribution at the SFI stream X event build rate / total event building rate stream X event rate / total event rate overlapping events (calib + phys) need to be stripped ID alignment event rate comparable to physics event rates Pure ID alignment events use very small fraction of total bandwitdh Partial event building minimizes the bandwidth fraction used for calibration events

19. Event stripping performance – event size Event size vs streams in the SFO (full events from calibration_LArCells, bug in menu) Full event size ~2MB (MC data) Partial event ~50kB (~3% of full event) (ignore calibration_LArCells, bug in menu) stripped events for LAr calibration

20. Event stripping performance – event size Event size vs streams in the SFO (full events from calibration_LArCells, bug in menu) ID alignment events not stripped bug, fixed in 15.0.0 Full event size ~2MB (MC data) Partial event ~50kB (~3% of full event) (ignore calibration_LArCells, bug in menu) stripped events for LAr calibration

21. Event stripping performance – rates Contribution to event rate at the SFO Bandwidth contribution at the SFO stream X event build rate / total event building rate stream X event rate / total event rate LAr calib event rate comparable to physics event rates LAr calib events use very small fraction of total bandwitdh Event stripping minimizes the bandwidth fraction used for calibration events

22. Examples from cosmic data taking runs • Previous slides showed only tests with MC data • Tile calibration and ID alignment stream exercised during cosmic data taking runs in following slides

23. LVL1 CALREQ LVL2 fill list with all tile ROBs partial event build Tile calibration – example from cosmic • Exercised during cosmic runs, example here from run 92082 • Full event size ~5MB (LAr requesting 10 samples) • Tile calibration event size ~200kB, i.e. ~1% of full event • No rate monitoring plots for this run (Atlas release 14.2.23), but with given event size uses only ~1% of bandwidth • Event successfully used for calibration partial event ~200kB full event ~5MB Number of events Event Size (kB)‏

24. Inner detector alignment with cosmics • Use chains developed for cosmic data (e.g. CosmicsAllTeTRTxK*, CosmicsAllTeSiTrack*, CosmicsAllTeIDScan*) and apply PEB on those • … CosmicsAllTe chains do not have ‘well defined’ RoIs • … chains that do tracking at the TRT are not good for testing since TRT does not provide η measurement. • Modify the TrigROBListWriter.cxx to loop over the tracks and get η-φ range from the tracks • Apply modified TrigROBListWriter.cxx to two chains: • CosmicsAllTeIDSCAN_AllPhysics_TrkHypo and • CosmicsAllTeSiTrack_AllPhysics_TrkHypo • Generate new cosmic menu that contains these new chains, and run with this menu on cosmic data. • The generated raw data will contain information from the ID only and for the η-φ range that is defined by the modified TrigROBListWriter.cxx • Compared raw data generated with and without the PEB applied in the chains Run 90943 Partial events gives same results as full event data (Anna Sfyrla, Carlos Escobar)

25. HLT rate studies with partial events • Use partial event to store only trigger decision bits from L1/L2/EF • Allows high statistics rate studies • https://twiki.cern.ch/twiki/bin/view/Atlas/BeatenbergActions#General_Action_Items , see “T4” • Could also be combined with monitoring information for TDAQ performance tests • Basic TDAQ infrastructure ready. Need to configure menu “chain”, study the impact on the resources

26. Open issues • Only one list of ROBs (subdetectors) per event. List is union of all partial events. Results in overlapping ROBs (e.g. LAr ROBs showing up in ID alignment events) • Tests on MC show no overlap between LAr calibration and ID alignment. More studies needed

27. Open issues • Only one list of ROBs (subdetectors) per event. List is union of all partial events. Results in overlapping ROBs (e.g. LAr ROBs showing up in ID alignment events) • Tests on MC show no overlap between LAr calibration and ID alignment. More studies needed • Bandwidth budget per sub-detector calibration? • Need to establish rules/policy • Needs to be discussed across sub-detector groups and TDAQ group

28. Summary • Calibration with partial events • Using partial events for calibration of sub-detectors allows to increase the event rate for a given bandwidth budget • Flexible system that allows calibration data taking with all subdetectors • Being used by LAr, ID, Tile • Partial event already tested and used at P1 during technical runs, cosmic data taking • Muon MDT t0 calibration? Standalone system does not offer flexibility of partial events. Next customer on the list • Potential new use case discussed in Beatenberg • Partial event build with only trigger decision bit information for trigger rate studies • New calibration types? • Advantage of partial event is flexibility, can accommodate all sub-detectors

29. Outlook • Run test at P1 • Uses real system • Larger statistics available • Can re-play cosmics and/or pre-load MC data • Implement monitoring • Rate plots shown here not implemented yet in official SFI/SFO tags • New monitoring plots?