Trigger, data acquisition and control

1. Trigger, data acquisition and control CERN

2. Foreword • The subject of this course is not an exact science! The challenges are often similar but the aim and the requirements of a particular experiment may be unique. Thus the solutions vary enormously. • While I will try to give a general view of the strategies adopted, I will use the LHC experiments as examples. • This course presents the work of many people that have been developing trigger, data acquisition and control system for high energy experiments over many years! To prepare this course I have received input and comments from many sources. In particular thanks to: • Clara Gaspar, CERN • Sergio Cittolin, CERN • Jean-Pierre Dufey, CERN • Irakli Mandjavidze, Saclay

3. Introduction Setting the scope Rates at collider experiments Basic concepts Multi-level trigger systems and readout structures Front-end electronics General architecture Digitizers Signal processing Trigger principles and implementations Setting the scope Trigger basics Trigger design and performance Fast and synchronous hardware triggers Software triggers Experiment timing Timing and fast control systems Event readout Readout networks (buses, switches, etc) Event building and online farm processing (event filters) Data storage Configuration, control and monitoring Operating modes Run control Data monitoring and quality control Conclusions Course outline

4. 1. Introduction

5. Setting the scope Detector Accelerator Trigger and DAQ Trigger system Clock Analog signals Decisions Data acquisition system Design feedback Mass storage Raw data Detector & trigger simulation Reconstruction & analysis Physics results DAQ = Data AcQuisition

6. Trigger and DAQ • Trigger System: • Selects in real time “interesting” events from the bulk of collisions • Decides if YES or NO the event should be read out of the detector and stored • Data AcQuisition System • Gathers the data produced by the detector and stores them (for positive trigger decisions) • Front End Electronics: • Receive detector, timing and trigger signals and produce digitized information • Readout Network • Reads front end data and forms complete events (sometimes in stages) - Event building • Processing farm and storage • Stores event data, can do data processing or filtering

7. Detector Channels Trigger Front End Electronics Timing and Fast Control Readout Network Control and Monitoring Processing/Filtering Storage Experiment control Lecture 1

8. Context diagram • The main role of trigger and DAQ is to process the signals generated in the detector and write the information onto data storage, but: Accelerator Detectors Database Timing Accelerator status Detector status Raw signals Detector, readout descrip. constants Trigger & DAQ System Information Event data Data Store Conditions Experimenter Status Settings Experiment Control

9. LEP : e+e- crossing rate 45 kHz 22 ms SPS : pp crossing rate 260 kHz 3.8 ms Tevatron : pp crossing rate 2.5 MHz 396 ns HERA : ep crossing rate 10.4 MHz 96 ns LHC : pp crossing rate 40 MHz 25 ns Bunch crossing rates • LHC • ATLAS • CMS • LHCb • ALICE (Pb + Pb) 106 – 108 electronics channels LHC

10. 2. Basic concepts

11. Trivial DAQ External view • How does the ADC know when to make the conversions? Sensor Physical view CPU Sensor ADC Card Disk Logical view Processing Storage ADC

12. Trivial DAQ with real trigger • What if a trigger is produced when the ADC or Processing is busy ? Sensor Trigger Delay Discriminator Start ADC Processing Interrupt Storage

13. Set Q Clear Trivial DAQ with real trigger • Deadtime (%) : the ratio between the time the DAQ is busy and the total time • Efficiency (%) : (1- deadtime) Sensor Trigger Discriminator Delay Busy Logic Start ADC NOT AND Processing Interrupt Ready Storage

14. Trivial DAQ with real trigger Sensor Trigger Discriminator Delay Busy Logic Start ADC Full FIFO DataReady Processing • Buffers to de-randomize data • Decouple data production from data consumption • Better performance Storage

15. Set Q Clear Derandomizer buffers - Queues Sensor Sensor Trigger Trigger Discriminator Discriminator Delay Delay Busy Logic Start Busy Logic Start and ADC ADC Full FIFO Processing Interrupt DataReady Processing Ready • Busy during ADC conversion time + processing time • Busy during ADC conversion time if FIFO not full Storage Storage Output rate No deadtime 1%/Hz 5%/Hz Input rate

16. Queueing theory • For simple cases, the behaviour of the system can be calculated analytically using queueing theory. Soon you have to use simulation techniques. Arrival time following exponential distribution (Poisson process) Efficiency 1.0 FIFO Depth 1, 3, 5 5 buffers 3 buffers 0.5 1 buffer Process Processing time may also depend on event… 1.0 2.0 <Processing time>/<Input period>

17. Less trivial DAQ N channels N channels N channels Trigger ADC ADC ADC Processing Processing Processing Event building Processing • Parallel processing and event building Storage

18. Trivial DAQ at a collider Timing Sensor BX Beam crossing Start ADC Abort Trigger Discriminator Busy logic Full FIFO DataReady Processing • Timing : We know when a collision happens • Fully synchronous operation up to Processing • Trigger is used to “reject” the data Storage

19. LEP collider timing • Level 1 trigger latency < inter bunch crossings -> No “L1 deadtime” • No event overlapping • Most of the electronics outside the detector e+e– crossing rate 45 kHz (4 bunches) 45 kHz 22ms Level 1 6 ms Level 2 100 Hz 30 ms Readout 10 Hz 8 ms Level3 8 Hz 100 ms

20. LHC collider timing pp crossing rate 40 MHz (L = 1033 - 4×1034cm-2s-1) 25 ns 40 MHz Level 1 O(ms) 100 kHz Level 2 » ms 1 kHz Level n • Level 1 trigger time exceeds bunch interval • Event overlap and signal pileup (multiple crossings since the detector cell memory greater than 25 ns) • Very high number of channels

21. Trivial DAQ at LHC Timing Sensor BX Beam crossing Clock Analog Pipeline Pipelined Trigger Accept/Reject ADC Busy logic Full FIFO DataReady Processing • Front-end pipelines • Waiting for trigger decision + transmission delays are longer than beam crossing period Storage

22. LHC readout structure Detector 40 MHz Trigger level 1 Pipelines ADC, Zero Suppression Formatting 100 kHz Trigger level 2 Buffers Event Building 1 kHz Trigger level 3 Buffers 100 Hz

23. 3. Front-end electronics

24. Front-end sequence • Detector dependent (Home made) • On Detector • Pre-amplification • Discrimination • Shaping amplification • Multiplexing of a few channels • Transmission • Long cables (tens of metres), copper or fibre-optics • Near the detector or in the counting rooms (hundreds of FE crates) • Reception • Analog buffering • A/D conversion • Digital buffering Example: Charged particle creates ionization drifted by electric field onto sensor wire ADC Rec ShAmp PreAmp

25. Vin sample/ hold comparator Shift register Vref DAC Digital Output Analog to digital conversion • Digitizing means measuring something (charge, amplitude, time, ...) comparing it with a reference unit. Ex. Successive approximation ADC Compare entity with a series of rulers in sequence (standard ADC, counting) or in parallel (flash ADC) Entity to be measured Ruler unit Try various output codes(binary search) by feeding them into a DAC and compare with the analog input. #bits Not very fast Usually 10/12 bits Time

26. V in V ref Clock R #1 Encoder N to Log2 N bits R #2 R #3 Log2 N bits R #N Differential comparators Faster: Flash ADC Flash ADC, resolution N #bits Power • FADC • The fastest method • Difficult > 10 bits (12 bits)

27. Front-End structure Detector Amplifier Filter Shaper Range compression Sampling clock Digital filter Zero suppression Buffer Feature extraction Buffer Formatting & Readout

28. Amplitude (V) Time (ns) Ex. beam phase monitoring Signal from beam pick-up Measure phase between bunch clock and beam arrival (TDC) Coefficients RAM Amp LPF Pulse detect Threshold Linearization Block ADC Flip Flop 12 bit BCLK Histogram Calculation Block Delay Line 3 Delay Line 4 RAM FPGA

29. Ex. beam phase monitoring • Simulation (PSPICE) Input Flip-flop Integrator BCLK ADC sampling

30. FE challenges at LHC • Small bunch interval • Pipeline buffering (analog) to keep events until trigger decision (several ms) • Very fast A/D conversion • Very precise timing distribution (order of sub-ns) • Large #channels • High integration (Custom VLSI chips) • Piled-up Events • Digital Signal Processors (DSP) to sort them out • Large amount of data • Data compressors (instead of zero skipping) • Power consumption • Heat dissipation • Radiation levels • Rad-hard or tolerant technologies…. • Limited access • Fault tolerance and redundancy

31. Front-End at LHC Detector channels 106 - 108 Rate 40 MHz Analog pipeline 100 – 1000 evts L1 trigger (1-10 ms) 100 KHz – 1 MHz MUX-ADC-DERANDOMIZERS MUX-ADC DERANDOMIZER L2 trigger (10 ms- ms) DSP Digital buffer 10 – 60000 evts 1 - 40 kHz ~ 1-100 Gbytes/s Readout Data links (100 – 1000) Event builder L2/L3 trigger (Event filter) CPU CPU CPU 100 Hz ~ 1- 100 Mbytes/s CPU CPU CPU CPU CPU CPU

32. 4. Trigger

33. Setting the scope • Inelastic cross-section at LHC (14 TeV) : s = 70 mb (70 * 10-27 cm-2) • Luminosity : L = 1034 cm-2 s-1 = 107 mb-1 Hz • Event rate = L * s = 7 * 108 Hz • 40 MHz bunch crossings (25 ns) • Events/25 ns ~ 20 simultaneous events/crossing! LHC

34. Looking for the needle CMS experiment ATLAS experiment

35. Looking for the needle Event rate : ~ 109 Hz Event selection : ~ 1/1013

36. Trigger system The trigger system is the system which triggers the recording of interesting data in an experiment The trigger is a function of: T( ) REJECTED ACCEPTED Event data & Apparatus Physical channels & Parameters Since the detector data are not promptly available and the function is highly complex, T(...) is evaluated by successive approximations called: TRIGGER LEVELS

37. Trigger acceptance Trigger principle • Interesting (signal) and uninteresting (background) events overlap in the space spanned by the discriminating variables • Trigger selection compromise • Find criteria that select as much as possible of the signal • To maintain a good signal efficiency, background is unavoidable Phase space Uninteresting Interesting

38. Trigger design - e+e- vs. pp • Lepton collider (e+e-) “Precision machine” • sinteresting ~ s tot (All interactions are interesting) • No physics rejection needed (After Level-1 trigger: 0.1% rejected, 99.9% accepted) • Background rejection is crucial to perform precision measurements: • Cosmics, beam-gas, off-momentum e+/-, detector noise • Criteria • Very good efficiency for selected channel ~100% • Good background rejection ~10 • Good monitoring of efficiency ~0.1% • Hadron collider (fixed target or pp) “Discovery machine” • High rejection of physics events • Physics rejection! (After Level-1 trigger: 99.9% rejected, 0.1% accepted) • Criteria • Good efficiency for selected channel > 50% • Good rejection of uninteresting events > 106 • Good monitoring of efficiency ~0.1%

39. Trigger design • Search for physical criteria to select event candidates from background processes • Track multiplicity • Direction of particles • Particle momenta, at LHC in particular transverse momenta • Coplanarity of event • Particle identification • Deposited energy • Missing energy • Invariant mass • Interaction point / secondary vertices • … • Simulation studies • It is essential to build into the experiment's simulation the trigger simulation. • For detector/machine background rejection • Try to include detector noise simulation • Include machine background (not easy). • For example at LEP the trigger rate estimates were about 50-100 times too high. • For physics background rejection, the simulation must include the generation of these backgrounds

40. Ç N ( A B ) N ( B ) Trigger monitoring • “A 60% well known efficiency is much better than a 90% uncertain efficiency” • Efficiency monitoring • Key point: redundancy (independent) • Each event should fire at least two independent sub-triggers (the global trigger is a logic OR of all sub-triggers) • Use the simulation (or common sense!) to evaluate for different sub-triggers how redundant they are. • Use data (after reconstruction) to compute efficiency. • From probability for independent events • P(A Ç B) = P(A) * P(B) Ex: Electron trigger efficiency A = Calorimeter A B = Tracking A Ç B B eA =

41. Trigger levels • Since the detector data is not promptly available and the trigger function is highly complex, it is evaluated by successive approximations. • We need to optimize the amount of data needed and the time it takes to provide a decision. • Compromise between amount of data needed, time to obtain it and trigger complexity (processing power) • Trigger levels • Hardware trigger: Fastsynchronous trigger which uses coarse data from few detectors and has normally a limited time budget and is usually implemented using hardwired logic. Level-1 (Level-0) sometimes Level-2 • Background rejection ~10 000 • Software triggers: Several trigger levels which refines the crude decisions of the hardware trigger by using more detailed data and more complex algorithms. It is usually implemented using processors running a program.Level-2, Level-3, Level-4, ...

42. Trigger levels at LHC 10-7 s Collision rate 40 MHz Level-1 selected events 100 kHz – 1 MHz • Particle ID (high PT, e, m, jets, missing ET) • Local pattern recognition • Energy evaluation based on prompt macro-granular information Level-2 selected events 1 kHz – 40 kHz • Clean particle signature (Z, W, …) • Finer granularity and precise measurements • Kinematics, effective mass cuts and event topology • Track reconstruction and detector matching • Vertex detection Level-3 events to tape ~100 Hz • Physics process identification • Event reconstruction and analysis 10-6 s 10-3 s 100 s

43. Detector Channels Trigger Front End Electronics Timing and Fast Control Readout Network Control and Monitoring Processing/Filtering Storage Lecture 2/3 Lecture 1 Lecture 2

44. Reminder from previous lecture (Repetition) Timing Sensor BX Beam crossing • Trigger • Timing • Front-end pipelines • Derandomizer buffers • Busy logic • Deadtime/efficiency • Parallel processing • (Event building) Clock Analog Pipeline Pipelined Trigger Accept/Reject ADC Busy logic Full FIFO DataReady “Real time”…. Processing Storage

45. Front-End at LHC (Repetition) Detector channels 106 - 108 Rate 40 MHz Analog pipeline 100 – 1000 evts L1 trigger (1-10 ms) 100 KHz – 1 MHz MUX-ADC-DERANDOMIZERS MUX-ADC DERANDOMIZER L2 trigger (10 ms- ms) DSP Digital buffer 10 – 60000 evts 1 - 40 kHz ~ 1-100 Gbytes/s Readout Data links (100 – 1000) Event builder L2/L3 trigger (Event filter) CPU CPU CPU 100 Hz ~ 1- 100 Mbytes/s CPU CPU CPU CPU CPU CPU

46. Deadtime (Repetition) Arrival time following exponential distribution (Poisson process) Probability FIFO Depth 1, 3, 5 Buffer size Process Processing time may also depend on event… Buffer occupancy Efficiency 1.0 • = 1/(N+1) , r = 1 • = (1-r)rN/(1-rN+1) , r = 1 5 buffers 0.5 3 buffers 1 buffer 1.0 2.0 r = <Processing time>/<Input period>

47. Trigger generals (Repetition) • Select as efficiently as possible signal events and reduce as much as possible of the background • Careful with trigger algorithm!!  There may be surprises, physics theorist have not predicted! • Select some background….to check with simulation… • Trigger monitoring • Trigger redundancy • Simulating trigger and all types of background • Since the detector data is not promptly available and the trigger function is highly complex, it is evaluated by successive approximations

48. Trigger levels at LHC (Repetition) 10-7 s Collision rate 40 MHz Level-1 selected events 100 kHz – 1 MHz • Particle ID (high PT, e, m, jets, missing ET) • Local pattern recognition • Energy evaluation based on prompt macro-granular information Level-2 selected events 1 kHz – 40 kHz • Clean particle signature (Z, W, …) • Finer granularity and precise measurements • Kinematics, effective mass cuts and event topology • Track reconstruction and detector matching • Vertex detection Level-3 events to tape ~100 Hz • Physics process identification • Event reconstruction and analysis 10-6 s 10-3 s 100 s

49. Level-1 trigger technologies • ASICs (Application Specific Integrated Circuits) • Highest performance • Better radiation tolerance • Lower power consumption • Expensive • FPGAs (Field Programmable Gate Arrays) • Enormous evolution: millions of gates operating at O(100 MHz) • Flexibility (reprogrammable) • Communication technologies • High-speed serial links (copper or fibre) • LVDS up to 10 m and 400 Mb/s, Glink, … • Backplanes (buses) • Large number of connections, multiplexing data (~160 Mb/s)

50. Efficiency versus background rejection Ex: B meson trigger in LHCb • L0 trigger(~2 ms) : Calculates transverse momentum (PT) of muons, electrons and hadrons • Discriminating variable: highest PT Transverse Momentum (GeV/c) Transverse Momentum (GeV/c)