FTK: a Fast Track Trigger for ATLAS

1. FTK: a Fast Track Trigger for ATLAS Anton Kapliy

2. Overview • FTK is a hardware system for fast online track finding in the ATLAS detector that will be installed in 2014 • Chicago plays a central role in FTK collaboration: • 3 professors, 4 postdocs, 5 graduate students, 2 engineers • Principal Investigator: Mel Shochet • I’ve been involved in FTK since 2006 and participated in design, simulation, hardware, and initial commissioning. • Motivation for having FTK • Conceptual description of the system • Simulation and expected performance • Electronics development and commissioning Anton Kapliy - Sugarman Award Presentation

3. An LHC collision April 15th 2012: a Z➛𝝻𝝻 candidate event • LHC collides bunches of protons, not individual protons • Currently: ~25interactions per collision. Future: >70 • Most of these interactions are “boring” • Our job is to find interesting physics among this mess Anton Kapliy - Sugarman Award Presentation

4. Interesting physics Rates of production: • LHC goals: Higgs, SUSY, other BSM • This stuff is rare! • Tracking system is critical for selecting such events from vast backgrounds: • Isolated electrons and muons • Produced in decays of W and Z bosons • 3rd generation fermions: taus, b-quarks • Expect a lot of them because whatever participates in EWK symmetry breaking couples to particle mass new physics Example I: Three collimated tracks in a triple-prong tau decay: Example II: Displaced secondary vertex in B-hadron decay Anton Kapliy - Sugarman Award Presentation

5. ATLAS Tracker A typical charged track bends its way out through 3 Pixel and 8 SCT layers: • Pixel and SCT tracking detectors are relatively small • But due to high granularity, they contain a whopping 86M readout channels (>98% of the ATLAS total!) 2T solenoid B-field Anton Kapliy - Sugarman Award Presentation

6. ATLAS Trigger System • Trigger necessary to reduce event rate in real-time from 40 MHz to ~200 Hz • Of course, we would like to keep all “interesting physics” and reject boring stuff • Using tracking detectors for this rejection is very challengingdue to high occupancy • Impossible in LVL1 hardware. In LVL2, it is only possible within narrow cones (even that is slow!) • Available data: • 40 million events/sec • Each eventis O(1 MB) • → 1 Library of Congress / sec Able to save on disk: ~100 MB/s → about 200 events/sec Anton Kapliy - Sugarman Award Presentation

7. FTK comes to the rescue • FTK receives a complete copy of tracking data at LVL1 rate (75-100 KHz) • Quickly reconstructs all tracks with pT>1GeV and makes them available for LVL2 • Tracks are ready by the time LVL2 starts its selection of interesting physics! FTK A dedicated hardware systemfor track-finding that sits between LVL1 and LVL2 trigger stages FTK FTK Buffer Anton Kapliy - Sugarman Award Presentation

8. FTK approach: pattern recognition 1. Given a collection of hits in an event, we gang them together to form coarse “superstrips” – about 1 mm wide. 2. Build a pre-calculated lookup table of all coarse paths (“patterns”) that a charged track might take through the tracking layers. For full ATLAS detector, we expect about 1 billion such patterns 3. Load these patterns into specialized hardware – Associative Memories– that can simultaneously compare the event with ALL stored patterns and quickly return only those that match. MATCH! Superstrips: ~1mm Monte-Carlo tracks Original hits: ~50 microns Anton Kapliy - Sugarman Award Presentation

9. FTK approach: track fitting • 4. Restore full-resolution hits inside each matched pattern. • Ultimately, we need to create a list of tracks with: • χ2 (track quality) • Curvature (~ 1/pT) • Phi • Impact parameter, etc 5. Because our patterns are narrow, we have very few hits inside of them. The remaining combinatorial problem can be solved via the brute-force method– i.e., trying out every combination. 6. For each combination, perform a linearized fitto arrive at final track parameters. Since these fits involve only scalar products, they can be performed VERY quickly in modern FPGA chips: 1 fit / ns track parameters and χ2components hit coordinates This pattern has two combinations: Pre-computed constants Finally, we apply a χ2 cut to remove bad tracks, perform duplicate removal, and send all final tracks to LVL2 trigger. Matched pattern from previous stage along with all of its hits Anton Kapliy - Sugarman Award Presentation

10. Simulation – challenges • If we could simulate FTK very quickly, we wouldn’t need to build it – just stick the software algorithm into LVL2 trigger! • In reality, FTK simulation is very challenging: • Associative Memories need a lot of RAM to store 1 billion patterns • CPUs are not well-suited to perform millions of track fits • → each event must be simulated in ~100 separate jobs • Requires unprecedented parallelismnot seen anywhere else in ATLAS! • To simulate 1 million FTK events for 2016-2017 luminosity profile, we needed about 500k PC-hours • This would take ~50 years to simulate on a single PC • We wrote an advanced simulation framework that fully exploits the parallelism afforded by the HEP grid. • Simulation time reduced to about 1 week Anton Kapliy - Sugarman Award Presentation

11. Simulated FTK performance Reconstruction of track impact parameter (D0) is particularly important to identify jets with a B-hadron. • FTK performance is comparable to offline reconstruction • However, FTK hardware is 3 orders of magnitude faster! • Per-event time: ~25 µs for FTK, 10-100 ms for offline D0 Anton Kapliy - Sugarman Award Presentation

12. Example: FTK selects W boson events • Identification of W→µν decays becomes challenging as the number of interactions per collision increases: • Electromagnetic calorimeter isolation used in the trigger becomes powerless due to other “stuff” contaminating the event • Track isolation using FTK tracks restores good efficiency Add FTK tracks Anton Kapliy - Sugarman Award Presentation

13. FTK architecture Design started @ Fermilab Design + prototypes @ Chicago Finished and commissioned by Chicago Installed @ CERN FTK operates in 64 θxϕ towers divided between 8 core crates. Each crate consists of a number of boards, each with a dedicated function. Prototypes @ Italy HOLAs Design started @ Illinois Design started @ Argonne Anton Kapliy - Sugarman Award Presentation

14. First FTK installation @ ATLAS Installation & commissioning @ CERN Development & testing @ Chicago HOLA: a high-speed (2 GBps) link that provides a copy of all tracker hits to FTK to FTK FPGA to ATLAS • Highlights: • Data transmission protocol similar to IEEE 802.3 (Ethernet) • Transmission medium: optical fiber • Flow control from downstream cards on both channels • If this card fails, ATLAS loses the tracker • → extensive stress-testing to ensure reliable operation • Produced 270 cards (total system bandwidth ~50 GB/sec): • 268 passed our stress tests; 2 had a soldering problem • 31 installed at CERN and are being used now Anton Kapliy - Sugarman Award Presentation

15. Conclusions • As LHC luminosity increases, it will be essential to efficiently select interesting physics from the vast background of boring events • Tracking plays a key role in identifying isolated leptons, b-quarks, and taus • FTK brings the power of tracking-based event selection to LVL2 trigger • Schedule: • Tested board prototypes & TDR – by Summer 2013 • Full system ready after the 2013-2014 LHC shutdown • FTK is a challenging, exciting and very educational project: • Development and simulation of a complex trigger system • State-of-the-art electronics development, testing, and commissioning THANK YOU! Anton Kapliy - Sugarman Award Presentation