1 / 47

Architecture and implementation of the ALICE Data-Acquisition system

Architecture and implementation of the ALICE Data-Acquisition system. T. Anticic, F. Carena, W. Carena, S. Chapeland , O. Cobanoglu, E. Dénes, R. Divià, U. Fuchs, I. Makhlyueva, J.C. Marin, F. Ozok, K. Schossmaier, C. Soós, P. Vande Vyvre, A. Vascotto, S. Vergara for the ALICE collaboration.

martha-ayala
Download Presentation

Architecture and implementation of the ALICE Data-Acquisition system

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Architecture and implementationof the ALICE Data-Acquisition system T. Anticic, F. Carena, W. Carena, S. Chapeland, O. Cobanoglu, E. Dénes, R. Divià, U. Fuchs, I. Makhlyueva, J.C. Marin, F. Ozok, K. Schossmaier, C. Soós, P. Vande Vyvre, A. Vascotto, S. Vergara for the ALICE collaboration CHEP 06 – 13/17 Feb. 2006

  2. ALICE DAQ Flexibility • 17 detectors • Pb-Pb • pp and pA • Synchronized / standalone tuning • Big events (86.5 MB) • Large bandwidth (1.25 GB/s to tape) • Low interaction rate (10 KHz) • Complex triggers • Different sets of detectors • Small events (2.5 MB) • High interaction rate (200 KHz) • Less bandwidth • Simpler triggers 25 GB/s raw from the detectors => 1.25 GB/s data files CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  3. TDS TDS ALICE DAQ Architecture Rare/All CTP BUSY BUSY LTU LTU DDL H-RORC HLT Farm TTC TTC FEP FEP FERO FERO FERO FERO Event Fragment 163 DDL 343 DDL 10 DDL 10 D-RORC 10 HLT LDC 425 D-RORC LDC LDC LDC LDC LDC 200 Detector LDC Load Bal. Sub event Event-Building Network EDM Event 50 GDC 10 DS GDC GDC GDC DS DS GDC File PDS Storage Network 25 TDS CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  4. TDS TDS ALICE DAQ Architecture Trigger High-Level Trigger CTP LTU LTU HLT Farm TTC TTC FEP FEP Detectors FERO FERO FERO FERO LDC LDC LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC PDS Storage Network Data Acquisition CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  5. CTP LTU LTU TTC TTC FERO FERO FERO FERO TDS TDS Trigger Initiate detectors readout of selected events Rare/All BUSY BUSY HLT Farm FEP FEP LDC LDC LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC PDS Storage Network CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  6. CTP LTU LTU TTC TTC FERO FERO FERO FERO TDS TDS Trigger Central Trigger Processor: decisions Rare/All BUSY BUSY HLT Farm FEP FEP LDC LDC LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC PDS Storage Network CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  7. CTP LTU LTU TTC TTC FERO FERO FERO FERO TDS TDS Trigger Local Trigger Unit: receives decisions for each detector Rare/All BUSY BUSY HLT Farm FEP FEP LDC LDC LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC PDS Storage Network CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  8. CTP LTU LTU TTC TTC FERO FERO FERO FERO TDS TDS Trigger Timing, Trigger and Control system: broadcast toelectronics Rare/All BUSY BUSY HLT Farm FEP FEP LDC LDC LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC PDS Storage Network CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  9. CTP LTU LTU TTC TTC FERO FERO FERO FERO TDS TDS Trigger Front-End Readout Electronics: use triggers, signal BUSY Rare/All BUSY BUSY HLT Farm FEP FEP LDC LDC LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC PDS Storage Network CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  10. CTP LTU LTU TTC TTC FERO FERO FERO FERO LDC TDS TDS Trigger Some DAQ components can feed back information Rare/All BUSY BUSY HLT Farm FEP FEP LDC LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC PDS Storage Network CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  11. LDC LDC LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC Storage Network TDS TDS Data Acquisition • Data-flow from electronics to permanent storage CTP LTU LTU HLT Farm TTC TTC FEP FEP FERO FERO FERO FERO Event Fragment 163 DDL 343 DDL 425 D-RORC 200 Detector LDC Load Bal. Sub event Event 50 GDC 10 DS File PDS 25 TDS CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  12. LDC LDC LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC Storage Network TDS TDS Data Acquisition • Detector Data Links: optical transmission CTP LTU LTU HLT Farm TTC TTC FEP FEP FERO FERO FERO FERO Event Fragment 163 DDL 343 DDL PDS CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  13. LDC LDC LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC Storage Network TDS TDS Data Acquisition • DAQ Read-Out Receiver Card: PCI interface CTP LTU LTU HLT Farm TTC TTC FEP FEP FERO FERO FERO FERO Event Fragment 425 D-RORC PDS CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  14. LDC LDC LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC Storage Network TDS TDS Data Acquisition • Local Data Concentrators: ship sub-events CTP LTU LTU HLT Farm TTC TTC FEP FEP FERO FERO FERO FERO Event Fragment 200 Detector LDC Sub event PDS CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  15. LDC LDC LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC Storage Network TDS TDS Data Acquisition • Global Data Collectors: build events CTP LTU LTU HLT Farm TTC TTC FEP FEP FERO FERO FERO FERO Sub event Event 50 GDC PDS CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  16. LDC LDC LDC LDC LDC Event-Building Network GDC GDC GDC DS DS GDC Storage Network TDS TDS Data Acquisition • Event Distribution Manager: load balancing CTP LTU LTU HLT Farm TTC TTC FEP FEP FERO FERO FERO FERO Load Bal. EDM PDS CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  17. LDC LDC LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC Storage Network TDS TDS Data Acquisition • Transient Data Storage: local storage before migration CTP LTU LTU HLT Farm TTC TTC FEP FEP FERO FERO FERO FERO File PDS 25 TDS CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  18. LDC LDC LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC Storage Network TDS TDS Data Acquisition • Permanent Data Storage: file archive CTP LTU LTU HLT Farm TTC TTC FEP FEP FERO FERO FERO FERO PDS CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  19. LDC LDC LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC Storage Network TDS TDS Data Acquisition • DAQ Services: database and central facilities CTP LTU LTU HLT Farm TTC TTC FEP FEP FERO FERO FERO FERO 10 DS PDS CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  20. HLT Farm FEP FEP LDC LDC TDS TDS High-Level Trigger • Filter events to optimize amount of valuable data CTP LTU LTU DDL H-RORC TTC TTC FERO FERO FERO FERO 343 DDL 10 DDL 10 D-RORC 10 HLT LDC LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC PDS Storage Network CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  21. HLT Farm FEP FEP LDC LDC TDS TDS High-Level Trigger • Detector Data-Links: transfer of event fragments CTP LTU LTU DDL TTC TTC FERO FERO FERO FERO 343 DDL LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC PDS Storage Network CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  22. HLT Farm FEP FEP LDC LDC TDS TDS High-Level Trigger • HLT Read-Out Receiver Card: data reception CTP LTU LTU H-RORC TTC TTC FERO FERO FERO FERO LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC PDS Storage Network CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  23. HLT Farm FEP FEP LDC LDC TDS TDS High-Level Trigger • Front-End Processors: data analyzing and decision CTP LTU LTU TTC TTC FERO FERO FERO FERO LDC LDC LDC Event-Building Network EDM GDC GDC GDC DS DS GDC PDS Storage Network CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  24. HLT Farm FEP FEP LDC LDC TDS TDS High-Level Trigger • DDLs, D-RORCs & LDCs: injection of results in DAQ CTP LTU LTU TTC TTC FERO FERO FERO FERO 10 DDL 10 D-RORC 10 HLT LDC LDC LDC LDC HLT decision Event-Building Network EDM GDC GDC GDC DS DS GDC PDS Storage Network CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  25. TDS TDS ALICE DAQ Rare/All CTP BUSY BUSY LTU LTU DDL H-RORC HLT Farm TTC TTC FEP FEP FERO FERO FERO FERO Event Fragment 163 DDL 343 DDL 10 DDL 10 D-RORC 10 HLT LDC 425 D-RORC LDC LDC LDC LDC LDC 200 Detector LDC Load Bal. Sub event Event-Building Network EDM Event 50 GDC 10 DS GDC GDC GDC DS DS GDC File PDS Storage Network 25 TDS CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  26. Implementation • Transfer of data to the DAQ • Computers • Network • Infrastructure • Software CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  27. Detector Data Link Transfer of data to the DAQ • Radiation tolerant SIU • D-RORC • DDG: DDL Data Generator (for tests) CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  28. D-RORC data throughput PCI-64 bandwidth fully used – 1GB/s with 4 D-RORCS CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  29. Test setup DDL, DDG, Trigger TRG2 LTU TRG1 DDGs DDG1 DDG2 DDL2 DRORC D-RORC DDL1 LDC Trigger crate CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  30. Test setup LTU TTC DDL, DDG, Trigger PC memory PC memory DDL DDG DRORC DDG DDL PC memory Trigger LDC Electronics emulation CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  31. Computer choices • Intel 32 (LDC,GDC) • AMD 64 (DS) • Scientific Linux CERN – SLC3 • GDC: CPU/memory performance • LDC: number / architecture of PCI slots • DS: database, multithreading performance SLC4 when certified CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  32. CPU performance Ubench/#CPU 64-bit promising GHz CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  33. Memory performance DDR 400 - Opteron DDR2 400 DDR 266 SDRAM 133 Speed and architecture important CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  34. DB performance tests CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  35. Operator console MySQL server DAQ nodes DB performance tests Configuration DB: concurrent access 65 LDC, 65 GDC CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  36. Experimental area CR1: DAQ CR2: HLT CR3: DCS/HLT CR4: MISC 50 meters Counting rooms in the pit CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  37. DAQ Counting Room • Tight space: 70m2 • 33 racks • ~300 computers CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  38. Racks 56U per rack 100 kW total 37 kW on UPS cooling doors CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  39. Racks • Remote control • Rack control unit (T°, smoke) • Power Distribution Units • KVM switch • Optical patch panel • Computers CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  40. ALICE Control Room • 30 workstations • separate room for PCs • console access to • machines in • all counting rooms • Multiscreen PCs • Nvidia NVS 280 • Linux Xinerama CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  41. DAQ Network 2x2x10Gb Ethernet Computing Center ALICE Control Room ALICE DAQ CR1 TDS DHCP DNS ECS Central Data Recording Event building DHCP DNS NTP DAQ control DAQ services 1Gb Ethernet gw General Purpose Network gw gw gw • Private network • Gateways: IPtables Trigger HLT DCS CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  42. Network performance GDCs @ wire speed Force10 E-1200 CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  43. Software • DATE: DAQ software handling the data flow • ECS: Experiment Control System • AFFAIR: fabric monitoring CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  44. ECS Unified view of experiment ECS DAQ HLT DCS TRG DIM/SMI: Communication/State machines CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  45. AFFAIR DAQ performance monitoring CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  46. AFFAIR DAQ performance monitoring Event Building DDL CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

  47. Conclusion • ALICE DAQ built with distributed components • General purpose and custom hardware/software • Synchronization by state machines • Architecture tested extensively • Deployment progressing • DAQ/ECS for detector commissioning installed at Point 2 http://cern.ch/alice-daq CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

More Related