1 / 30

DAQ for FAIR: uDAPL and XDAQ on an InfiniBand test cluster

DAQ for FAIR: uDAPL and XDAQ on an InfiniBand test cluster. J. Adamczewski, H.G. Essel, S. Linev IT/EE Palaver 20.06.06. Outline. Requirements of DAQ for CBM The InfiniBand Cluster, uDAPL XDAQ tests on the IB-Cluster Summary and further evaluation. Detector. Detector. Detector.

landry
Download Presentation

DAQ for FAIR: uDAPL and XDAQ on an InfiniBand test cluster

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DAQ for FAIR: uDAPL and XDAQ on an InfiniBand test cluster J. Adamczewski, H.G. Essel, S. Linev IT/EE Palaver 20.06.06 FAIR DAQ - J.Adamczewski, S.Linev

  2. Outline • Requirements of DAQ for CBM • The InfiniBand Cluster, uDAPL • XDAQ tests on the IB-Cluster • Summary and further evaluation FAIR DAQ - J.Adamczewski, S.Linev

  3. Detector Detector Detector FEE deliver time stamped data ~50000 FEE chips ~1000 collectors CNet collect data into buffers collect ~1000 active buffers data dispatcher TNet time distribution ~1000 links a 1 GB/sec 1000x1000 switching ~10 dispatchers →subfarm BNet sort time stamped data event dispatcher subfarm subfarm subfarm PNet process events level 1&2 selection ~100 subfarms ~100 nodes per subfarm processing HNet high level selection to high level computing and archiving ~1 GB/sec Output CBM data acquisition FAIR DAQ - J.Adamczewski, S.Linev

  4. CBM DAQ features • Triggerless data transport until filter farm • Event building on full data rate ~1TB/s • B-net: ~1000 nodes, high-speed interconnections • Linux may run on all DAQ nodes (even FPGAs) • Test cluster with InfiniBand: small „demonstrator“ set-up within next year FAIR DAQ - J.Adamczewski, S.Linev

  5. InfiniBand test cluster • 4 nodes: • Double Opteron 2.2 GHz, 2GB RAM • Mellanox MHES18-XT host adapter • 2x Gigabit Ethernet host adapters • SuSE Linux 9.3, x64bit version • Mellanox MTS2400 24X InfiniBand switch PCI Express x8 FAIR DAQ - J.Adamczewski, S.Linev

  6. Testcluster networks InfiniBand switch: MTS2400 Gigabit Ethernet switch Ethernet cabling InfiniBand cabling InfiniBand HCA: MHES18-XT FAIR DAQ - J.Adamczewski, S.Linev

  7. Mellanox IB Gold 1.8.0 IPoIB: IP over InfiniBand driver uDAPL: User Direct Access Programming Layer MPI: Message Passing Interface others ... InfiniBand software FAIR DAQ - J.Adamczewski, S.Linev

  8. Direct Access Transport • Developed by DAT collaborative - http://www.datcollaborative.org • Objectives: • Transport and platform (OS) independence • Define user- (uDAPL) and kernel-level (kDAPL) APIs • Features: • Zero-copy data transfer • RDMA - Remote Direct Memory Access FAIR DAQ - J.Adamczewski, S.Linev

  9. uDAPL test application • C++ wrapper for C-based uDAPL library • Same program runs on all nodes • All-to-all connectivity • Time synchronization between nodes ~2 µs • One master (deliver commands) and many slaves • Several kinds of traffic schedule can be generated and executed • Statistics like transfer rate, lost packets, schedule accuracy can be measured FAIR DAQ - J.Adamczewski, S.Linev

  10. master node02 node01 node03 All-to-all communication test • Each node able to send data to any other node • Message- and RDMA-based transfer are supported • Master generate schedule and distribute it to all slaves • At predefined time schedule execution is started • Several schedules were tested: • One-to-all schedule • All-to-one schedule • All-to-all round-robin • All-to-all fix target FAIR DAQ - J.Adamczewski, S.Linev

  11. Slight difference between message- and RDMA-based transfer Reasonable transfer rates starting from 16 KB buffers Scheduled test results FAIR DAQ - J.Adamczewski, S.Linev

  12. Chaotic (schedule-less) transfer tests • Packets send without schedule • Next packet send when previous is completed • Performance depends on receiving queue size • For small packets performance 2 times better than scheduled approach • Low CPU usage FAIR DAQ - J.Adamczewski, S.Linev

  13. Data Acquisition framework requirements • Configuration of multiple nodes (connections and algorithms, database?) • Controls and monitoring • Data transport (format, protocol, hardware drivers?) • Message logging • Error handling and failure recovery (ARMOR?) • Modular architecture, extensability! (sub-detector tests) • User interface? FAIR DAQ - J.Adamczewski, S.Linev

  14. The CMS XDAQ framework • Standard DAQ framework for LHC CMS experiment(Orsini, Gutleber) http://xdaqwiki.cern.ch • C++ libraries on Linux, modular packages • Each node: context, xdaq process with embedded xdaq applications • Configuration: XML • Data transport: I2O protocol (Intelligent IO) • Communication: http, cgi; SOAP messages FAIR DAQ - J.Adamczewski, S.Linev

  15. The CMS XDAQ framework • XDAQ features continued... http://xdaqwiki.cern.ch • State machines (sync/async FSM) • Message logger, error handler • Monitoring tool • Hardware access library (HAL) • Front End Driver (FED kit, for CMS!) • Job Control (task handler for node control) • others: • exceptions • threads • infospace • data (de)serializers , ... FAIR DAQ - J.Adamczewski, S.Linev

  16. XDAQ on the CBM InfiniBand cluster • Installation problems: • XDAQ distribution not 64 bit save! • Code adjustments required • kernel modules for CERN linux excluded! • drivers for CMS hardware (FED) excluded • coretools done, worksuite partially,… •  Basic evaluation possible! FAIR DAQ - J.Adamczewski, S.Linev

  17. XDAQ tests on CBM InfiniBand cluster • Tested features on IB cluster: http://wiki.gsi.de/cgi-bin/view/Daq4FAIR/DaqClusterSoftwareXdaqStatus • hyperdaq webserver (main test controls ui) • xml configuration setup • control variable access and export (infospace) • SOAP messaging (test controller application) • state machines (sync./async. with web/SOAP ui) • monitoring application (very raw ui!) • multithreading (workloop framework) • exceptions, data wrappers, message logging, toolbox,… • i2o data transport (tcp roundtrip example) FAIR DAQ - J.Adamczewski, S.Linev

  18. XDAQ: web server hyperdaq web interface FAIR DAQ - J.Adamczewski, S.Linev

  19. XDAQ executive with applications • Common XML configuration file: • Each node knows all applications on all nodes! • Unique addressing by context (url) and application id FAIR DAQ - J.Adamczewski, S.Linev

  20. XDAQ: SOAP messaging • example commands:Configure, Enable, Halt, Reset,.. • any new commands may be defined • web interface not suited as real control system UI! • XDAQ applications may exchange SOAP messages with other UI (Labview?) xrelay: send SOAP control messages FAIR DAQ - J.Adamczewski, S.Linev

  21. http SOAP fifo peer transport new development i2o tcp DAPL XDAQ: messaging and peer transport messages transport layer FAIR DAQ - J.Adamczewski, S.Linev

  22. XDAQ: I2O messaging and peer transport FAIR DAQ - J.Adamczewski, S.Linev

  23. Peer Transport DAPL (ptdapl) • GSI development 03/2006-05/2006 • C++ wrapper class for uDAPL C library (class TBasic) • peer transport tcp as „template“ (starting point) • uDAPL buffers managed within XDAQ memory pool(class TEndpointBuffer) • avoids memcopy and new buffer allocation for each send package: • lookup if posted memory reference is known as send buffer • user code can write directly into uDAPL send buffer • multiple threads for sending, releasing, and receiving buffer FAIR DAQ - J.Adamczewski, S.Linev

  24. Peer Transport DAPL structure sender receiver peer transport uDAPL channel messenger message (I2O) transmitter receiverloop buffer pool FAIR DAQ - J.Adamczewski, S.Linev

  25. MyRoundTrip state machine MyRoundTrip XDAQ executive I2O callback web application I2O callback HyperDAQ I2O roundtrip setup XDAQ executive sender receiver EnableAction() token() token() Peer Transport default() FAIR DAQ - J.Adamczewski, S.Linev

  26. MyDataSource state machine workloop (thread) I2O callback web application HyperDAQ HyperDAQ I2O sender/receiver setup XDAQ executive XDAQ executive sender receiver MyDataDrain EnableAction() Benchmark() token() Peer Transport default() default() FAIR DAQ - J.Adamczewski, S.Linev

  27. P: package size t : transfer time („latency“) Observation: t linear with P: P big: network limit P small: framework latency ! Transport bandwidth B and latency t t P B P FAIR DAQ - J.Adamczewski, S.Linev

  28. ptDAPL benchmark results FAIR DAQ - J.Adamczewski, S.Linev

  29. Benchmark summary 955 MB/s plain uDAPL 905 MB/s XDAQ, I2O, ptDAPL 234 MB/s XDAQ, I2O, TCP over IB FAIR DAQ - J.Adamczewski, S.Linev

  30. Summary and Outlook • CBM DAQ requires fast builder network: InfiniBand test cluster • IB hardware and software installed, tested,... • uDAPL: developed C++ wrapper and test applications • many tests with different traffic patterns were performed • more tests on bigger InfiniBand clusters are required • XDAQ as software framework: • tested features: SOAP, web server, data transport,... • I2O implementation for uDAPL done! • cluster configuration? control system? scalability? hardware access? FAIR DAQ - J.Adamczewski, S.Linev

More Related