ATLAS TDAQ

1. Linux QoS for Atlas Event Builder ATLAS TDAQ TDAQ week Yoshiji Yasu (KEK) on behalf of the Atlas Trigger/DAQ group Contents QoS Application to Atlas Event Builder Functionality of QoS Basic performance measurements Conclusions and Issues Y.Yasu

2. ATLAS TDAQ QoS application to Atlas Event Builder TDAQ week Congestion avoidance on data path Guaranteed bandwidth of control path and data path Bandwidth management for unbalanced data paths Y.Yasu

3. ATLAS TDAQ QoS application to Atlas Event Builder TDAQ week UDP/IP or TCP/IP UDP/IP Multicast Y.Yasu

4. ATLAS TDAQ What is QoS( Quality of Service) TDAQ week Functionality of QoS Admission control decides whether input QoS request is acceptable or not. Provisioning sets up network configuration. Classification classifies input packets according to the QoS. Queue management and Scheduling processes packets at output port. Y.Yasu

5. What is QoS:Class-Based Queueing (CBQ) ATLAS TDAQ TDAQ week Class 1 Set over-limit estimator packet Class 2 Packet scheduler classifier Class 3 CBQ can classify incoming packets into multiple classes CBQ can share and limit the transfer rate Y.Yasu

6. What is QoS :Token Bucket Filter (TBF) ATLAS TDAQ TDAQ week rate Token buffer Bucket size Data packet Data Buffer Token buffer is fulfilled with tokens at start time. Y.Yasu

7. What QoS is :Traffic control in Linux kernel ATLAS TDAQ TDAQ week Upper layers(TCP,UDP,...) Traffic Control Input de-multiplexing Forwarding Output queueing Traffic control is done only at output queueing. Outgoing packet is only controlled. Y.Yasu

8. ATLAS TDAQ What QoS is : a configuration of QoS TDAQ week Classifier u32: Specified IP Address Qdisc (TBF) Rate:500kbit Buffer size :100kB Class（CBQ） 1:1 rate:1000Mbit Qdisc（CBQ）1: bandwidth:1000Mbit U32 is a classifier. TBF limits the transfer rate to 500kbit/s at this example. Y.Yasu

9. Configuration of Testbed ATLAS TDAQ TDAQ week Whether QoS can assign transfer rates on TCP/IP? Gigabit Ethernet Switch Sender Netperf in best effort Message size : 1kB Receiver Netperf PC500 PC500 PC500 PC500 Sender Receiver Configuration of PC500 system CPU: PentiumIII/500MHz Chipset 440GX Memory: 100MHz/SDRAM/256MB PCIbus: 32-bit/33MHz NIC: AceNIC(1MB) Linux kernel version : 2.4.5 gcc version: egcs-2.91-66 acenic driver version : 0.8 Y.Yasu

10. ATLAS TDAQ QoS performance on 1x1 system TDAQ week Result from our paper in DAQ2000 conference No QoS Measured rate in Mbit/s On PC500s Assigned rate in Mbit/s Bandwidth assignment worked well. The overhead of QoS was very small and nearly 1 %. Y.Yasu

11. ATLAS TDAQ Congestion control with QoS TDAQ week Constant Bit Rate of ATM network can control the congestion on event builder by managing the bandwidth. However, Ethernet does not have the functionality. Linux QoS can manage the bandwidth. Can Linux QoS with Ethernet control the congestion? We use UDP/Multicast with pseudo trigger for the measurement. Y.Yasu

12. Configuration of Testbed ATLAS TDAQ TDAQ week Gigabit Ethernet Switch UDP/Multicast Sender Pseudo Trigger Message size : 1kB Receiver Measurement by tcpdump utility PC500 PC500 PC500 PC500 Sender Receiver Receiver Receiver Configuration of PC500 system CPU: PentiumIII/500MHz Chipset 440GX Memory: 100MHz/SDRAM/256MB PCIbus: 32-bit/33MHz NIC: AceNIC(1MB) Linux kernel version : 2.4.5 gcc version: egcs-2.91-66 acenic driver version : 0.8 Y.Yasu

13. Message rate(100Hz ) without QoS ATLAS TDAQ TDAQ week Periodic Trigger : 100Hz, Message size : 1kB, without QoS Time interval of message in msec. First message Last message Data rate :800kbps(100kB) Without QoS Time ----> Message number Y.Yasu

14. Message rate(100Hz ) with QoS ATLAS TDAQ TDAQ week Periodic Trigger : 100Hz, Message size : 1kB, Assigned rate : 500kbps QoS begins scheduling (bandwidth:500kbps) Time interval of message in msec. Messages were scheduled in 10msec and 20msec. Token is still enough here. Data rate :800kbps(100kB) Without QoS Message number Y.Yasu

15. Message rate(1kHz ) with QoS ATLAS TDAQ TDAQ week Periodic Trigger : 1kHz, Message size : 1kB, Assigned rate : 5Mbps Average transfer rate was 5Mbps. Time interval of message in msec. Messages were sent immediately. This is due to Linux scheduling time. THIS IS PROBLEM! Message number Y.Yasu

16. ImportantLinux Scheduling time ATLAS TDAQ TDAQ week Linux scheduling time is usually 10msec. It is defined by a parameter HZ in the kernel. Namely, HZ is 100. This value is not convenient for the traffic shaping because the average trigger rate of Event Builder input is over 1kHz. Let’s make the value 1500(667usec). Y.Yasu

17. Message rate(1kHz ) with QoS ATLAS TDAQ TDAQ week Periodic Trigger : 1kHz, Message size : 1kB, Assigned rate : 5Mbps Time interval of message in msec. Messages were sent around 1.6msec. Data rate :5Mbps (Assigned rate :5Mbps) Message number Y.Yasu

18. QoS measurement with random trigger ATLAS TDAQ TDAQ week Periodic trigger  OK However, actual trigger is random. Can QoS make random traffic flat? Namely, does it simulate Constant Bit Rate as ATM? Let’s make random trigger. Y.Yasu

19. Message rate(1kHz ) with QoS ATLAS TDAQ TDAQ week Gaussian Trigger : 1kHz, Message size : 1kB, Assigned rate : 5Mbps QoS makes random traffic flat! Message rate is constant. Time interval of message in msec. Data rate :5Mbps (Assigned rate :5Mbps) Message number Y.Yasu

20. Conclusions and Issues ATLAS TDAQ TDAQ week Conclusions • Functionality of QoS was introduced. • Linux QoS with Ethernet can manage the transfer rate (bandwidth). Its overhead was small. This feature makes the feasibility of not only Guaranteed bandwidth of control path and data path but also bandwidth management for unbalanced data paths • Linux QoS with Ethernet can make 1kHz random traffic flat. This feature makes the feasibility of congestion control on Atlas Event Builder data path. Issues • We will continue the precise measurements. • We would like to apply the QoS features to recent Data Collection software framework. Y.Yasu