1 / 22

The ATLAS Switch Tester

The ATLAS Switch Tester. An application of the GETB platform Matei Ciobotaru CERN and “Politehnica” University of Bucharest. Outline. Motivation GETB architecture Hardware and firmware Control software Features Descriptor-based traffic generation

kara
Download Presentation

The ATLAS Switch Tester

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The ATLAS Switch Tester An application of the GETB platform Matei Ciobotaru CERN and “Politehnica” University of Bucharest

  2. Outline • Motivation • GETB architecture • Hardware and firmware • Control software • Features • Descriptor-based traffic generation • Client-Server traffic generation (Atlas-like) • Measurements Methodology • Sample results • Conclusions Matei Ciobotaru - TDAQ Workshop - Frascati

  3. Motivation • Atlas TDAQ DataFlow network • Large high-speed Layer 2 network (~700 nodes) • Central switches: ~250 ports, chassis based • Concentrating switches: 24 - 48 ports, pizza box • Packet loss or excessive latency  performance drop • Specific performance requirements for DataFlow switches • Need to evaluate devices with realistic test scenarios • See the “Switch Features” document • Testing equipment on the market is not fully adequate • Cost per channel is high • Not enough flexibility in defining traffic patterns • Layer 4 - 7 functionality – not essential for Atlas Matei Ciobotaru - TDAQ Workshop - Frascati

  4. GETB Platform – Hardware Configuration Flash GPS Clock SDRAM 2 x 64Mb 2 x Gigabit Ethernet SRAM 2 x 512Kb Altera Stratix EP1S25 FPGA 3.3V PCI • FPGA Firmware • 90% Handel-C, 10% VHDL • Commercial IP cores (IP = Intellectual Property) • Gigabit Ethernet MAC • PCI Controller • Logic utilization – approx 85 - 90% • Single FPGA controls 2 Eth ports • Multiple projects using the GETB • GETB Tester, Network Emulator, ROS Emulator • Common firmware / control infrastructure Matei Ciobotaru - TDAQ Workshop - Frascati

  5. GETB Platform – Control • Client (runs on any platform) • Talks to servers using XML-RPC • Runs user-defined scripts • Displays statistics in a GUI • Manages cards from multiple servers GETB Servers GPS Clock Distribution Gigabit Ethernet Links DUT Control GETB Client (Control PC) Device Under Test (DUT) • Distributed System • 15 servers hosting 65 cards • Entirely based on Python • Server (runs on Linux) • Configures the cards • Monitors status / collects statistics • Handles remote client requests • Accesses cards using IO-RCC (DataFlow package) Matei Ciobotaru - TDAQ Workshop - Frascati

  6. GETB Servers Device Under Test GETB Platform – Photo Matei Ciobotaru - TDAQ Workshop - Frascati

  7. Packet descriptors created offline Any traffic distribution is supported: Poisson, random, bursts, etc. Multiple packet types Raw Ethernet, IPv4, IPv6 Special packets: VLAN, Flow-Control Works at Gigabitline-speed for all packet types and sizes Each packet contains: Sequence number Timestamp Statistics at receiver Global counters Bytes, frames, packet types Counters per remote source Packet loss Average latency Average IPG Histograms Packets can be classified according to source or VLAN ID Latency, IPG and packet size can be histogrammed User defined resolution and histogram window (start offset, length) Traffic Generation / Reception Transmission (descriptor-based) Reception Matei Ciobotaru - TDAQ Workshop - Frascati

  8. Servers Requests Replies Client Low Watermark High Watermark Atlas-like Traffic (Client-Server) • Tester ports are divided into clients and servers • Client ports send requests to server ports, asking for data • Servers send replies with single or multiple frames (bursts) • Number of replies is configurable • Data rate is self tuned by limiting the number of client outstanding requests  OutReq • Each client monitors the number of outstanding requests • Request sent: OutReq = OutReq + 1 • Reply received: OutReq = OutReq - 1 • Two watermarks are defined • OutReq > High watermark •  Stop sending requests • OutReq < Low watermark •  Resume sending requests • Recovery mechanisms are implemented on both ends to deal with packet loss Matei Ciobotaru - TDAQ Workshop - Frascati

  9. Emulation of DataFlow applications • The Client-Server mode emulates the traffic between fundamental Atlas applications • L2PU, SFI, ROS Matei Ciobotaru - TDAQ Workshop - Frascati

  10. Test Description (Python) Collect statistics Integrity checks Analysis Report Run Test Measurements Methodology • User defines the test in Python • Script, pre-defined function • Multiple iterations are executed over a given space of parameters • Log files and test data are saved automatically • DUT is configured and its internal statistics are collected • Consistency checks are done • Results are analyzed • Tables and graphs are generated Matei Ciobotaru - TDAQ Workshop - Frascati

  11. Results for device A • Full mesh traffic • Each iteration with a different packet size • All tests running at 100% line speed • Fine level of granularity • Result reveals some “features” • Packets of certain sizes are not forwarded at line-speed • The “standard” RFC packet sizes have zero loss • Currently discussing with the manufacturer to better understand the results 1% loss – small, but non-zero Packet sizes used in standard RFC benchmarks have zero loss (64b, 128b, 512b, etc.) Matei Ciobotaru - TDAQ Workshop - Frascati

  12. DUT DUT Results for device B – Full Mesh • Device B • 120 ports  3 modules of 40 ports • Tested with fully-meshed and client server traffic • Fully-meshed traffic • DUT seems to perform very well under heavy load • This type of test can be executed using any testing equipment Full-Mesh Client-Server 99.9 % line speed Matei Ciobotaru - TDAQ Workshop - Frascati

  13. Device B Clients Servers Results for device B – Client Server • The DUT is not as good as we initially thought • The cabling of nodes on ports has a strong impact on device performance • In real Atlas we have constraints for media type • Optical fiber for servers (ROSs) • Copper for clients (SFI, L2PU) • Random node/port distribution not possible in practice • Working with the manufacturer to solve the issue Device starts losing at 88% load at the receivers (clients) Matei Ciobotaru - TDAQ Workshop - Frascati

  14. Hardware design Brian Martin Firmware and control software Matei Ciobotaru Contributions Stefan Stancu Micheal LeVine Markus Joos Applications currently using the GETB platform : GETB Tester Atlas ROS Emulator LAN / WAN Network Emulator Credits / Applications Matei Ciobotaru - TDAQ Workshop - Frascati

  15. Conclusions • The GETB platform provides a flexible environment for the design and development of Gigabit Ethernet applications • A tester able to evaluate switches for the DataFlow network has been created • 128 ports running at line-speed • Client-server traffic emulation provides a way to test devices under realistic Atlas conditions • Evaluating the device in a worst-case scenario • We have a comprehensive set of test procedures to check that devices meet our requirements Matei Ciobotaru - TDAQ Workshop - Frascati

  16. The End End of slide show. Backup slides Matei Ciobotaru - TDAQ Workshop - Frascati

  17. = Request = Reply Atlas-like Traffic – details • Watermarks • Low = 2 • High = 6 Servers Client 3 1 2 4 5 6 Matei Ciobotaru - TDAQ Workshop - Frascati

  18. classStack: "A well-known data structure" def __init__(self): # constructor self.items = [] def push(self, x): self.items.append(x) def pop(self): x = self.items[-1] del self.items[-1] return x def empty(self): return len(self.items) == 0 Python Language • Features • Object oriented, • Easy to learn, read, use • Extremely portable • Extensible (new modules) • What is it used for? • rapid prototyping • scientific applications • extension language • web programming Matei Ciobotaru - TDAQ Workshop - Frascati

  19. GETB Firmware Architecture • TODO – block diagram Matei Ciobotaru - TDAQ Workshop - Frascati

  20. SequentialBlock ParallelBlock // 3 Clock Cycles { a=1; b=2; c=3; } // 1 Clock Cycle par{ a=1; b=2; c=3; } Handel-C • Hardware description language • Like VHDL, but with syntax similar to C • The result of the compilation is the description of an electrical circuit • Contains built-in parallel constructs • Special features • Arbitrary widths on variables • Enhanced bit manipulation operators • Simple timing model • Each assignment is one clock cycle • Support for hardware constructs • Multiple clock domains, on-chip memories, external interfaces • Synchronization primitives: channels, semaphores Matei Ciobotaru - TDAQ Workshop - Frascati

  21. Test Description (Python) Collect statistics Integrity checks Report Run Test Analysis Measurements Methodology (variant) • User defines the test in Python • Script, pre-defined function • Multiple iterations are executed over a given space of parameters • Log files and test data are saved automatically • DUT is configured and its internal statistics are collected • Consistency checks are done • Results are analyzed • Tables and graphs are generated XMLRPC Control / Results Control SNMP / Telnet Statistics GETB Server DUT GETB Server Matei Ciobotaru - TDAQ Workshop - Frascati

  22. Small packet Big packet Wasted memory space Useful packet data Results for device A (variant) • Full mesh traffic • Each iteration with a different packet size • All tests running at 100% line speed • Result reveals some limitations • Device uses a paged memory management • Memory bandwidth is insufficient • Certain packet sizes more affected than others Matei Ciobotaru - TDAQ Workshop - Frascati

More Related