1 / 26

FABRIC 4 Gigabit Work & VLBI-UDP Performance and Stability.

FABRIC 4 Gigabit Work & VLBI-UDP Performance and Stability. Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then “Talks”. Multi-Gigabit Trials on GEANT Collaboration with Dante. What is inside G É ANT2 What is the collaboration interesting? 10 Gigabit Ethernet

love
Download Presentation

FABRIC 4 Gigabit Work & VLBI-UDP Performance and Stability.

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. FABRIC 4 Gigabit Work&VLBI-UDP Performance and Stability. Richard Hughes-Jones The University of Manchesterwww.hep.man.ac.uk/~rich/ then “Talks” EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  2. Multi-Gigabit Trials on GEANTCollaboration with Dante. • What is inside GÉANT2 • What is the collaboration interesting? • 10 Gigabit Ethernet • UDP memory-2-memory flows • TCP flows with allocated Bandwidth • Options using GÉANT Development Network • 10 Gbit SDH Network • Options Using the GÉANT LightPath Service • PoP Location for Network tests EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  3. GÉANT2 Topology EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  4. EXPReS PC 10 GE 1678 MCC GÉANT2 POP A Dark Fiber L2 Matrix TDM Matrix GÉANT2 POP B 1678 MCC 1626 LM L2 TDM 1626 LM Managed Lambda’s EXPReS PC 10 GE GÉANT2:The Convergence Solution NREN Access Existing IP Router Existing IP Router EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  5. What do we want to do? • Set up 4 Gigabit Lightpath Between GÉANT PoPs • Collaboration with Dante • PCs in their PoPs with 10 Gigabit NICs • we FABRIC provide the SX XFPs, Dante the MCC ports • VLBI Tests: • UDP Performance • Throughput, jitter, packet loss, 1-way delay, stability • Continuous (days) Data Flows – VLBI_UDP and • multi-Gigabit TCP performance with current kernels • Experience for FPGA Ethernet packet systems • Dante Interests: • multi-Gigabit TCP performance • The effect of (Alcatel) buffer size on bursty TCP when using BW limited Lightpaths • Need A Collaboration Agreement EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  6. 10GE-LAN interface board in 1678 MCCDante interested in MCC performance 4x 10GE LAN board in 1678MCC Data Mapper Generic solution: • Rate adaptive mapping into VC4-nv with flow ctrl. and LCAS, n=1..64; • up to 4 ports per board Full rate option: • Ethernet full rate up to VC4-68v, flow ctrl. and LCAS for n<68 Optical Transport Hierarchy option: • Mapping into OpticalDataUnit1/2 with flow ctrl. O/E MAC BKPdr MAC O/E Data Mapper Data Mapper O/E MAC MAC O/E Data Mapper To matrix 4x10GE LAN interface ports Data Mapper Device - GFP-F into VC-4-nv - GFP-F into ODU1/2 EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  7. 10 Gigabit SDH backbone • Alkatel 1678 MCC • Node location: • London • Amsterdam • Paris • Prague • Frankfurt • Can do traffic routingso make long rtt paths • Available Feb 07 • Less Pressure for long term tests Options Using the GÉANT Development Network EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  8. Options Using the GÉANT LightPaths • 10 Gigabit SDH backbone • Alkatel 1678 MCC • Node location: • Budapest • Geneva • Frankfurt • Milan • Paris • Poznan • Prague • Vienna • Can do traffic routingso make long rtt paths • Ideal: London Copenhagen • Set up 4 Gigabit Lightpath Between GÉANT PoPs • Collaboration with Dante • PCs in Dante PoPs EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  9. Test PCs Have Arrived • Boston/Supermicro X7DBE • Two Dual Core Intel Xeon Woodcrest 5130 • 2 GHz • Independent 1.33GHz FSBuses • 530 MHz FD Memory (serial) • Chipsets: Intel 5000P MCH – PCIe & MemoryESB2 – PCI-X GE etc. • PCI • 3 8 lane PCIe buses • 3* 133 MHz PCI-X • 2 Gigabit Ethernet • SATA EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  10. Lab Tests 10 Gigabit Ethernet • 10 Gigabit Test Lab set up in Manchester • Cisco 7600 • Cross Campus λ <1ms • Server quality PCs • Neterion NICs • 4 Myricom 10 Gbit NICs – delivery this week • Chelsio being purchased • B2B performance so far • SuperMicro X6DHE-G2 • Kernel (2.6.13) & Driver dependent! • One iperf TCP data stream 4 Gbit/s • Two bi-directional iperf TCP data streams 3.8 & 2.2 Gbit/s • UDP Disappointing • Installed Fedora Core5 Kernels 2.6.17 & 2.6.18 (+web100 + packet drop) on the Intel dual-core PCs (Stephen Kershaw) EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  11. iBOB EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  12. Prototype iBOB with two sampler boards attached FPGA based signal processing board from UC Berkeley See Ralph’s Talk this afternoon EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  13. iBOB block diagram Bryan Anderson RAM 10GE CX4 Station 10GE CX4 - fibre media converter iBOB VSI Board VSI or headstack • 10 Gigabit Ethernet now available • UDP/IP module exists • Use for Demonstration of FPGA driven IP networking • Link iBoB to PC NIC – basic tests • Test over GÉANT • 4 Gigbit flow Onsala - Jodrell Disk based system EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  14. vlbi_udp Multiple stream tests over the Academic network EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  15. Ring buffer vlbi_udp Flow Diagram Vlbi_control vlbi_send vlbi_recv Control thread Control thread TCP Control Disk Memory Disk Memory Input thread Output thread Ring buffer UDP Data Send thread Receive thread From Simon Casey EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  16. e-VLBI at the GÉANT2 Launch Jun 2005 Dwingeloo DWDM link Jodrell BankUK MedicinaItaly TorunPoland EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  17. Geant2 Launch event, June 2005 EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  18. UDP Performance: 3 Flows on GÉANT • Throughput:5 Hour run • Jodrell: JIVE2.0 GHz dual Xeon – 2.4 GHz dual Xeon670-840 Mbit/s • Medicina (Bologna):JIVE 800 MHz PIII – mark6231.2 GHz PIII330 Mbit/s limited by sending PC • Torun:JIVE 2.4 GHz dual Xeon – mark5751.2 GHz PIII245-325 Mbit/s limited by security policing (>400Mbit/s  20 Mbit/s) ? • Throughput:50 min period • Period is ~17 min EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  19. UDP Performance: Flows on GÉANTPacket Loss & Re-ordering • Jodrell: 2.0 GHz Xeon • Loss 0 – 12% • Reordering significant • Torun: 2.4 GHz Xeon • Loss 6 - 12% • Reordering in-significant EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  20. Instability: 2 streams into JIVE, March 2006 From Simon Casey EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  21. EVN-NREN Gbit link Chalmers University of Technology, Gothenburg Metsähovi OnsalaSweden Jodrell BankUK Gbit link TorunPoland DedicatedGbit link Dwingeloo DWDM link MedicinaItaly EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  22. Any Questions? EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  23. Backup Slides EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  24. Connect. Communicate. Collaborate IP Links 1678 MCC Transport Node VLANs Router VC-4-nv Channels L2 Matrix TDM Matrix 1/10 Gigabit Ethernet From PoS to Ethernet • More Economical Architecture • Highest Overall Network Availability • Flexibility (VLAN management) • Highest Network Performance (Latency) EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  25. 10 Gigabit Ethernet: UDP Throughput • 1500 byte MTU gives ~ 2 Gbit/s • Used 16144 byte MTU max user length 16080 • DataTAG Supermicro PCs • Dual 2.2 GHz Xenon CPU FSB 400 MHz • PCI-X mmrbc 512 bytes • wire rate throughput of 2.9 Gbit/s • CERN OpenLab HP Itanium PCs • Dual 1.0 GHz 64 bit Itanium CPU FSB 400 MHz • PCI-X mmrbc 4096 bytes • wire rate of 5.7 Gbit/s • SLAC Dell PCs giving a • Dual 3.0 GHz Xenon CPU FSB 533 MHz • PCI-X mmrbc 4096 bytes • wire rate of 5.4 Gbit/s EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

  26. mmrbc 512 bytes mmrbc 1024 bytes mmrbc 2048 bytes CSR Access PCI-X Sequence Data Transfer Interrupt & CSR Update mmrbc 4096 bytes 5.7Gbit/s 10 Gigabit Ethernet: Tuning PCI-X • 16080 byte packets every 200 µs • Intel PRO/10GbE LR Adapter • PCI-X bus occupancy vs mmrbc • Measured times • Times based on PCI-X times from the logic analyser • Expected throughput ~7 Gbit/s • Measured 5.7 Gbit/s EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester

More Related