Evolution of S-LINK to PCI interfaces

1. Evolution of S-LINK to PCI interfaces • W. Iwanski • (Henryk Niewodniczanski Institute of Nuclear Physics) • M. Joos, R. McLaren, J. Petersen, E. van der Bij • (CERN)

2. System evolution Tomorrow Today Yesterday 4 S-LINKs 4 S-LINKs S-LINK S-LINK S-LINK S-LINK FILAR SSP SSP FILAR S32PCI64 S32PCI64 slow PCI fast PCI fast PCI Eth. MEM MEM MEM slow PCI slow PCI Eth. Eth. CPU CPU CPU PC PC PC W.Iwanski LECC, Colmar 9-13 September 2002

3. Current use of S-Link to PCI interfaces in test-beams • Features of SSP / SSPCI • Simple hardware design (AMCC S5933 PCI controller) • Max. speed = 80 (RIO2) / 132 (PC) MB/s • Event overhead = ~8 us • One ROL per PMC / PCI slot TileCal test beam Muon test beam VMEbus VMEbus VMEbus VMEbus VMEbus SPS SPS SPS SPS SPS S-LINK S-LINK S-LINK S-LINK S-LINK SSPCI SSPCI SSPCI SSPCI SSPCI PC PC W.Iwanski LECC, Colmar 9-13 September 2002

4. S32PCI64 • 32/64 bit PCI • 33/66 MHz • 3.3 V PCI slots only • 3.3 V S-LINK LDC plug-in W.Iwanski LECC, Colmar 9-13 September 2002

5. Overview of S32PCI64 S 32 L to N K Map PCI PCI BURST INPUT 32 INPUT DMA L BUFFER FIFO I to BUFFER FIFO I FIFO ENGINE 64 FIFO 64 (1024 x 64 - bit) 128x 1024 x 64-bit 128x 64 64 - bit REQUEST REQUEST FIFO FIFO PCI 64 - BIT address, length 64 - BIT 33/66 MHz 32/64 - 32/64-bit PCI PCI BACKEND PCI ACKNOWLEDGE CONTROL BACK END CORE FIFO CORE FIFO LOGIC ctrl words, length LOGIC CONTROL, STATUS & CONTROL, INTERRUPT STATUS & REGISTERS FPGA Local logic Commercial IP Core W.Iwanski LECC, Colmar 9-13 September 2002

6. Features of S32PCI64 • 32/64 bit, 33/66 MHz PCI (3.3 V slots only) • 32-bit S-LINK (3.3 V LDC plug-in only) • 32-bit PCI-bus addressing • No Initiator wait states during a burst • Max. 64 Mbytes long DMA transfer • Protocol overhead per event: 5 PCI single cycles • Based on a commercial PCI IP core (PLDA) • Highly autonomous data reception • Interrupt generation selectable on several conditions • Data packets longer than 1kB are segmented W.Iwanski LECC, Colmar 9-13 September 2002

7. Features and applications of S32PCI64 based systems • Less load on the CPU to handle link protocol • event overhead =~2.5 us • 3.3 V PCI slots required • For applications requiring full ATLAS input rate • Applications: • CosmoDAQ • mainly for prototypes • Commercially available from NE (Nowoczesna Elektronika) W.Iwanski LECC, Colmar 9-13 September 2002

8. S32PCI64 test set-up • Elonex PC • SuperMicro 370DLE motherboard • memory bandwidth > 528 MB/s • SLIDAS data generator • VMEtro PBT-515BX PCI Bus Analyser • PCI bandwidth exerciser (PCI-Blaster) SLIDAS SLIDAS 160 MB/s 160 MB/s S32PCI64 S32PCI64 PCI 66 MHz/ 64 bit ServerSet III LE CPU MEM PC W.Iwanski LECC, Colmar 9-13 September 2002

9. PCI-Blaster • PCI bandwidth exerciser • firmware designed and implemented on S32PCI64 board to benchmark PCI bridge and memory in PC • Features • full speed PCI capability (528 MB/s) • read and write capability • read and write modes can be set-up simultaneously • simple programming model • data transfers in specified number of times or in infinite loop • 32-bit and 64-bit PCI bus (3.3V PCI bus only) • 33 and 66 MHz MHz PCI clock speed • 32-bit PCI-bus addressing • no interrupt capability W.Iwanski LECC, Colmar 9-13 September 2002

10. Overall throughput W.Iwanski LECC, Colmar 9-13 September 2002

11. Other measurements • Overhead of S32PCI64 hardware in single access from host CPU to S32PCI64 • write -> 30 ns (2 wait states) • read -> 45 ns (3 wait states) • Minimal time interval between 2 consecutive single commands seen on PCI bus (limited by CPU/PCI bridge) • write-write -> 75 ns • write-read -> 105 ns • read-read -> 330 ns • read-write -> 345 ns • Gap between DMA sub-bursts (packets > 1kB) -> 180 ns W.Iwanski LECC, Colmar 9-13 September 2002

12. FPGA Pluggable f/o transceiver LDC protocol Ser- des S32PCI64 Pluggable f/o transceiver ROL LDC protocol Ser- des S32PCI64 Pluggable f/o transceiver ROL Pluggable f/o transceiver ROL LDC protocol Ser- des S32PCI64 ROL LDC protocol Ser- des S32PCI64 3.3V PCI 64 bit / 66 MHz FILAR Four Integrated Links for ATLAS Readout W.Iwanski LECC, Colmar 9-13 September 2002

13. Main features of FILAR • Four 2.5 Gbit/s HOLA S-LINK Destinations integrated • Data channels individually enabled/disabled • 32/64-bit 33/66 MHz PCI bus • 32-bit S-LINK • 32-bit PCI-bus addressing • Protocol overhead per event and channel: 2 PCI single cycles • Highly autonomous data reception • Card temperature readout • Module scheduled for 1Q/2003 W.Iwanski LECC, Colmar 9-13 September 2002

14. Steps towards FILAR • New library written • in ROS context • FILAR emulator: simplified specification of FILAR implemented in existing hardware (S32PCI64) • Data from one S-link connector copied to all four data channels • One common flow control signal • logical OR of all four XOFF signals W.Iwanski LECC, Colmar 9-13 September 2002

15. Features of FILAR software • Package consists of Linux driver and user library • Detection of data packets based on interrupts • Support for: • multiple PCI cards • multiple channels • Code optimised for low overhead • ~1.5 us / event • Current API will also be valid for final FILAR W.Iwanski LECC, Colmar 9-13 September 2002

16. Performance of FILAR emulator W.Iwanski LECC, Colmar 9-13 September 2002

17. Performance (cont’d) • Improved performance of FILAR emulator for small packets with respect to S32PCI64 • Better performance of FILAR emulator running one data channel than that inS32PCI64 • Performance of FILAR emulator running two, three or four data channels is compromised by board limitation • flow control signal, working for all channels here, stops new data to whole interface whenever any of data channels is getting full.It prevents other, already empty buffers from being re-filled • gaps up to 4.5 us seen between some DMAs • If necessary, (still) costly single PCI cycles will be replaced by DMA W.Iwanski LECC, Colmar 9-13 September 2002

18. Summary • Transition from 32-bit/33 MHz to 64-bit/66 MHz PCI • full bandwidth of S-LINK (160 MB/s) • Stable design: evolution from S32PCI64 to FILAR • great part of software re-used • Use of FPGA and IP core in designs • easy debugging, testingand upgrading • re-usability of existing hardware • PCI-Blaster • FILAR emulator • Powerful and cost optimised FILAR design • integration of 4 links in one card reduces costs and overcomes a limit of fast 66 MHz PCI slots in PC • pluggable optical transceivers further reduce costs when less than 4 data channels are needed W.Iwanski LECC, Colmar 9-13 September 2002