Who We Are?

1. Who We Are? Detector Building Group of KFKI-RMKI(Research Institute for Particle and Nuclear Physics), Budapest, HUNGARY

2. Our Results Previous projects: • Designing Fibre Channel test equipments (1994-1998) • Portable, 266 Mb/s Fibre Channel Tester • Fibre Channel Preprocessor for Logic Analyzers • Designing high-speed data transmission interfaces for CERN detectors (1996 - ...) • Protocol design and verification • FC, GbE, physical layer components • Hardware design • Software: linux drivers, test programs, program library

3. S-LINK (CERN) • S-LINK interface cards for CERN (mostly used at ATLAS) • simple, unidirectional link interface • first successful version was designed by KFKI-RMKI • newer versions, now: 2.5 Gb/s • Also used at MPI, Garching and at several HEP and other scientific institutes all over the world

4. ALICE Detector Data Link (DDL)

5. DDL • Detector Data Link(DDL) for the ALICE detector at CERN • 2.5 Gb/s, duplex link • advanced featues • Test devices for DDL(Front-end emulator, DDL link emulator, etc.) • PCI Read-Out Receiver Card (RORC) for DDL • 1st version: 33 MHz, 32-bit PCI card • 2nd version: 66 MHz, 64-bit PCI with 2 integrated DDL interfaces

6. DDL Interfaces DIU-RORC Interface FEE - SIU Interface Read-out Receiver Card (RORC) 66 MHz 64-bit PCI DDL Destination Interface Unit (DIU) Front-end Electronics DDL Source Interface Unit (SIU) Front-end Bus 32 bit 32 bit 4 T A P optical cable, max 350 m 1 32 bit JTAG BST lines DDL

7. DDL Features Interface: • Full duplex 32-bit data path on the destination interface (DIU card) • Half duplex 32-bit data path on the source interface (SIU card) • Full duplex flow control (XON/XOFF) • Interface clock up to 66 MHz (easy integration with PCI 66) • 264 MB/s peak data rate, 240 MB/s sustained bandwidth (max.) Implementation: • Duplex LC optical link up to 300 m • 2x FC or 2x GbE physical layer components • Small Form Factor Pluggable (SFP) optical transceivers • Bit error rate < 10 -12 • Robust error detection: very low undetected bit error rate < 10-40 • Automatic link synchronization and management Extras: • Stand-by support (low power consumption) • In-system reconfiguration /  Remote system upgrade • Monitoring of the aging of laser diode of optical transceivers • JTAG Boundary Scan Test interface for the Front-End electronics

8. Outlook to PCI Express PCI Express (formerly 3GIO) a „third generation” high performance I/O bus (1st generation: ISA, EISA, VESA, 2nd generation: PCI, PCI-X) PCI evolution (PCI, PCI-X, PCI-Express) PCI Express is software compatible to PCI and PCI-X

9. PCI Evolution • PCI • „multi-drop” parallel bus • conventional PCI: 33 MHz, 32-bit, max 4-5 card slots per bus • newer versions: 66 MHz, 64-bit, 1 (max 2) card slots per bus (!) • PCI-X • parallel bus, backward compatible (hw and sw) • 66 MHz , 32/64-bit, max 4-5 card slots per bus • 133 MHz, 32/64-bit, max 1-2 card slots per bus • 266, 532 MHz versions: max 1 card slot per bus (!) Buses can be bridged to each other (complex, expensive) • PCI-XP • It is still a local /IO bus, a „PCI bus”, but the connections between devices are serial, point-to-point interconnections • devices are interconnected via switch(es) • large number of devices can be interconnected • highly scalable, hot-plug, hot-swap, QoS, etc.

10. Limitations of PCI and PCI-X • Conventional 33 MHz PCI system • low bandwith to nowaday’s needs • 66 MHz, 133 MHz (PCI-X), ... • only few devices can be interconnected(only 1 or two) on a single bus because of the strict electrical load and timing constraints • Further limitations of PCI architecture • inefficient solutions in: • data transfer cycles (wait states) • accessing of system memory • interrupt handling • error handling

11. PCI Express • However, the basic problems of a parallel bus system(electrical load and timing constraints, lack of hot-pluggability, lack of scalability, etc.) can be solved only by a complete redesign of the architecture. • This resulted in PCI Express.

12. The PCI Express Link • Serial point-to-point interconnect between two devices • 1x, 2x, 4x, 8x, 12x, 16x, 32x type links • 1, 2, 4, ... 32 bidirectional signal pairs (lanes) • 1 lane: 2.5 Gb/s now, up to 10 Gb/s later • Low voltage, differential signaling (LVDS) • AC coupled • Data is encoded: 8B/10B a PCI Express link Device A Device B (e.g. a switch) 1 to 32 lanes

13. Bandwith • Scalable: more lanes /link: higher bandwith • 2.5 Gb/s per lane, 8B/10B encoding • Simultaneous traffic in both directions • 1x type link • 500 MByte/s aggregate bandwith • 250 MByte/s per direction • 32x type link: • 16 GByte/s aggregate bandwith • (8 GByte/s per direction)

14. Backward compatibility • Supports familiar PCI transactions such as memory read/write, IO read/write and configuration read/write • Same memory, IO and configuration address space as in PCI and PCI-X • Existing OSs and driver software will run in a PCI Express system without any modifications • Supports chip-to-chip interconnect and board-to-board interconnect via cards and connectors similar to the present PCI systems • PCI Express motherboard will have a similar form factor to existing ATX motherboards in PCs

15. Improvements, Benefits • Fast, highly scalable, serial point-to-point interconnect between two devices • bytes striped accross the lanes  more lanes per link: faster transmission • Packet-based communication protocol • Packets are transmitted serially • CRC embedded in each packet (auto retry: link level error correction) • Buffer-to buffer (link level) flow control  no need the packet retry • Message Signaled Interrupt (MSI) architecture • No side-band signals • hot-plug, error handling, interrupt signaling and else are accopmlished in-band • Multiple devices are interconnected via switches • Large number of devices can be connected together in a system • Much fewer pins per device package • Reduces chip and board design cost and design complexity • Reconfigurable, hot-plug, hot-swap, improved power management, etc. • Quality of Service (QoS) features: Traffic Classes, Virtual Channels • Configuration address space of devices is extended from 256B to 4KB • This needs new software

16. Outside the box? • It is expected that later it will also be used outside the box for I/O expansion • Initial focus of usage: inside the box PCI-XP links PCI-XP links Add-in card Add-in card switch switch opticalcable connector PCI-XP link

17. Mechanical Form Factors • Connector and daughter card form factors are under specification • The main type of connectors is very similar to the present PCI card edge connector • A 1x type card can be inserted in a x4 type slot, a.s.o. • Like with PCI, there will be other form factors • Server I/O module • Mini PCI Express card and connector (e.g. for notebooks) • mezzanine type card • NEWCARD (will replace CardBus PC card), etc.

18. Compete or complete? • PCI Express will coexist with PCI / PCI-X in the same system • PCI Express intends to replace AGP (graphics card if), but will not replace Serial ATA (hard disk, CD, etc) • HyperTransport • onboard chip-to-chip interconnect • said to be complementary to PCI Express • RapidIO • as an onboard chip-to-chip interconnect: complementary to PCI-Express • as a system interconnect: a competitive technology • 1394b, USB2.0, Fibre Channel, Gb Ethernet, Infinband, etc. • PCI Expresscan be a system interconnect that bridges these technologies • Will it be?