17 th IEEE Real Time Conference, Lisboa , May 24-28, 2010

Passive Optical Networks for Timing-Trigger and Control Applications in High Energy Physics Experiments 17th IEEE Real Time Conference, Lisboa, May 24-28, 2010 I. Papakonstantinou, C. Soos, S. Papadopoulos, J. Troska, F. Vasey, S. Detraz, C. Sigaud, P. Stejskal, S. Storey PH-ESE CERN ioannis.papakonstantinou@cern.ch

Outline • PON Definition • Motivation • Prototype Demonstrator • Protocol of Communication • Transceiver Design • Prototype characterization • Future work and Conclusions

PON Definition OLT • PON is a Point-to-MultiPoint (P2MP) optical network with no active elements in the signal’s path • Upstream and Downstream are multiplexed into ONE fiber • In the downstream direction (Master→Slave) PON is a broadcast network • In the upstream direction a number of slaves share the same transmission medium • Commercial PONs run at 1.25 Gb/s or 2.5 Gb/s but 10Gb/s standards have just emerged • Cost: ~900$/OLT cost ONU ~90$/ONU. Gigabit Ethernet TRXs Master Downstream D2 D1 U1 U2 UN Upstream U1 U2 UN Slave1 ... Slave N Slave2 ONUs

Motivation • LHC P2MP links are used to transmit timing-trigger-control information • Current TTC system is unidirectional (optically) • Advantages of proposed implementation a) It is based exclusively on COTS components b) The inherent bidirectionality of PONs means that TTC and throttle/busy networks can be merged simplifying the infrastructure and reducing the amount of connectors and cabling in the counting rooms c) the bidirectionality of the link can be also exploited to measure and to correct for latency variations d) FPGA based implementation offers a simple path to upgradability e) It is a potentially clock free solution

System Specifications

PON Demonstrator • 2 slave nodes have been implemented • One master and one slave node are implemented on the same Virtex 5 platform. • The second slave node is implemented on a Spartan 6 platform • Master and slave nodes are in different transceiver tiles and clocked by uncoupled sources TTCrx TTCex TTCrx OLT/TTCex ONU1/TTCrx ONU2/TTCrx

Downstream Frame • Synchronous transmission of super-frames with a period of 1625ns = 65*25ns at 1.6Gbit/s • Comma, “K”, character for frame alignment and for sync • T character carries trigger info. “F” for trigger protection or other functions • D1 and D2 carry broadcast or individually addressed information depending on the first bit of the D1 byte. • “R” field contains the address of the next ONU to transmit • 590.8 Mb/s are available for data downstream Slave1 Slave2 Slave64 TWEPP 2009 21-25 Sep. Paris – Optoelectronics and Links Session

Upstream Frame • Slave N1 receives an R character with its address and switches its laser ON • IFG between successive emissions allows to master receiver to adapt to different bursts • Channel arbitration is a logic built-in in the OLT • Total BW 800Mb/s, 7.7 Mb/s are available per Slave node for pure data

OLT Transmitter Clk ref • Latency issues at the Tx arise at clock domain crossing points • Gear Box logic ensures correct transition from 40MHz to 80MHz in the Tx-PCS • It is particularly important to bypass any elastic buffers in the data path • In GTX Tx this is achieved by advanced mode which forces PMA PLL to phase align XCLK and RXUSRCLK clocks 40MHz TI PLL Virtex- 5GTX Transmitter 80MHz 40MHz 80MHz DCM PMA PLL 80MHz RXUSRCLK 80MHz XCLK 800MHz Frame Generator / Gear Box 8B/10B 16 20 F I F O P I S O Input ... ... 1.6Gb/s 2 2 Output 1 1 clock domain crossing Tx-PMA Tx-PCS

ONU Receiver Clk ref REFCLK 80MHz Virtex- 5 GTX Rx or Spartan-6 GTP Rx • 80 MHz parallel clk can lock on any of the 20 first edges of the 800 MHz serial clk • That affects the order with which parallel data are exiting the SIPO • By fitting “K” characters into the frame and identifying them in a Barrel shifter we can predict the starting point of the XCLK PMA PLL 800MHz Rxin 1.6Gb/s Retimed Barrel Shifter 20 S I P O CDR 2 ... Input 800 MHz Serial 1 80 MHz XCLK DIV 1÷10 Shift value DCM ctrl Phase corrected 40MHz Phase corrected 40MHz 80MHz RXUSRCCLK DCM :2 Comma detect logic 180° 40MHz 800MHz clk . . . 17 18 19 0 1 2 3 80MHz clk

Latency Map OLT ONU 2.2 ns 2.1 ns ONU TRx FPGA FPGA OLT TRx GTX Receiver GTX Transmitter bias Tx+ Rx+ Tx+ Rx+ Tx- Rx- Tx- Rx- 137.5 ns 75ns 5 ns/m

Latency Stability Measurements • Histogram of 100 reset cycles • TI PLL power cycle →Tx reset→ Rx reset • Stability to within <150ps • But a better look-up-table implementation will give better results

Jitter Map After PLL RMS: <10 ps Ref 40 MHz Ref clk RMS: 3.17 ps TI PLL CDCL6010 After DCM RMS: 36.72 ps TI PLL Out 40 MHz After TI PLL RMS: 4.48 ps Before DCM RMS: 17.33 ps 80 MHz OLT ONU GTX Receiver FPGA Design DCM bias Tx+ Rx+ Tx+ Tx- Rx+ BS Rx- Tx- Rx-

Latency Monitoring Feeder fiber monitoring Full Ranging OLT ONU OLT φ Downstream Downstream 1490nm D2 D1 D2 D2 D1 D1 U1 D1 Upstream U1 D2 U2 Upstream 1310nm U2 ONU1 ONUN ... ONU1 ONUN ... Rx Rx Tx Tx

Outline • PON Definition • Motivation • Prototype Demonstrator • Protocol of Communication • Transceiver Design • Prototype characterization • Conclusions

Summary • A passive optical network for TTC applications has been successfully demonstrated • Fixed and deterministic latency has been achieved in the direction of the trigger transmission within ±150ps • Recovered clocks had <10ps RMS jitter with external PLLs • Network is bidirectional allowing for the slave nodes to provide the master with feedback • Also allow for latency variation detection and correction

Acknowledgements This work was supported in part by the ACEOLE, a Marie Curie mobility action at CERN, funded by the European Commission under the 7th Framework Programme

Questions? ioannis.papakonstantinou@cern.ch

BACK UP SLIDES

PONs in OSI Architecture • PONs reside in the last two layers in the OSI architecture namely • Data link layer which is responsible for the access to the medium and for error correction • Physical layer which is responsible for transmitting and receiving the information • In GPON terminology the two layers are called: G-Transmission Convergence (GTC) and Physical Media Dependent (PMD) • EPON modifies MAC layer to allow for bridging data back to the same port

System Power Budget Master Downstream Upstream Slave1 ... Slave N Slave2

Transceivers for PONs EPON GPON (1.24 Gb/s) • GPON has far more stringent requirements than EPON • For this reason GPON components are in general more expensive than EPON • OLT needs a 1490 nm laser (usually DFB-DBR) • Burst Mode Receiver • ONU needs 1310 nm F-P laser with burst mode laser drivers • PIN diode or APD at 1490 nm ONU OLT Coarse WDM Coarse WDM 1490 nm DFB TOSA ROSA (PIN) Post-amp CDR LD … 1310 nm AGC+ CDR Post-amp BM ROSA (APD) Optical signal F-P TOSA BM-LD Electrical signal

Burst Mode Rx Decision Threshold set APD LA Data Out TIA + - R Vref Vref Vref C Reset Peak Detection Circuit

P2MP Timing Parameters TWEPP 2009 21-25 Sep. Paris – Optoelectronics and Links Session

Barrel Shifter REFCLK PMA PLL Clk ref 80MHz 800MHz . . . 1/10 20 b16 S I P O Barrel Shifter CDR D I V 2 b18 . . . Rxin 1.6Gb/s 1 b17 RXRECCLK “K” 80MHz … b0 b3 b1 b2 b4 b5 800MHz clk 80MHz clk

Slave Rx Latency, Barrel Shifter bs = 0 bs = 5 bs = 10

OLT Tx – Gear Box 40MHz TI PLL x2 x1 80MHz REF 40MHz 40MHz R/K/T/F D1/D2/T/F TXUSRCLK IscharK 32 16 ... ... Frame Generator Gear Box 2 2 80MHz (o) (o) (e) (e) 1 1 IscharK (o) (e)

Comma Detect Logic ONU Rx – Comma Detect 0 180 Barrel Shifter EN 20 EN 20 Rearranged data ... 1 ... 1 80MHz RXUSRCCLK K? K? Shift value DCM ctrl Comma detect logic Phase corrected 40MHz DCM :2 0 B 90 B 40 MHz 180 B 270 0 T/F R/K 180

Upstream Frame • Slave N1 receives an R character with its address and switches its laser ON • IFG between successive emissions allows receiver to adopt between bursts • Upstream contains 32 bytes of <5555> for CDR followed by two SFD, <D555>, bytes for frame alignment • A 2 byte address field and data are following • Channel arbitration is a logic built-in at the OLT • Total BW 800Mb/s, 7.7 Mb/s are available per Slave node for pure data . . . K Addr Data 5555 Data 32 B 2 B 2 B 90 B

OLT Burst Mode Receiver REFCLK PMA PLL Clk ref Rxin 20 Oversampling x5 20 S I P O 800Mb/s 2 2 1 1 Δφ Slave1 Slave2 0 X 1 1 1 1 1 0 0 0 0 0 1 1 1 1 1 0 0 Samples

Upstream frame (2) • Dynamic range (Pslave1/Pslave2) affects IFG and thus available upstream BW/slave • It also affects the period between successive transmissions Slave1 Slave2 Slave1 Slave2 IFG 24/11/2009

Future PON • Tri-band PONs • WDM PONs

Tri-Band PONs 1550 nm Tx (trigger) • Tri-band PON transceivers utilize 1440nm band • They can be used in our context to separate the trigger from the control information • That can simplify protocols of communication and complexity at the OLT Master 1440 nm Tx (control) 1310 nm Rx Slave2 Slave1 1310 nm Tx 1310 nm Tx 1440 nm Tx 1440 nm Rx 1550 nm Rx 1550 nm Rx

Wavelength Division Multiplexing PON OLT RN ONUs WDM Tx • In a WDM PON scenario each channel (or channel group) is assigned one wavelength upstream and one downstream for communication with OLT • Benefits are higher bandwidth per channel, loss is independent from splitting ratio, less complicated scheduling algorithm at OLT, easy expansion, better delivery of services • Main disadvantage is the need for expensive WDM components such as AWGs, filters, tunable lasers / laser arrays / laser per ONU, broadband receivers etc • “Colorless” WDM PONs are developed to tackle cost issues λ1 Receiver Band A SLED λ1 … λ1, λ2, … λ16 λ17 λ17, λ18, … λ32 RSOA 3 dB Coarse WDM λ17 CW 1 x 16 AWG WDM Rx … λ16 λ17, λ18, … λ32 Receiver Coarse WDM λ16 Modulated … λ32 Upstream band REAM Downstream band Coarse WDM Band B λ32 Receiver Array DEMUX λ

17 th IEEE Real Time Conference, Lisboa , May 24-28, 2010