Advanced Collaboration and Development Strategies in Firmware Projects

1. CRU Weekly Meeting Erno DAVID, Tivadar KISSWigner Research Center for Physics (HU) 18 November, 2015

2. News • TPC turbulences • Maybe the bandwith of the GBT links are not enough – new read-out options is investigated • Maybe the CRU buffer size is not enough – CRU for TPC can be questioned • The TPC team promises some answers to these question by next week • Let us be patient and cooperative • Let us continue our work • 2nd CRU Firmware Development Workshop at CERN • proposed dates: 19-20 Jan • by that time all the Indian colleagues will arrive at CERN (Tapan said) • topics to be defined and sand out a call 2

3. Central Git Server Service • GitHub vs. CERN GitLab: • O2 Plenary Meeting, Tuesday, 10 November 2015,CWG 11: Software Repository - Dario Berzano(https://indico.cern.ch/event/347916/contribution/1/attachments/1184582/1716981/20151110_-_O2_Plenary_-_CWG11_Git.pdf) • The proposal: • GitHub for open source software • CERN GitLab for software (and firmware) with restrictions (e.g. fw with licensed vendor IPs) • CRU CERN GitLab repository: • https://gitlab.cern.ch/groups/alice-cru • Currently alice-cru and alice-tpc-cru added as developers. • Please check your GitLab account if you plan to contribute. 3

4. Quartus flow • Central Git repository • Local Git repository • Local working directory for Quartus project • Quartus II 64-Bit Version 15.0.2 Build 153 • Tested under Windows (7, 8.1) and Linux (Ubuntu 14.04) • /core/firmware/src – contains firmware source codes • /core/firmware/scripts – contain the Quartus TCL build scripts (project creation, I/O settings, source codes) • /tcp/firmware/src and /tcp/firmware/sripts – TPC specific source codes and scripts • quartus_sh --script %CRU_GITDIR%/tpc/firmware/scripts/build.tcl • +------------------------------------------------------------------------------------------------------+ • ; Flow Elapsed Time ; • +----------------------+--------------+-------------------------+---------------------+----------------+ • ; Module Name ; Elapsed Time ; Average Processors Used ; Peak Virtual Memory ; Total CPU Time ; • +----------------------+--------------+-------------------------+---------------------+----------------+ • ; Analysis & Synthesis ; 00:11:00 ; 2.1 ; 5483 MB ; 00:12:39 ; • ; Fitter ; 00:34:58 ; 4.9 ; 19125 MB ; 02:25:57 ; • ; Assembler ; 00:01:47 ; 1.0 ; 7626 MB ; 00:01:46 ; • ; Timing Analyzer ; 00:05:09 ; 7.4 ; 11914 MB ; 00:24:53 ; • ; Total ; 00:52:54 ; -- ; -- ; 03:05:15 ; • +----------------------+--------------+-------------------------+---------------------+----------------+ 4

5. Modelsim flow • Central Git repository • Local Git repository • Local working directory for Modelsim • ModelSim ALTERA STARTER EDITION 10.3d (VHDL only, no mixed mode) • /core/firmware/sim – contains the basic simulation framework (TTS, GBT, PCIe, DCS pattern gen/chk) • /tpc/firmware/sim and /tpc/firmware/scripts –detector specific user logic, testbenches and scripts • The main idea: • Provide a functional test ability before the synthesis • Provide the detector specific user logic with some inputs (TTS, GBT, DCS) • Accept the output from user logic and check it (PCIe) 5

6. Git flow • Folder based permissions: • LHCb’s soulution • It requires Gitolite features for folder permission (git push control) • It is not supported by GitLab • Protected Git branches: • Supported by GitLab (the master branch protected by default) • Possible flow: separated branches for core and tpc development, and merge into master periodically • Fork / Pull request model: • The proposed flow by O2 team • Fork the central repository, implement features, send a pull request to central team 6

7. 10G PON 2014 (11.2 Gbps)

8. 10G PON 2015 (9.6 Gbps)

9. LHC Clock Distribution CRU PON Frame 6x40 bits @ 40 MHz 9.6 PON Downlink LTU K28.5 K28.5 K28.5 K28.5 K28.5 K28.5 . . . PON Receiver 40 bit 40 bit LHC clock (40 MHz) PON RX clock (240 MHz) PON TX clock (240 MHz) 1 Reconstructed LHC clock (40 MHz) 2 GBT REF clock (120 MHz) 3 GBT TX clock (120 MHz) GBT Frame 3x40 bits @ 40 MHz 4.8 GBT Downlink GBTX H H H H H H GBT Transmitter Reconstructed LHC clock (40 MHz)

10. GBT Bank Clocking Scheme CRU Arria 10 FPGA 240MHz 10G PON Transceiver TTS 10G PON REFCLK 240MHz 40MHz 1 2 External PLL + Jitter Cleaner 184 bit TTS_RX_CLK 240 MHz 7 0 3 120MHz 40MHz IOPLL 3 GBT Transmitters Bank 0 GBT Gear Box Downlink Logic Bank 0 250MHz 40 bit 80 bit GBT_BANK_0_REFCLK 120 MHz 256 bit TX SCA 2 bit TX GBTx GBT Bank 0 – Link 0 .. 5 (Consist 6 adjacent bonded Arria 10 transceiver) 2 bit 120MHz 40MHz IOPLL GBT Gear Box Uplink Logic Bank 0 250MHz GBT Receivers Bank 0 80 bit 256 bit 40 bit RX SCA 2 bit RX GBTx 2 bit The number of GBT Banks is equal to GBT Links / 6 10

11. LHC Clock Distribution Uncertainty • Point 1: • The 40 MHz LHC clock must be reconstructed based on the K28.5 symbol position • K28.5 position precision, 1/9.6 GHz => 104.2 ps • How to adjust and measure the 40 MHz PLL phase? • Point 2: • The phase uncertainty originated from external jitter cleaner and PLL • Based on the PON teams report this can be controlled • Point 3: • The Arria 10 transceiver first produce the 2400 MHz serial clock from the external 120 MHz reference clock then it divides back to produce the 120 MHz TX clock • Is it controllable the phase difference between the 120 MHz ref. clk and TX clk? 11

12. Arria 10 Transmitter Clocking

13. Arria 10 Receiver Clocking