SoC Technology for Embedded Control and Interlocking in Fast Pulsed Systems

2. SoC technology for embedded control and interlocking within fast pulsed systems P. Van Trappen, N. Magnin, J. Schipper, M. Gauthier, E. Carlier, E. Oltedal System-on-Chip Workshop – CERN https://indico.cern.ch/event/799275/timetable/

3. Abstract Document reference • The control of CERN’s beam-transfer kicker magnet high-voltage pulse generators requires often the use of digital control in FPGAs to allow tight timing control (jitter better than one ns) and fast protection of the high-voltage thyratron and semiconductor switches. A FPGA is a perfect candidate for the digital logic, it is however limited in integration possibilities with the actual and future control systems. • TE-ABT has recently levered the market push for integrated devices, so called Systems on a Chip (SoC). This technology allows for a tightly coupled ARM (Cortex A9 in case of the Xilinx Zynq-7000) processing system and programmable logic in a single device thus avoiding difficult placement and routing on an electronic card. We present here two projects where the Xilinx Zynq-7000 has been used: • It is used in the fast interlock detection system to integrate it neatly within the CERN control framework, by using Embedded Linux and running a Snap7 server. • In the second project it implements a lower-tier communication bridge between a front-end computer, using serial communication in software and high-fan out multiplexing in the programmable logic for timing and analogue control logic.

4. Project 1 description Document reference Fast Interlocks Detection System (FIDS) • Single hardware replacement for a wide variety of old, often obsolete, ABT hardware that implements the fast-interlocking functionality, i.e. detect and act on faulty behavior of fast power-switches (thyratrons, GTO-stacks) for kicker installations that the CERN accelerator complex • Choice for a solution based on the Xilinx Zynq-7000 SoC on a custom card because: • Fast reaction times (<100 ns to retrigger) and variable comparator thresholds require a FPGA • VME-based solution would require development of a custom-card anyway; VME considered soon-to-be-obsolete (bandwidth sufficient); complex and costly architecture with separate CPU card and crate • No integration of a SBC to keep design and firmware 100% in our control (and free) • Use of integrated CPU allows: • Implementation of an embedded (SILECS) Snap7 server for operational communication • Reduced software development time because of peripheral device drivers for Embedded Linux and use of C / C++ / Python  e.g. i2c driver for external path panel • Automated test-procedure running on the card itself; permanent diagnostics running on-board (Python) • Integrated digitizer & analysis functionality: streaming FPGA data to the CPU using DMA • Status: 35 boards produced, 10 being used >1 year without problems; LS2 operational installation started

5. Project 1 description Document reference Fast Interlocks Detection System (FIDS) Some more info on the SILECS VC (Virtual Controller): • SILECS = Software Infrastructure for Low-level Equipment Controllers (CERN) https://wikis.cern.ch/display/SIL/SILECS+Home • Set of libraries and a configuration tool for FEC <> PLC/hardware communication • Ensures a common memory map declaration, with checks on both sides • Virtual Controller = embedded stand-alone server process, supports ARMv7 compilation • Uses the Snap7 TCP/IP protocol, originally for native communication with Siemens S7 PLCs http://snap7.sourceforge.net/; also supports ModBUS-TCP. • Works well but requires polling • Possible improvement: use UPC-UA, industry-standard middleware, with a subscription model (issue with feature request at SILECS gitlab) • Newest PLCs support it by default • Open-source OPC-UA libraries exist for the VC • Improved security by encryption (S7 relies on ISO-TSAP which is old and possibly insecure - link) • Allows authentication (now anyone can create a socket and read/write to a Siemens S7 PLC)

6. Project 1 description Fast Interlocks Detection System (FIDS) – main controller board FASEC https://www.ohwr.org/project/fasec/wikis/home • Miscellaneous • UCD90120ARGC power controller (programmable over JTAG) to survey power rails and manage power-on and power-off sequence • Xilinx-style JTAG connector • 2x hexadecimal switches (for settings selection) • UART over a Micro-USB connector (FT232) • Front panel • 6x LEMO-00 analog input (+-10V range; 200 KSPS) • 1x SFP port (White Rabbit compatible) • 2x LEMO/SMC inputs (5V) • 2x LEMO/SMC outputs capable of driving 3.3V @ 50 ohm • 1x USB3.0 A connector for high-speed serial card-to-card GTP link (no USB functionality!) • 8x Programmable LED (4 by PL, 4 by PS) • POR Reset push button • Back panel • 2x 100 Mbit Ethernet RJ45 ports (1 interface, switched) • RJ11 connector for FIDS patch panel (isolated I2C)_ • D-Sub 15 connector with isolated 4x fail-safe NC contact channels and 2x 24V input channels • 10-layer PCB • Not reproduced outside of this project, however there’s a fork in the making at ESRF (European Synchrotron Radiation Facility, Grenoble) – a RFoWR project called CITY Document reference • XC7Z030 controller, SoC with Kintex-7 fabric (called PL, i.e. Programmable Logic) and Dual ARM Cortex-A9 MPCore at 1 GHz (called PS, i.e. Processing System) • Two Low-Pin Count FMC slots • FMC1 connectivity: Vadj fixed to 2.5V, 34 differential pairs, 1 GTP transceiver with clock, 2 clock pairs, JTAG, I2C • FMC2 connectivity: Vadj fixed to 1.8V, 34 differential pairs, JTAG, I2C • FPGA configuration • From QSPI flash, Ethernet (through U-Boot bootloader) or MicroSDcard • Clocking resources • 1x 33.33 MHz fixed oscillator, SoC main clock (clock distribution to PL possible) • 1x 125 MHz fixed oscillator for the FPGA fabric • 1x 25 MHz TCXO controlled by a DAC with SPI interface (AD5662, used by White Rabbit PTP core) • 1x 20 MHz VCXO controlled by a DAC with SPI interface (AD5662, used by White Rabbit PTP core) • 2x low-jitter frequency synthesizer/fanout (TI CDCM61004, fixed configuration, Fout=125 MHz, used by White Rabbit PTP core) • On-board memories • 2x 512 MByte (4 Gbit) DDR3L (MT41K256M16HA-125:E) • 2x 128 Mbit QSPI flash for FPGA bitstream and Linux kernel & root file system storage (S25FL128SAGMFIR01)

7. Project 1 description • Gateware & software development workflow: • Gateware with Vivado, using git submodules for HDL libraries • Using the typical Embedded Linux development workflow: • FSBL, U-Boot, kernel and rootfs cross-compiled on host (PetaLinux / Yocto OE) • Initramfs from SD card loaded to memory • Sometimes NSF rootfs during development Document reference Fast Interlocks Detection System (FIDS) Project gateware block diagram:

8. Project 1 description Document reference Fast Interlocks Detection System (FIDS)

9. Project 2 description Adapter board sbRIO Document reference G-64 eradication project Several ABT kicker installations still rely on G-64 hardware (bus & CPU) for which all hardware has become obsolete. No guarantee to keep the Motorola 6809 C-compiler working on modern hosts! No resources to consolidate the full control system, hence keeping all hardware but the PS/PO CPU card and adapters, replacing it by a NI sbRIO 9607 housing a Xilinx Zynq-7000 We designed a simple adapter board (EDA-03907) for power supplies, IO buffers and mechanical integration Motivation: RS-232 communication to FEC (CPU), multiplexing >60 outputs to the existing ABT electronic cards without changing crate and/or or hardware (FPGA) Also: project engineer with LabView expertise Future improvement: eradicate RS-232 and use Ethernet communication Status: prototype fully tested; 30 boards produced for LS2 operational installation

10. Project 2 description • Gateware & software development workflow: • Gateware with NI LabVIEW, not using any HDL (direct use of Vivado not supported for the sbRIO9607) • Not your typical Embedded Linux development workflow: • Using precompiled FSBL, U-Boot, kernel and rootfs • Mounting a host NSF directory for software development • Cross-compiling on CERN BE/CO VPC machines, Linaro cross-compiler at /acc/local/share/abt/ARM7/ Document reference G-64 eradication project Different gateware and software development workflow, because of National Instruments’ environment (‘NI Linux Real-Time’)

11. Environment Here are we today… Document reference • Support needed, preferably CERN-wide: • Development environment (BE/CO Makefile with ARM CPU target, common kernel, rootfs, uboot) • Boot solutions and server (TFTP, BOOTP) • Especially required for a quick and easy board swap during operation! • Centralized monitoring and syslog support (Kibana, BE/CO COSMOS) • Would happily use CERN CentOS 7 (CC7) if it is managed centrally, supports 32-bit, and a recent kernel is used to limit the drivers porting effort

12. Environment Document reference Today ABT has several FIDS boards connected to the GPN & TN, see cfe-86% (e as embedded): • running PetaLinux 2018.3 with a patched xlnx-kernel 4.14; • Poky rootfswith own kernel-space drivers and user applications; • limited effort done to reduce attack vector (ftpd and telnetd disabled); • boots from SD card; no common compiler, sources or boot-server! How to manage NFS mount authorization? Nmap scan, disabled busyboxftpd and telnetd: Starting Nmap 6.40 ( http://nmap.org ) at 2019-06-12 00:11 CEST Nmap scan report for cfe-867-mkbidisfids (172.18.192.199) Host is up (0.00049s latency). rDNS record for 172.18.192.199: cfe-867-mkbidisfids.cern.ch Not shown: 997 closed ports PORT STATE SERVICE VERSION 22/tcp open sshDropbearsshd 2017.75 (protocol 2.0) 53/tcp open domain dnsmasq 2.78 | dns-nsid: |_ bind.version: dnsmasq-2.78 102/tcp open iso-tsap 111/tcp open rpcbind | rpcinfo: | program version port/proto service | 100000 2,3,4 111/tcprpcbind | 100000 2,3,4 111/udprpcbind | 100024 1 42952/udp status |_ 100024 1 53763/tcpstatus Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel