Experiences in the Development of an FPGA Based Radiation Tolerant Design

1. Experiences in the Development of an FPGA Based Radiation Tolerant Design Mark Jones Mark.l.jones@nasa.gov 757-864-7878 Dr. Robert Klenke rhklenke@vcu.EDU (804) 827-7007

2. Gifts • Geosynchronous • Imaging • Fourier • Transform • Spectrometer • Was not funded to completion

3. Gifts Modules Modulators (Downlink) Control Module Sensor Module

4. IC (Instrument Controller) BAE 750 Gifts Control Module IO DL EDS 6U CPCI 33MHz, 32 Bit, 3.3 Volt MEM 2 MB SRAM

5. Gifts Control Module Data Flow SM Data I/F Serialized LVDS 21-bit data,16 MHz MEM IC Actel PCI Core CPCI Bus SM Command I/F 422 Differential X-Band 80 Mbps IO DLINK EDS Actel PCI Core Actel PCI Core Actel PCI Core SMQ-11 Spacecraft I/F 1553 Requires Sequential “Block” Readout to Downlink

6. MEM overall function and purpose • CM must accept SM LVDS data up to 256Mbits/sec, average 160Mbits/sec and transfer that data to the communications payload at a rate of 80Mbits/sec • This throughput requires the data must be compressed (DWNLNK) resulting in a non-continuous and variable throughput • Compression ratio achieved by the DWNLNK varies with the entropy of the current image. • Worst case is unable to match the bandwidth of the incoming data.

7. MEM overall function and purpose, continued • This mismatch of data rates between the SM and the DWNLNK output during periods of low image compression requires that the MEM be able to store incoming data from the LVDS interface until the DWNLNK is ready to process and send it • System engineering studies, using typical values of data entropy across an image, indicate that the MEM must contain at least 1Mbytes of data storage space to avoid overflow and the resulting loss of image data, therefore to provide a 2X margin, a 2Mbyte buffer is used • The MEM 2Mbytes of SRAM memory are organized as a First In First Out (FIFO) buffer.

8. Memory Board Design Flow and Tools • Actel Libro Platinum Toolset • VHDL design entry using text editor • Pre-synthesis Functional Simulation using ModelSim and highly modified Actel-supplied PCI test bench • Synthesis using Synplicity synthesis tool • Post-synthesis simulation using ModelSim • Full licensed copy of ModelSim (as opposed to the reduced performance version supplied with Libro) necessary to complete post-synthesis and post-place & route simulations in reasonable time • Place & route with Actel’s Designer tools • Timer static timing analysis tool used to check rough timing and apply timing constraints • Post-place & route timing simulation with ModelSim

9. Functional Block Diagram – Clock Domains cPCI Clock Domain Core Clock Domain (Delayed cPCI Clock) LVDS Clock Domain Actel RT54SX32S FPGA (2 – bit sliced) DMA Engine Actel RT54SX32S FPGA Actel RT54SX32S FPGA Deserializer LVDS Data LVDS FIFO 64 64 DMA FIFO Actel PCI Core 16 32 cPCI Bus SRAM Controller 64 64 BAE 238A792 SRAM (2) BAE 238A792 SRAM (2)

10. GIFTS Memory Board Architecture • 100 ohm Differential LVDS Impedance requirement • CPCI protocol includes: • Bus terminating resistors within .5 inches of cPCI connector • 65 ohm board impedance for CPCI signals • Specific trace lengths which will affect FPGA placement relative to CPCI connector • The data received at the LVDS data interface is received in a serial bit stream. The serialization however is transparent to the MEM as the output of the Deserializer is parallel. • There are 3 clock domains, the incoming LVDS, the PCI clock domain, and because the CPCI spec only allows 1 load on the CPCI clock and a rad hard(low SEU) low skew buffer or PLL was not found, the clock for the DMA Engine and SRAM controllers use a 3rd clock domain delayed from the PCI clock. • For a design independent of the amount of clock skew and not rely on place and route for proper function, this resulted in • Chip to chip asynchronous data transfer using control signals that employ 4 state handshaking • The “initiator” asserts the “Ready” signal • The “receiver” asserts the “Acknowledge” signal • The “initiator” de-asserts the “Ready” signal when done with transfer • The “receiver” de-asserts the “Acknowledge” signal indicating it is done with transfer and ready for next

11. GIFTS Memory Board Architecture • The cPCI bus has the capability to transmit one 32-bit double word (Dword) each clock cycle of the 33MHz cPCI clock. This capability of course, requires that the PCI core be supplied with a 32-bit Dword on its input queue each 30.3 ns clock cycle in order to maintain 100% bandwidth on the cPCI bus. However, the SRAM devices that were available for use in the MEM design originally had a fastest access time of 35 ns. This access time restriction required that, in order to achieve improved bus bandwidth, two Dwords must be retrieved from the SRAM and transmitted to the PCI core during a single memory read operation. Furthermore, because data will be arriving across the LVDS interface during the DMA operation, it is necessary to be able to perform both a read operation, to retrieve data for the current DMA operation, and a write operation, to store the just received LVDS data, on the SRAM concurrently. These two considerations required that the SRAM in the MEM be configured as two 64-bit banks that are 128Kbytes deep, thus allowing simultaneous read and write operations to be performed on opposite banks in a ping-pong fashion. The MEM design requires that all DMA data transfers must be performed on a 64-bit boundary (i.e., an even number of 32-bit Dwords).

12. Functional Block Diagram – Data Flow Left SRAM 128K X 64 64 LVDS Fifo SRAM Controller PCI Core 64 64 cPCI Bus LVDS Data 64 128K X 64 Right SRAM

13. Functional Block Diagram – Data Flow Left SRAM 128K X 64 64 LVDS Fifo SRAM Controller PCI Core 64 64 cPCI Bus LVDS Data 64 128K X 64 Right SRAM

14. Functional Block Diagram – Data Flow Left SRAM 128K X 64 64 LVDS Fifo SRAM Controller PCI Core 64 64 cPCI Bus LVDS Data 64 128K X 64 Right SRAM

15. Functional Block Diagram – Data Flow Left SRAM 128K X 64 64 PCI Core LVDS Fifo PCI Core 64 64 SRAM Controller LVDS Data cPCI Bus 128K X 64 128K X 64 w0 w1 w2 w3 w4 w5 w6 w7 Right SRAM w0 w1 w2 w3 w4 w5 w6 w7 w8 w9 w10 w11 Right SRAM Left SRAM LVDS Fifo 128K X 64 [w3 w2], [w1 w0] [w7 w6], [w5 w4]; [w3 w2], [w1 w0] … w7, w6, w5, w4, w3, w2, w1, w0 64

16. GIFTS Memory Board Architecture • Since Actel PCI core IP was designed for a standard memory module, all of the data that is to be DMAed must be available (otherwise the Core is starved for data). This is not the case with this design since at any given time the IC does not know exactly how much data is available for DMA transfer. Therefore, it was necessary to design a “DMA engine” into the MEM that can keep track of how much data has been received by the MEM and stored in the SRAM, and configure and initiate the individual DMA operations in the cPCI core until the requested total amount of data has been transferred to the destination across the cPCI bus.

17. GIFTS Memory Board Architecture • Bit-slicing SRAM Controller Functionality • 64 bit data paths as a result of memory speed and throughput • However cannot be implemented in one RT54SXS due to I/O limitations • Results in SRAM Controller Core bit-sliced across 32-bit boundaries

18. Bit-Sliced FPGA Functional Block Diagram Left SRAM 128K X 64 32 64 Left SRAM Controller 32 32 LVDS Fifo PCI Core 64 64 Right SRAM Controller 32 32 64 32 128K X 64 Right SRAM

19. GIFTS Memory Board Architecture • Requires 4 Actel RT54SX32S FPGAs • SRAM Controller Core bit-sliced into 2 FPGAs • Based on total cell utilizations for • PCI CORE FGPA was 69% • Left Sram Controller was 67% • Right Sram Controller was 53% • LVDS FIFO was 72% • PCI Core and LVDS FIFO each implemented on a separate FPGA resulting in 4 FPGAs

20. Block Diagram BAE 238A792 SRAM (2) Core Clock Domain (Delayed cPCI Clock) cPCI Clock Domain LVDS Clock Domain 32 64 Actel RT54SX32S Actel RT54SX32S FPGA Actel RT54SX32S FPGA 32 32 Left SRAM/ DMA Controller Deserializer LVDS Data LVDS FIFO DMA FIFO Actel PCI Core 64 64 16 32 cPCI Bus Actel RT54SX32S Right SRAM Controller 32 32 64 32 BAE 238A792 SRAM (2)

21. MEM Design Status • Initial prototype design completed • Limitations imposed by available FPGA size, speed, and I/O pins overcome • Limitations imposed by Actel-supplied PCI Core overcome • Limitations imposed by available SRAM speed overcome • Complexity of design resulting from 3-asynchronous clock domains overcome • Initial brassboard completed and tested • Demonstrated fundamental design concept • Successfully completed reliable DMA of data from LVDS stimulator to PCI-resident memory • Uncovered several significant bugs and needed additional functionality • Successfully redesigned and simulated at the behavioral and post-synthesis level • Revised version of the design will work with existing PCB without redesign

22. MEM Rev. 2 Design to-be-completed • Complete timing layout and test in post layout simulation with back annotated timing • Test on the brassboard • Complete documentation of the design

23. Backup Backup Slides

24. Brassboard Layout • Layout details • CPCI requirements met for terminating resistor placement and trace lengths to the PCI FPGA. • CPCI 65 ohm Impedance requirement met on all signal layers facilitated by stackup design and trace width and separation relationship • 30mils separation between asynchronous CPCI signals and others • 100 ohm Differential LVDS Impedance requirement met on all signal layers facilitated by stackup design and trace width and separation relationship, and custom thru-hole arrangement for Micro_Twinax cable connection • Differential LVDS signals surrounded by ground guard plane • spaced 20mils (2 x distance between pair) away from pairs • Clocks are routed internal to the board, between planes, and have ground guard traces on both sides. • Edges of Board will be milled to allow 90mil board to slide into 62mil guides • Layout approximately 99% complete

25. Brassboard Layout

26. Brassboard Layout – Stackup Layer 1,9 65 Ohms: 5 mil wide/10 mil sep. 100 Ohms Diff. 7 mil wide/10 mil sep. Layer 3, 6 65Ohms: 5 mil wide/10 mil sep. 100 Ohms Diff. 7 mil wide/12 mil sep.