Felix design upgrades of detector readout meeting 9 june 2014 lorne levinson for the felix group
This presentation is the property of its rightful owner.
Sponsored Links
1 / 18

FELIX Design Upgrades of detector readout meeting 9 June 2014 Lorne Levinson, for the FELIX group PowerPoint PPT Presentation


  • 86 Views
  • Uploaded on
  • Presentation posted in: General

FELIX Design Upgrades of detector readout meeting 9 June 2014 Lorne Levinson, for the FELIX group. Guiding principles. Goal: a GBT to LAN “switch” with as little awareness of detector specifics as possible No user code in FELIX (so far): configuration params for options

Download Presentation

FELIX Design Upgrades of detector readout meeting 9 June 2014 Lorne Levinson, for the FELIX group

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Felix design upgrades of detector readout meeting 9 june 2014 lorne levinson for the felix group

FELIX DesignUpgrades of detector readout meeting9 June 2014Lorne Levinson, for the FELIX group

Upgrades of detector readout meeting, 9 July 2014


Guiding principles

Guiding principles

  • Goal: a GBT to LAN “switch” with as little awareness of detector specifics as possible

    • No user code in FELIX (so far): configuration params for options

  • Separate data movement from data processing

  • Configurable data paths from E-link to network endpoint

  • Scalable at Front end, Back end, and switch levels

  • Do as much as possible in software

  • COTS FPGA PCIe board

    • But… see talk on TTC clock recovery by Jinlong Zhang

  • Readout, so latency is not important,but provide low latency GBT-to-GBT links

  • Future proof by following COTS market, separately upgradeable components:GBT, FPGA PCIe board, serve PC, network cards

  • Routing is for E-links; they are the logical connections

Upgrades of detector readout meeting, 9 July 2014


Virtex 7 pcie board

Virtex-7 PCIe board

From Global HiTech, PCIe gen 3, 8 lanes

Virtex X690T

2x12 bidir CXP connectors, FMC

Upgrades of detector readout meeting, 9 July 2014


Off detector architecture with felix

Off- detector architecture with FELIX

“ROD”

Functions can now be separated, physically, or just logically.

“ROD” = Event fragment builder/processor for ROS

Upgrades of detector readout meeting, 9 July 2014


Felix design upgrades of detector readout meeting 9 june 2014 lorne levinson for the felix group

Upgrades of detector readout meeting, 9 July 2014


Routing

Routing

  • Configured via control port on main or separate network

  • Simple/fixed:FELIX port # / E-link-range → IPaddress / portor → Infiniband local identifier / queue pair

  • E-link packet protocol:

    • Allows several logical streams to use the same E-link

       header with: stream-id + packet len (if not fixed length packets)FELIX port # / stream-id → IPaddress / port…

  • Cloning, broadcasts / multicasts possible, in either direction

  • Reliable transport from FELIX to end-point via TCPIP

  • Configurable Quality of Service per logical connection

    • Less-than-best-effort for event data for monitoring

Upgrades of detector readout meeting, 9 July 2014


Use cases 1

Use cases – 1

  • Receive data from an E-link or bonded group of E-links (a channel) and route it to one or more network endpoints via TCP or UDP. Bits must be arranged into correct order.

    • Optional 8b/10b or HDLC checking and/or decoding

    • Optionally verify checksum

    • Packet boundary identification either by SOP/EOP or length, optionally there can be no packet boundary, just a byte stream.

      • If a maximum packet length is configured, data arriving after the maximum is discarded and this error is marked in the packet status word.

    • If streams are defined for this channel packets are routed according to the stream. The stream ID may contain an extension filled by the detector with part of the Level-1 ID or the Trigger Type for advanced event routing.

    • Sampling can be requested

Upgrades of detector readout meeting, 9 July 2014


Felix design upgrades of detector readout meeting 9 june 2014 lorne levinson for the felix group

Upgrades of detector readout meeting, 9 July 2014


Use cases 2

Use cases – 2

  • Receive BUSY from an E-link command, internally from FELIX itself, or from the network. Then route it to a network endpoint or to a standard ATLAS RODBUSY signal (open collector Lemo). In Phase II it may be sent to the TTC system.

  • Receive TTC information (BC, L1A, ECR, BCR, user bits) and broadcast them to the requested E-links

    • The BC clock is supplied to the GBT because the FELIX's GBT link clock is the BC clock supplied by the TTC system to FELIX.

    • Three options: 2-bit, 4-bit, 8-bit as described in the datasheet.

  • Receive data from a TCP port and route it to one or more E-links

    • Optional 8b/10b or HDLC encoding

    • A specific multicast channel can broadcast packets to all out-going E-links with the "MC" attribute.

Upgrades of detector readout meeting, 9 July 2014


Use case 3

Use case – 3

  • Scheduled data transfer lists: setting and activating (list is per GBT, but E-links are enabled by a bit map.) Open Issue: Or, several lists per FELIX and each E-link configures to use a specific list. Each item in the list is qualified by a delay to wait from the previous item in the list.

    • Activation by TTC user bit

    • Activation by command received via TCP

  • Receive data from an E-link or bonded group of E-links (a channel) and route it to a Direct Output Low Latency link (probably a GBT). Typically only a particular stream in the channel would be so routed.

Upgrades of detector readout meeting, 9 July 2014


Felix design upgrades of detector readout meeting 9 june 2014 lorne levinson for the felix group

Low latency path is

between predefined

GBT pairs.

Uses “cut-through”

Routing.

Upgrades of detector readout meeting, 9 July 2014


Felix fw sw interface

FELIX: FW/SW interface

Data interface

PCIe low level packet format (fixed length)

PCIe high level packets, spans low level packets

Push/pull? Hints from RobinNP architecture

Configuration parameter interface

Need description of parameters that can generate:XML for OKS, VHDL for FPGA

See (but needs to be updated): https://twiki.cern.ch/twiki/pub/Atlas/GBT2LAN/MappingGBT_DataToNetworkEndpoints.pdf

Monitoring and statistics interface

FPGA configuration interface

IP core from HitechGlobal ~4k$

Upgrades of detector readout meeting, 9 July 2014


Felix design upgrades of detector readout meeting 9 june 2014 lorne levinson for the felix group

Upgrades of detector readout meeting, 9 July 2014


Felix design upgrades of detector readout meeting 9 june 2014 lorne levinson for the felix group

Upgrades of detector readout meeting, 9 July 2014


Backup

Backup

Upgrades of detector readout meeting, 9 July 2014


Functionality of e lane handlers

Functionality of E-lane handlers

From GBT: need to decode 8b/10 or HDLC data

  • Collect 2, 4, 8, 16 bits per 40MHz clock

  • Align using comma symbols

  • Decode 8b/10b, discard comma, Start-of-Packet, End-of-Packet

  • Add 1KB block headers and “chunk” trailers

    • Optionally verify checksum

      To GBT: need to encode 8b/10b or HDLC data

  • Read from FIFO

  • Encode (8b/10b(?) or HDLC) 2 bytes at a time from FIFO

  • Chop into 2, 4, 8, 16 bits per 40MHz clock

    Probably run at 80MHz to get extra clocks for headers/trailers

Upgrades of detector readout meeting, 9 July 2014


Clock recovery

Clock recovery

  • The 2x40MHz clock comes from the TTC link, i.e. a recovered clock.

  • Probably OK for GTH’s in the same GTH block of 4 transceivers

  • ???

    • Can it be distributed to the other blocks?

    • Does the recovered clock have sufficiently low jitter for the other GTH’s?

  • We should probably consider a TTC receiver and jitter remover on an FMC card.

    • rework the GLIB version so that the FMC is on top of the PCIe mother board instead of extending beyond the PCIe board boundary

Upgrades of detector readout meeting, 9 July 2014


Error handling

Error handling

  • Just beginning the considerable thought that needs to be done here.

  • Two issues:

    • Discovery of corrupted data

    • Surviving errors without out loosing event boundaries

      • This requires some effort on the front ends

  • Corruption within a GBT 80-bit word is detected by the GBT’s FEC

  • Hard to believe E-links can be mixed into each other, so expect errors within E-link channels: repeated clocks, lost clocks

  • If E-links use 8b/10b with packet framing codes, fixed length, or variable length with counts, then errors (not all) can be detected.

  • FELIX can flag these errors in the header it sends with a stream’s data

Upgrades of detector readout meeting, 9 July 2014


  • Login