felix design upgrades of detector readout meeting 9 june 2014 lorne levinson for the felix group
Download
Skip this Video
Download Presentation
FELIX Design Upgrades of detector readout meeting 9 June 2014 Lorne Levinson, for the FELIX group

Loading in 2 Seconds...

play fullscreen
1 / 18

FELIX Design Upgrades of detector readout meeting 9 June 2014 Lorne Levinson, for the FELIX group - PowerPoint PPT Presentation


  • 137 Views
  • Uploaded on

FELIX Design Upgrades of detector readout meeting 9 June 2014 Lorne Levinson, for the FELIX group. Guiding principles. Goal: a GBT to LAN “switch” with as little awareness of detector specifics as possible No user code in FELIX (so far): configuration params for options

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'FELIX Design Upgrades of detector readout meeting 9 June 2014 Lorne Levinson, for the FELIX group' - latif


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
felix design upgrades of detector readout meeting 9 june 2014 lorne levinson for the felix group

FELIX DesignUpgrades of detector readout meeting9 June 2014Lorne Levinson, for the FELIX group

Upgrades of detector readout meeting, 9 July 2014

guiding principles
Guiding principles
  • Goal: a GBT to LAN “switch” with as little awareness of detector specifics as possible
    • No user code in FELIX (so far): configuration params for options
  • Separate data movement from data processing
  • Configurable data paths from E-link to network endpoint
  • Scalable at Front end, Back end, and switch levels
  • Do as much as possible in software
  • COTS FPGA PCIe board
    • But… see talk on TTC clock recovery by Jinlong Zhang
  • Readout, so latency is not important, but provide low latency GBT-to-GBT links
  • Future proof by following COTS market, separately upgradeable components: GBT, FPGA PCIe board, serve PC, network cards
  • Routing is for E-links; they are the logical connections

Upgrades of detector readout meeting, 9 July 2014

virtex 7 pcie board
Virtex-7 PCIe board

From Global HiTech, PCIe gen 3, 8 lanes

Virtex X690T

2x12 bidir CXP connectors, FMC

Upgrades of detector readout meeting, 9 July 2014

off detector architecture with felix
Off- detector architecture with FELIX

“ROD”

Functions can now be separated, physically, or just logically.

“ROD” = Event fragment builder/processor for ROS

Upgrades of detector readout meeting, 9 July 2014

routing
Routing
  • Configured via control port on main or separate network
  • Simple/fixed:FELIX port # / E-link-range → IPaddress / portor → Infiniband local identifier / queue pair
  • E-link packet protocol:
    • Allows several logical streams to use the same E-link

 header with: stream-id + packet len (if not fixed length packets)FELIX port # / stream-id → IPaddress / port…

  • Cloning, broadcasts / multicasts possible, in either direction
  • Reliable transport from FELIX to end-point via TCPIP
  • Configurable Quality of Service per logical connection
    • Less-than-best-effort for event data for monitoring

Upgrades of detector readout meeting, 9 July 2014

use cases 1
Use cases – 1
  • Receive data from an E-link or bonded group of E-links (a channel) and route it to one or more network endpoints via TCP or UDP. Bits must be arranged into correct order.
    • Optional 8b/10b or HDLC checking and/or decoding
    • Optionally verify checksum
    • Packet boundary identification either by SOP/EOP or length, optionally there can be no packet boundary, just a byte stream.
      • If a maximum packet length is configured, data arriving after the maximum is discarded and this error is marked in the packet status word.
    • If streams are defined for this channel packets are routed according to the stream. The stream ID may contain an extension filled by the detector with part of the Level-1 ID or the Trigger Type for advanced event routing.
    • Sampling can be requested

Upgrades of detector readout meeting, 9 July 2014

use cases 2
Use cases – 2
  • Receive BUSY from an E-link command, internally from FELIX itself, or from the network. Then route it to a network endpoint or to a standard ATLAS RODBUSY signal (open collector Lemo). In Phase II it may be sent to the TTC system.
  • Receive TTC information (BC, L1A, ECR, BCR, user bits) and broadcast them to the requested E-links
    • The BC clock is supplied to the GBT because the FELIX's GBT link clock is the BC clock supplied by the TTC system to FELIX.
    • Three options: 2-bit, 4-bit, 8-bit as described in the datasheet.
  • Receive data from a TCP port and route it to one or more E-links
    • Optional 8b/10b or HDLC encoding
    • A specific multicast channel can broadcast packets to all out-going E-links with the "MC" attribute.

Upgrades of detector readout meeting, 9 July 2014

use case 3
Use case – 3
  • Scheduled data transfer lists: setting and activating (list is per GBT, but E-links are enabled by a bit map.) Open Issue: Or, several lists per FELIX and each E-link configures to use a specific list. Each item in the list is qualified by a delay to wait from the previous item in the list.
    • Activation by TTC user bit
    • Activation by command received via TCP
  • Receive data from an E-link or bonded group of E-links (a channel) and route it to a Direct Output Low Latency link (probably a GBT). Typically only a particular stream in the channel would be so routed.

Upgrades of detector readout meeting, 9 July 2014

slide11
Low latency path is

between predefined

GBT pairs.

Uses “cut-through”

Routing.

Upgrades of detector readout meeting, 9 July 2014

felix fw sw interface
FELIX: FW/SW interface

Data interface

PCIe low level packet format (fixed length)

PCIe high level packets, spans low level packets

Push/pull? Hints from RobinNP architecture

Configuration parameter interface

Need description of parameters that can generate:XML for OKS, VHDL for FPGA

See (but needs to be updated): https://twiki.cern.ch/twiki/pub/Atlas/GBT2LAN/MappingGBT_DataToNetworkEndpoints.pdf

Monitoring and statistics interface

FPGA configuration interface

IP core from HitechGlobal ~4k$

Upgrades of detector readout meeting, 9 July 2014

backup
Backup

Upgrades of detector readout meeting, 9 July 2014

functionality of e lane handlers
Functionality of E-lane handlers

From GBT: need to decode 8b/10 or HDLC data

  • Collect 2, 4, 8, 16 bits per 40MHz clock
  • Align using comma symbols
  • Decode 8b/10b, discard comma, Start-of-Packet, End-of-Packet
  • Add 1KB block headers and “chunk” trailers
    • Optionally verify checksum

To GBT: need to encode 8b/10b or HDLC data

  • Read from FIFO
  • Encode (8b/10b(?) or HDLC) 2 bytes at a time from FIFO
  • Chop into 2, 4, 8, 16 bits per 40MHz clock

Probably run at 80MHz to get extra clocks for headers/trailers

Upgrades of detector readout meeting, 9 July 2014

clock recovery
Clock recovery
  • The 2x40MHz clock comes from the TTC link, i.e. a recovered clock.
  • Probably OK for GTH’s in the same GTH block of 4 transceivers
  • ???
    • Can it be distributed to the other blocks?
    • Does the recovered clock have sufficiently low jitter for the other GTH’s?
  • We should probably consider a TTC receiver and jitter remover on an FMC card.
    • rework the GLIB version so that the FMC is on top of the PCIe mother board instead of extending beyond the PCIe board boundary

Upgrades of detector readout meeting, 9 July 2014

error handling
Error handling
  • Just beginning the considerable thought that needs to be done here.
  • Two issues:
    • Discovery of corrupted data
    • Surviving errors without out loosing event boundaries
      • This requires some effort on the front ends
  • Corruption within a GBT 80-bit word is detected by the GBT’s FEC
  • Hard to believe E-links can be mixed into each other, so expect errors within E-link channels: repeated clocks, lost clocks
  • If E-links use 8b/10b with packet framing codes, fixed length, or variable length with counts, then errors (not all) can be detected.
  • FELIX can flag these errors in the header it sends with a stream’s data

Upgrades of detector readout meeting, 9 July 2014

ad