the lhcb event builder n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
The LHCb Event-Builder PowerPoint Presentation
Download Presentation
The LHCb Event-Builder

Loading in 2 Seconds...

play fullscreen
1 / 16

The LHCb Event-Builder - PowerPoint PPT Presentation


  • 84 Views
  • Uploaded on

The LHCb Event-Builder. Markus Frank, Jean-Christophe Garnier , Clara Gaspar, Richard Jacobson, Beat Jost , Guoming Liu, Niko Neufeld , CERN/PH 17 th Real-Time Conference Instituto Superior Tecnico Lisboa 2010. Architecture. Detector. VELO. ECal. HCal. Muon. RICH. ST. OT.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'The LHCb Event-Builder' - haruki


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
the lhcb event builder

The LHCb Event-Builder

Markus Frank, Jean-Christophe Garnier, Clara Gaspar, Richard Jacobson, Beat Jost, Guoming Liu, Niko Neufeld, CERN/PH

17th Real-Time Conference

Instituto Superior Tecnico

Lisboa 2010

architecture
Architecture

Detector

VELO

ECal

HCal

Muon

RICH

ST

OT

L0 Trigger

L0 trigger

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

TFC System

LHC clock

x 313

Readout Board

Readout Board

Readout Board

Readout Board

Readout Board

Readout Board

Readout Board

Front-End

MEP Request

Main Router

Event building

Edge

Router

Edge

Router

Edge

Router

Edge

Router

Edge

Router

Edge Router

SWITCH

CASTOR

x 50

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

STORAGE

HLT farm

Event data

Timing and Fast Control Signals

Control and Monitoring data

The LHCb Eventbuilder - RealTime 2010 - Niko Neufeld

key parameters
Key Parameters

The LHCb Eventbuilder - RealTime 2010 - Niko Neufeld

physical installation
Physical Installation
  • More than 4000 UTP Cat 6 patch connections
  • 3 floors of ~ 80 m2
    • 3rd (top) read-out boards,
    • 2nd (middle): DAQ network,
    • 1st (bottom):event-filter farm
  • Cat 6 patch-cords RJ45
  • High-density RJ21 6-port connectors for main-router

3 readout boards

2 DAQ network

1 event-filter farm

The LHCb Eventbuilder - RealTime 2010 - Niko Neufeld

the read out board tell1
The Read-out Board (“TELL1”)

Preprocessing

Feature Extraction

Zero-suppression

Event fragment merging

FDAQ framing

9.6 Gbit/s

12 * 1.6 Gbit/s

9.6 Gbit/s

To DAQ 4 Gbit/s

Frame transmission

4 x 1000 BaseT

The LHCb Eventbuilder - RealTime 2010 - Niko Neufeld

the protocol
The Protocol

Ether header

IP header

MEP

  • Event fragments from up to 16 triggers packed into one Multi-Eventfragment Packet (MEP)
  • A MEP must fit into one IPv4 packet (64 kB max!)
  • IPv4 packets will be fragmented into Ethernet frames of MTU size
  • MEP header is 12 bytes only (Event-number + length)
  • IP address information is used in event-building
  • Unreliable / no duplication / no re-transmission

Payload

Header

Event 1

Ether header

IP header

Event 2

Payload

Event n

Ether header

IP header

Payload

The LHCb Eventbuilder - RealTime 2010 - Niko Neufeld

dataflow in lhcb
Dataflow in LHCb

3

2

1

Readout Supervisor tells readout boards where events must be sent (round-robin)

Readout boards do not buffer, so switch must

Event Builders inform Readout supervisor about availability

Switch Buffer

Event Builder 1

EB1 available for events

Event Builder 2

Data Acquisition

Switch

EB2 available for events

“Send next event to EB1”

“Send next event to EB2”

Readout

Supervisor

7

The LHCb Eventbuilder - RealTime 2010 - Niko Neufeld

the many sources of packet loss
The many sources of packet loss
  • Packet corruption (normally < 10-12) – this usually is a badly handled overflow in the FPGAs (very rare now)
    • Depending on the corruption these events may be dropped already in the network
  • Packet loss due to buffer overflow (ingress, egress) in core-router, edge-switches or farm-node (between 10-9 and 10-11)
    • This packet loss often occurs in bursts
  • Packet loss in core-router (silent!) < 10-13
    • could only be fixed in session with company engineers

The LHCb Eventbuilder - RealTime 2010 - Niko Neufeld

diagnostic or what the shifter saw
Diagnostic: or “What the shifter saw”

The LHCb Eventbuilder - RealTime 2010 - Niko Neufeld

event assembly buffering in farm node
Event assembly / buffering in farm -node
  • IP packets re-assembled by OS
  • Event-building process (MEPRx) puts data in shared buffer
  • Monitoring in kernel difficult: need good (and generous) tuning!

Shared Memory

memcpy

MEPRx process

User-space 

OS-kernel 

socket-buffer

DMA buffers(sk_buf)

DMA

Main memory 

Network Interface

Card (NIC)

Frame buffer in NIC

The LHCb Eventbuilder - RealTime 2010 - Niko Neufeld

meprx resource usage
MEPRx resource usage

% CPU of one Intel 5420 core (a 4 core processor running at 2.5 GHz

MEM is resident memory (i.e. pages locked in RAM)

Precision of measurements is about 10% for CPU and 1% for RAM

The LHCb Eventbuilder - RealTime 2010 - Niko Neufeld

buffer usage
Buffer Usage

Buffer usage in core-router with a test using 270 sources @ 350 kHz event-rate

  • 256 MB shared between 48 ports
  • 17 ports used as “output”
  • Non-uniformity under investigation

The LHCb Eventbuilder - RealTime 2010 - Niko Neufeld

link aggregation
Link Aggregation
  • IEEE 802.3ad (now actually IEEE 802.1AX)
  • Does not define how traffic is balanced over multiple links
  • Available balancing algorithms are made to preserveframe-order
  • Typically use a hash out of destination and source Ethernet and/or IP address
    •  all links will be used when different destinations sent to the same host

Edge-

Router

1 Gbit/s

Core-Router

6 x 1 Gbit/s

1 Gbit/s

The LHCb Eventbuilder - RealTime 2010 - Niko Neufeld

clock tolerance in ieee 802 3ab
Clock Tolerance in IEEE 802.3ab

802.3ab transmits on 5 levels over

4 twisted pairs @ 125 MHz

Inteframe gap (IFG) is 96 bits

  • 40.6.1.2.6 Transmit clock frequencyThe quinary symbol transmission rate on each pair of the master PHY shall be 125.00 MHz ± 0.01%.
  • 40.6.1.3.2 Receiver frequency toleranceThe receive feature shall properly receive incoming data with a 5-level symbol rate within the range 125.00 MHz ± 0.01%.

Frame 313

IFG

Frame 312

IFG

Frame 1

IFG

Frame loss ~ 10-8

Edge-

Router

Core-Router

farm-node

The LHCb Eventbuilder - RealTime 2010 - Niko Neufeld

not covered
Not covered
  • Run-control (see talk by Clara Gaspar)
  • Monitoring
  • Process Management in farm-nodes (see poster of Juan Caicedo)
  • Event aggregation & storage (see poster of Jean-Christophe Garnier)

The LHCb Eventbuilder - RealTime 2010 - Niko Neufeld

conclusions
Conclusions
  • A (almost) pure push protocol for a 40 Gigabyte/s DAQ works
    • With very good hardware
    • And a lot of hard work
  • More than 3000 UTP links are no problem and you do not need TCP
  • Commercial will most of time not work off the shelf
  • We will now go on to see how it works at 40 MHz (1.2 Terabyte/s)

The LHCb Eventbuilder - RealTime 2010 - Niko Neufeld