1 / 44

Level-1 Trigger Upgrade Status

Level-1 Trigger Upgrade Status. Wesley Smith, U. Wisconsin US CMS Project Management Group December 18, 2013. L5 Active Mu. Trig. Milestones - Rice. L5 - MPCM Software and Firmware Start 401.04.03.02.01 2013-11-01 Initial firmware version is ready, software for test stand is being built.

roman
Download Presentation

Level-1 Trigger Upgrade Status

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Level-1 Trigger Upgrade Status Wesley Smith, U. Wisconsin US CMS Project Management Group December 18, 2013 PMG: Trigger Upgrade Status

  2. L5 Active Mu. Trig. Milestones - Rice • L5 - MPCM Software and Firmware Start 401.04.03.02.01 2013-11-01 • Initial firmware version is ready, software for test stand is being built. • L5 - MPCM Production Start 401.04.03.02.02 2013-11-01 • Started, 2 pre-production mezzanines are at Rice and being tested. All the parts except optical transmitters are at Rice. Should be ready to start assembly in a few weeks. • L5 -Start MPC-EMUTF Optical Fibers 401.04.03.03 2013-11-01 • Short optical pigtails for MPC boards are at Rice. Out of 36 trunk cables for the EMUTF 27 have been delivered to CERN and tested there. The remaining longest 100+ m cables are expected from Draca in January. Installation at p.5 is planned for spring. • L5 -Start Muon Sorter 401.04.03.06 2013-11-01 • Preliminary work on hardware design started. • L5 -Start Trigger Production 401.04.03.06 2013-11-01 • Started, covered above • L5-Start MSM Production 401.04.03.06.02 2013-11-01 • Not started yet PMG: Trigger Upgrade Status

  3. Muon Port Card Status (1) ■ 85 optical transmitter boards have been fabricated and assembled in October ■ Received 90 optical pigtails from CERN in November ■ Fabricated 81 new front panels ■ All parts for 80 mezzanine boards have been ordered and most of them are already at Rice. Optical transmitters from Avago should arrive in the second half of December. PMG: Trigger Upgrade Status

  4. Muon Port Card Status (2) • Implemented few minor layout changes in the mezzanine design. • Two pre-production boards have been assembled by Pactron in the middle of November and are being tested at Rice. PMG: Trigger Upgrade Status

  5. Muon Port Card Mezzanine Plans • Complete testing of the two pre-production mezzanines by the end of December • Assemble 80 production boards in the 1st half of January • Test 80 new mezzanines at Rice in January-March • Set up a uTCA test stand at Rice in January-March • Installation of 36 optical trunk cables at p.5: spring 2014 • Installation of the upgraded Muon Port Cards at p.5: late spring- early summer 2014 PMG: Trigger Upgrade Status

  6. L5 Active Mu. Trig. Milest. - Florida • L5-Start EMUTF Test, Install, and Commission 401.04.03.04.03 2013-11-04 • A control PC of the type recommended by CERN was ordered for tests in Florida • L5 -Start EMUTF Infrastructure 401.04.03.05 2013-11-01 • A control PC of the type recommended by CERN was ordered for tests in Florida • First Optical Patch Panel Prototype delivered (next slide) • L5 -Start EMUTF Infrastructure Production 401.04.03.05.01 2013-11-01 • Started, covered above • L5-Start EMUTF Software and Firmware 401.04.03.04.01 2013-11-04 • A first version of the software emulator (for the initial endcap TF algorithm) has started. It is nearly ready to be used in CMSSW, but likely will occur in early 2014.  • Initial firmware for all MTF cards (FPGA control board, optics board, and PT LUT mezzanine board) has been written and already used for testing of the prototypes. • L5-Start EMUTF Production 401.04.03.04.02 2013-11-04 • The preproduction of the MTF7 board with Virtex-7 has started with the submission for PCB manufacture this December, and components procured. PMG: Trigger Upgrade Status

  7. EndcapMuon TF block diagram Best 3 Muonsin each sector (36 total) To GMT From MPCs 60 12-corefibers, 8 cores used in each. 90 trig. primitives per 60° sector 3.2 Gbps Sector Processors 12 units 60° sector each Optical plant (fanouts and splitters) From RPC Up to 216 fibers at 1.6 Gbps (may be concentrated to higher bandwidth and fewer fibers) To Overlap Track Finder PMG: Trigger Upgrade Status

  8. EMUTF Optical Patch Panel Prototype patch panel to split, fan-out and bundle fibers to appropriate processor inputs. Units are only 1 U wide. PMG: Trigger Upgrade Status

  9. uTCA chassis • Sector Processor (SP) • Each occupies 2 uTCA slots • 12 units in system • All chassis use AMC13 (designed by Boston University) • Clocking, TTC, and DAQ • 3 units • Plan to control boards via PCI express for development (faster) • But will be compatible with IPbus as well for operations per CMS standard • IP-based protocol for controlling hardware devices adopted by CMS. Chassis #1 SP SP SP SP SP Chassis #2 SP SP SP SP SP Chassis #3 SP SP PMG: Trigger Upgrade Status

  10. Modular Muon Track Finder (MTF) Custom backplane Optical module Core logic module Pt LUT module Custom backplane connector Core logic module Optical module PMG: Trigger Upgrade Status • Modularity makes future partial upgrades easier (e.g. FPGA) • Card layout is simpler • Power density is less

  11. Optical module Optical transmitters (2 out of 3 installed) Tested, Done! Works with either V6 or V7 MTF versions Custom backplane connector Optical receivers (2 out of 7 installed) Backplane redrivers uTCA connector MMC PMG: Trigger Upgrade Status

  12. MTF6 Core logic module (Virtex6) Tested Custom backplane connector PT LUT module connector Control FPGA JTAG Core logic FPGA FMM connector Control FPGA SD card connector MMC USB console uTCA connector MMC JTAG 1Gb FLASH Main FPGA firmware storage MMC CPU Estimated power consumption: ~50 W (assuming FPGAs nearly full) PT LUT mezzanine not included MMC = Module Management Controller PMG: Trigger Upgrade Status

  13. MTF7 design (Virtex-7 version) PMG: Trigger Upgrade Status • Core logic module • FPGAs: • Core logic: XC7VX690T-FFG1927 or XC7VX550T-FFG1927 • Control: XC7K70-FB676 • Inputs: • 80 Virtex-7 GTH links (10 Gbps) directly to Core FPGA • Minimal latency • All available receivers are designated for trigger data • Maximum flexibility • 4 additional Kintex GTX links (10 Gbps) to Control FPGA • Delivered to Core FPGA via parallel channel • Longer latency • Outputs: • 28 Virtex-7 GTH links (10 Gbps) • Control: • PCI express Gen 2, 2 lanes • IPbus

  14. MTF7 Base board Main FPGA PT LUT connector Low-Dropout Linear regulators Custom backplane connector DC-DC converters (switchers) Control FPGA uTCA connector MMC • Layout finished • RFQ for board production submitted • Assembled boards should arrive by end of December PMG: Trigger Upgrade Status

  15. PT LUT module Tested! Works with either V6 or V7 MTF core logic modules Investigating further latency reduction Clock synthesis and distribution Base board connector Glue logic FPGA (Spartan-6) RLDRAM3 memory 16 chips, 8 on each side (clamshell topology) Total size: 512M x 18 bits ≈ 1GB Upgrade possible to 2 GB with bigger RLDRAM3 chips (no board redesign) DC-DC converters PMG: Trigger Upgrade Status

  16. Optical communication test Eye pattern @10 Gbps. GTH receiver input. PMG: Trigger Upgrade Status • @3.2 Gbps • 47 input channels • Transmission from: • Loopback • Muon Port Card • Earlier VME prototype (2010) • For MPC and VME prototype clock was synchronized with VME crate • Twisted pair LVDS connection to uTCA backplane • @10 Gbps • 6 input channels • Asynchronous clock • Transmission from: • Loopback • Earlier 10Gbps prototype (2006) • Results • Zero errors for hours

  17. Mimicking real system setup: Sending data from 2 MPCs to MTF6 Each MPC clocked by separate CCB MTF6 receives clock from AMC13 CCBs and AMC13 receive clock from common source (TTCvi) Types of tests: PRBS Random data via test FIFOs Latest MPC data format used Result: No errors. Integration tests PMG: Trigger Upgrade Status

  18. Optical components tests Couplers MTF6 MPC or MTF6 63-meter trunk cable (2 lengths) Endcap TF needs: 2-way splitters: 348 4-way splitters: 60 Rate: 3.2 Gbps Fanout Fanout Splitters PMG: Trigger Upgrade Status • Trunk cable identical to what will be used for Endcap TF • Total length (126 m) exceeds actual max fiber length in Endcap TF system (113 m) • Results: • @3.2 Gbps: no errors with 2-way and 4-way splitting [CSC signals] • @1.6 Gbps: 2-way splitting OK, errors with 4-way [RPC signals] • Newer optical receivers lose efficiency at low bitrates

  19. PT LUT tests PMG: Trigger Upgrade Status • Parameters: • RLDRAM clock : 200 MHz • Address & control: 200 Mbps each bit • Data: 400 Mbps each bit • RLDRAM can tolerate up to ~1GHz clock. However: • Hard to implement in FPGA • Needed for burst-oriented applications mostly • Does not change latency for random address access • Lower clk F  lower power consumption • Tests performed (random data, full 1GB space): • Writing into consecutive addresses • Reading from consecutive addresses • Reading from random addresses • No errors detected • Except soldering defects in one RLDRAM chip

  20. L5 Active Cal. Trig. Milest. – Wisc. • L5-Start Calorimeter Trigger Processor 401.04.04.02 2013-11-01 • Two Prototype Boards delivered this week • L5-Start CTP7 Software and Firmware 401.04.04.02.01 2013-11-01 • Initial SW and FW for Prototype Operation is being prepared • L5 -Start CTP7 Production 401.04.04.02.02 2013-11-01 • First two prototype PCB delivered, parts received and assembly underway (see next slides) • Ordered and received 8 Virtex-7 FPGAs. • L5 -Start CIOx Card 401.04.04.03 2013-11-01 • Prototype CIOx card made and is adequate for use. • L5 -Start CIOx Software 401.04.04.03.01 2013-11-01 • Included in CTP6 software package • L5 -Start CIOx Production 401.04.04.03.02 2013-11-01 • Production needed in 2016. Will reschedule. • L5 -Start CTP Infrastructure 401.04.04.04 2013-11-01 • Have 3 Vadatech Crates and optical fibers for interconnects in hand. • L5 -Start oRM 401.04.04.05 2013-11-01 • oRM Production underway (not US cost – this item is personnel for installation) • L5 -Start oSLB-oRM Optical Fibers 401.04.04.06 2013-11-01 • Fibers have been purchased (not US cost – this item is personnel for installation) PMG: Trigger Upgrade Status

  21. Current & Stage 1 Cal. Trigger • Stage 1 Calorimeter Trigger Upgrade • Improved processing of current RCT with new oRSC • Converts current system to optical • New HF (fwdcalo) data is available to upgrade path • Prepare for the Stage 2 Upgrade • Layer 1 cards will also read out RCT in Stage 1 PMG: Trigger Upgrade Status

  22. oRSC • Two oRSCs are at 904 since mid-October • Integration Row & Lab Test Crate • Continuing to validate FPGA builds at UW • Finishing up VME access now • New oRSC code to test VME • VME-Spartan BE-Kintex FE working • Integrating oRSC into Trigger Supervisor • Using new oRSCcode PMG: Trigger Upgrade Status

  23. Stage 2 Calo Trigger – Layer 1 • Layer 1 cards will receive the calorimeter trigger primitives via optical fiber from • ECAL TCCs with oSLB 4.8 Gbps optical to oRM on legacy RCT • To be installed this winter • Send duplicate to layer-1 • HF and HCAL mHTRs • Passive optical splitting before HTRs for deployment of mHTR in parallel PMG: Trigger Upgrade Status

  24. CTP7 Concept 1V 30A Supply 3.3V Supply 1.5V Supply • Main Layer-1 processor card • Virtex-7 690T for processing • ZYNQ for TCP/IP + Linux • 60 10Gbps optical Input links • 36 10Gbps opticalOutput links • Function: Find ET clusters & transmit to Layer-2 • Card Count = 46 • 36 total + 8 spares + 2 test setups CXP Module 12Tx + 12 Rx 2.5V Supply CXP Module 12Tx + 12 Rx Virtex-7 VX690T FPGA 12X Rx ZYNQ XC7Z030 EPP 12X Rx CXP Module 12Tx + 12Rx (CTP-6 BG View) PMG: Trigger Upgrade Status

  25. Calorimeter Trigger Processor Prototype for Layer-1 • 4 Rev. A and 8 Rev. B CTP6s have been built • 48 input links in 4 Avago AFBR-820B Rx Modules • 12 output links in 1 Avago AFBR-810B Tx Modules • Back-end & Front-end FPGA XC6VXH 250T/380T • Dual SDRAM for DAQ and TCPIP buffering • Custom power modules working at 1.0V, 1.5V, & 2.5V • Embedded Microblaze processor with Linux on back-end FPGA • Custom MMC for control • Phased power-supply enables • Add’l analog voltage, power good and FPGA Load Done sensors • Parallel FPGA Flash - ~6 seconds to load both FPGAs • SI5338 Prog. Clock Synthesizer, multiple options for Reference clocks • 4.8 Gbps: 120, 240, 300 MHz options • 6.4 Gbps: 160, 320, 400 MHz options • Nelco 4000-13 SI laminate • Lower losses & better εr for 50Ω trace geometry PMG: Trigger Upgrade Status

  26. CTP6 Layer-1 Prototype JTAG USB Interface Power Supplies MMC Avago Rx (4) FE FPGA BE FPGA Dual SDRAM For DAQ & TCP/IP AvagoTx PMG: Trigger Upgrade Status

  27. CTP6 Back Side Inter-FPGA Links 12X FrontpanelTx Link Source-Select Muxes (Front or Back-End FPGA) Link Clock Fanout & Distribution PMG: Trigger Upgrade Status

  28. UW Test Crate (Final system: 3 crates w/ 12 CTP7 ea. + 2 test setups + spare = 6) TTC Downlink U. Wisconsin designed backplane with dense card interconnects manufactured & installed in commercial Vadatech VT892 Crate available in Vadatech Catalog BU AMC13 UW CTP-6 UW CTP-6 UW Aux Vadatech MCH PMG: Trigger Upgrade Status

  29. CTP6 Status • CTP6 inter-board link tests show wide margins • HF from mHTR to CTP6 integration successful • 12 links tested at 6.4 Gbps – no errors • RAM based data between • One CTP6 at 904 to continue SW work, HW Integration PMG: Trigger Upgrade Status

  30. oRSC to CTP6 Integration Test • Data patterns successfully transferred from the oRSC, captured on the CTP6 and readout via embedded Linux • Fiber capture RAM readout • SW controlled capture • Capture RAM clears on each capture • All six links capture expected pattern from oRSC • Automatically checked by host PMG: Trigger Upgrade Status

  31. CTP6 and 7 Status • CTP6 at 904 at CERN since Mid-october • Checked out • Necessary tools work • Most effort with oRSC at the moment • CTP7 prototype board in UW lab, post-production final assembly • Installation of CXP modules – new innovation – see next slides PMG: Trigger Upgrade Status

  32. Avago AFBR-83 CXP Module • Conforms to 100GbE and 12x10Gbps Infiniband standards • 12 optical lanes in each direction • MPO-24 optical connector • Multi-rate capable from 1 Gbps to 10.5 Gbps • 850 nm wavelength • Single 3.3V supply • Uses same AvagoMicroPOD engine that is used in the AvagoMiniPODs • Optically compatible with Wisconsin oRSC Card and Imperial College MP7 Card • Pluggable frontpanel interface means module can be replaced without a card/crate PCB extraction cycle PMG: Trigger Upgrade Status

  33. CXP Module, cont’d 12-fiber Ribbon to Optical Tx Engine AC coupling caps (external caps not req’d) Optical Tx Board MicroPod Optical Rx Interface Optical Rx Board MicroPodTx Optical Engine (bottom side) PMG: Trigger Upgrade Status

  34. CXP Cage Press-Fit Insertion Operation • Post-reflow operation • 84 electrical pins, 4 screws and 2 alignment pins per housing • Electrical pins are press-fit • Requires a tool for properly force application, and a fixture for bottom side board support • Designed and made a fixture for the CTP7 Tool centers insertion force over electrical pins Molex Insertion Tool Molex I-Pass Plus EF Connector P/N 170465-0001 Fixture CTP7 PCB PMG: Trigger Upgrade Status

  35. CTP7 CXP Board Hole Pattern • 84 Electrical holes are spec’d for 14.6 mils, ±2 mils • Finished hole diameter spec should be given to the PCB vendor, letting them work backwards from there • PCB Decal has ringed border to chassis ground Ground Vias (Smaller Holes) Electrical Connection Press-Fit Holes Alignment & Screw Retaining Holes CTP7 PCB PMG: Trigger Upgrade Status

  36. CTP7 CXP Press-In Fixture • Milled from ¾” aluminum stock • For use after components are soldered on to the board • Clearances for bottom side components • Provides critical board support immediately under the 84 electrical pins • Custom-designed to match CTP7 board design Kapton tape for anti-gouge protection Direct board support immediately under the CXP Connector electrical Holes (3 places) Relieved for Bottom Side Component Clearance Alignment Pin PMG: Trigger Upgrade Status

  37. CXP Modules on CTP7 • Each CXP module provides 12x lanes of optical Tx and Rx, at speeds between 1Gbps and 10.5Gbps • CTP7 uses 3 CXP modules for front side optical connection • Front panel mounting simplifies PCB design • Avago CXP Modules use same optical engine as AvagoMiniPODs for complete compatibility with other CMS boards • Press-fit receptacles required design of special CTP7 holding fixture during the press-in process • Front-panel pluggable design means board extraction not required to replace module PMG: Trigger Upgrade Status

  38. Embedded processing • Xilinx provides two embedded solutions • Soft – Microblaze, on CTP6 • Hard – ZYNQ, on CTP7 • PetaLinux is an embedded Linux product from Xilinx • Runs on Microblaze and ZYNQ platforms • Standard, debugged drivers for SPI, I2C, Ethernet, USART, Flash, etc. • Communications and control functions are most easily handled by processors and software • Advantages • General communications & control functionsoriginally requiring HDL design now can be done by SW engineers writing Linux Apps • More SW engineers than FW engineers • Deployed on CTP6 & oRSC using Microblaze • On the CTP7 using ZYNC processor PMG: Trigger Upgrade Status

  39. CTP7 ZynqSoC Linux Control Chip • ZYNQhas hard ARM core • embedded Linux on transforms housekeeping tasks into Linux application • Xilinx AXI chip-to-chip exposes Target FPGA into ZYNQ memory space • Can store FPGA bitfiles on μSD FAT filesystem for easy upgrades via SCP/pop in a new μSD Target AMC Board Low Speed Peripherals (EEPROM, Clock Synths, Optical Modules, Crosspoint Switches, Console, etc.) ZYNQ SoC (Linux) SDRAM AXI Chip-to-Chip Interface SPI Config Interface GbE Connection microSD Flash Memory Target FPGA PMG: Trigger Upgrade Status

  40. CTP Software • CTP6 has two FPGAs • Front-end connected to optical links • Back-end connect to Ethernet • Inter FPGA Communication via serial bus: UART • CTP7 follows the same model, replace BE by ZYNQ chip • Development going on now should be portable PMG: Trigger Upgrade Status

  41. CTP6 Software • All of the FPGAs have a Microblaze core • On the CTP6 BE, PetaLinux is installed • So we use IPBus protocol, in our “softipbus” implementation available in the CACTUS SVN repository • softipbus includes: • A TCP server that runs on Linux and interprets the byte stream as IPBus • A TCP server that runs on Linux and forwards the byte stream along a file descriptor • Functions to de-serialize a byte stream into IPBus transactions • Functions to handle IPBus transactions (READ/WRITE/RWM) • Functions to serialize the responses into a bytes stream • Code is written so it can run on Linux as a server or be used as library on the standalone devices • All the standalone programs are in github • Problem Tracking and Wiki too PMG: Trigger Upgrade Status

  42. MMC Project: IPMI System Manager • μTCA features Gigabit Ethernet to address individual slots and IPMI to manage the power and sensors of individual cards • Commercial IPMI management tools available, but • Vendor specific • Not extensible (GUIs) IPMI: Intelligent Platform Management Interface MMC: Memory Management Controller PMG: Trigger Upgrade Status

  43. IPMI System Manger • API wrapping a socket interface • Act as a gateway for other applications into the IPMI system • Provide a robust IPMI-over-LAN connection to crates • Provide a service for automatic low-level initialization of AMC cards at startup Crate MCH Crate MCH Crate MCH PMG: Trigger Upgrade Status

  44. Calorimeter Trigger Plans • Complete oRSC testing and manufacture cards • Test prototype CTP7 in Madison Lab and then bring to CERN for integration tests with MP7 card • Work on implementation and optimzation of Stage-1 algorithms • Design and Performance documented in Level-1 TDR • Collaboration with FNAL and UIC • Continue Firmware development for oRSC and CTP7 • Integrate Software for oRSC and CTP7 with online Trigger Supervisor system PMG: Trigger Upgrade Status

More Related