Shutdown interventions and operation in 2011

1. Shutdown interventions and operation in 2011 Gianni Masetti & Ignacio Redondo (and others) Muon Barrel Workshop

2. Outline • Operations in 2011 • New oncall support structure • Comment on Interfill Cosmics • DocDB interlude • Attemp to summarize shutdown interventions • Everything touched,…but MCs & chambers • Several crucial soft points reinforced. • 2011 DT is a stronger system • Thanks to everyone involved Muon Barrel Workshop

3. Operations in 2011 • Special runs in first fills under discussion (only RPC in trigger for instance) • New oncall support scheme: • DT DOC (Detector On Call) is our first defense line: • 1 responsible per week • Reports in DT weekly meeting/daily run meeting • Earns central credit –points (21) • and 0.25 ManMonth of ESP for the training week? • More exposure (4 weeks min) and longer shift periods (1-2 week) • First discussion session held yesterday • Practical trainings to be organized • Population: 10/15 old DT oncall; 2/15 old DTFM; 3/15 newcomers • Infrastructure oncall support down to minimum (2) • DT expert cloud Muon Barrel Workshop

4. Le XV de DT (DOCs 2011)  Zoltan Szillasi Noemi Beni  Mario Pelliccioni Marco Dallavalle Daniele Fasanella Jesus Puerta Pelayo Daniel Dominguez Vazquez Juan Pablo Fernandez Alicia Calderon Shan-Huei Chuang Isidro González Caballero Massimo Nespolo Paolo Bellan Antonio Branca Daniel Teyser Muon Barrel Workshop

5. DT interfill Operations • No Heavy Stable Particle data this year • Remaining HV ON ~1 h after the fill dump was and is not a problem • Probably cosmics (What flavour of cosmics?) (How many?) • Expectacion is interfill time will narrow down • System operation will need (part of) this time for mantainance • MC/Mezzanine reconfiguration • System changes tests • Routine Test pulse runs • … Muon Barrel Workshop

6. CMS DocDB: (Yetanother) CMS database https://cms-docdb.cern.ch/cgi-bin/DocDB/DocumentDatabase • DB documental dropbox: designedto host anykind of documentsexcept: • Talksalreadyincluded in Indico • Engineeringdocs (EDMS) • Officialpapers (CDS) • Anyotherdocument can (must?) be uploadedtoDocDB • Open toany CMS member (NICE login) • Stillpreliminaryversion (someissueswithpermissions) Jesus Puerta Muon Barrel Workshop

7. A single «document» can contain several files Any entry is updateable a posteriori (versioning) Uploaded document data to be included: Title Description Privileges Topics/Tags (to be chosen from a predefined list) Several tags for same document can be used. Documents can be browsed by tags/topics! Any document or list of docs worth being kept, feel free to give DocDB a try. Let Jesús know about any other tag structure Preliminary list of DT-related tags (Cristina/Alberto/Jesús) Jesus Puerta Muon Barrel Workshop

8. Readout/Control A. Triossi et al. • New VME PC crate control board: Linco • SC power cycle required vmepc reboot • Sometimes Linco Driver reload results in blocked PC • More frequent after general PC reinstallation by system administrators • Exchange two damange fibres helps • Need long term monitoring • SC power cycle does not require to reboot PCs! • Still have two remember to hit 2 buttons in two programs • Thinking how to integrate this • 10 5 DDU configuration • Minor FW upgrade • Spares! • As a subproduct, get faster configuration (30 %) V. Monaco et al. Muon Barrel Workshop

9. ROS FW upgrade VME FPGA cannot be remotely programmed: delicate intervention Main changes: GOL does not power cycle on each configuration (to minimize stress), it will only power cycle at power boot (or at demand with expert stand alone program) Maximum Event Size register can be reseted (manually and with a TTC command): TTC BGo Channel 12 (command 0x30) performs Event Size reset, so we can choose the frequency we want to monitor max event size (once per orbit, etc) TTC Hard reset implemented (BGo Channel 6, command 0x18): • Reload the firmware of CEROS and ROSCTRL FPGAs (radiation SEE) • Drawbacks: • HU &TO resynch indep registers will give error after reload • ROS configuration will be erased (if something comes as mandatory it will have to be hardcoded in the firmware) DAQ software has been changed to make it faster (~50 % ) Cristina Fernandez Bedoya Muon Barrel Workshop

10. New MC Secondary link • Granularity increased from half wheel to one sector (maximum achievable) • Each sector chain completely independent (in the old board only 4 rs485 ports are available) • FW remotely upgradable • the old board was upgraded adding  some protection circuitry, this circuitry will be integrated in the new one • Fibre connectors inside the crate, so more protected • Added an 1-wire bus for linco monitoring • More robust link to USC: rs422 via linco (fast), 1 fiber compatible mode (same as now), 1 fiber fast mode • Power supply independent from trg/ro crate Franco Gonella Muon Barrel Workshop

11. the prototype seems • to work properly • 10 boards (+ 5 spares) will be ready at the end of the year SC crate USC UXC old new Secondary link board prototype installed on YB+1 TRG/RO upper crate (the old board was left in site for quickly restoring in case of problems) DCS PCs rack DCS fibers rack DCS in0 YB+1 I connector Ethernet connection USB Primary link fiber 5 sec2 (compatible mode) Primary link fiber 5 sec3 (new fast mode) PIC32 Ethernet Starter Kit With optical interface Franco Gonella Muon Barrel Workshop

12. Trigger • See Session with specific talks • Janos installed new VME Control FW in DTTF crates • Common effort: DT/DTTF, Bolonga/Padova (linco was very helpful), others,… • Enables optoRX monitoring • Enables automatic link synchronization • New crate tested (and removed) in sectors #1 # 2 • New Barrel Sorter FW (under test) • New Wedge sorter FW (under test) • DT technical trigger (to be commisioned) Muon Barrel Workshop

13. Infrastructure • Power Supply system • HV • LV • Gas Muon Barrel Workshop

14. Chambers with Problems Since January 24 HV ON • All problems but for YB+2 MB2 S03 SL2 L2 occurred at HV ON • Six chambers had problems at HV ON in 2010 and were quiet afterwards. Most probably due to A877 vagaries • In addition the A876 for YB0 S09 failed to turn ON and was replaced • No HV problems occurred during HV ON from STANDBY after ramp to 3.8 T Alberto Benvenuti Muon Barrel Workshop

15. Maintenance Report (NOT a complete list) • Mapping of the smoke detectors DSS alarms was checked and several wrong paths were corrected. It was not easy but now the map is correct for all tower racks. (D. Fasanella) • Several temperature sensors were replaced by David and Cristina, all sensors are now operational • The grounding of the cable braids was checked and standardized at all tower racks (Franco,Barcellan): • Braids connected to ground for FE LV cables • All MC LV cables are floating Alberto Benvenuti Muon Barrel Workshop

16. One major incident:Oct 5 post-mortem Reminder • During fill T sensor in turbine rack (wheel/4) fails, triggering DSS interlock. • Unfortunately, safety system was wrongly cabled, switching off wrong rack, bringing down wheel -2 • Problem understood (and short-term fixed) before the end of the fill. Interlock expert had to drive in (1/2h). • Final fix done and tested during last technical stop. • ~1 h to change conf to get CMS running (without wheel -2). • Good part of it because we needed trigger to change configuration accordingly • Prompt DT response: The “right” experts where involved. • Some time lost because power cycling crates involve rebooting control PCs to load control drivers, sometimes PCs block at load. • Upgrade control boards this Christmas shutdown so no reboot is required. • Would all DT experts have been equally fast ? • Documentation written • Need to spread the ability to change configuration among the DT FM pool Zurich Run Coordination Workshop Muon Barrel Workshop

17. LV VCON errors • Several cable connectors were checked/replaced that showed anomalous temperatures at the A3009 modules • No clear defects (L. Barcellan) were visible • Several LV modules have been exchanged thanks to David and Daniele: • MAO for YB0 S10, A3100 for YB0 Top Sector Collector • A3050 for YB0 MB2 S10 • NOT obvious what to do with these modules since there is no specific fault apart for repeated VCON VCC/VDD errors. • All connectors have been lubricated but those in YB0+SCs (already done in summer 2010) Alberto Benvenuti Muon Barrel Workshop

18. New LV Connector failures The temperature of one A3100 connector powering YB+2 SC 1-7 jumped to 32° and then dropped to 27° Checking the temperature after power cycle it dropped a bit. There was another sudden jump above 30° on 18-feb and then it dropped to a normal value on 17-feb 30° 25° Drops when “touching” PS Positive polarity connector (failure not inside the crate) This episode occurred all by itself. The Voltage is still higher than it should be albeit below the alarm threshold YB0 MB4L S10 Vcon from February 1 to 20 Alberto Benvenuti Muon Barrel Workshop

19. New LV Connector failures YB-1 MB3 S12 Vcon Error First alarm 21-feb at 22:52 MC switched OFF at 0:42. Problem persists after MC power up. UX access needed to fix it. Problems are no new in the sense that the pattern is the same as last year (positive polarity is warmer through an increase of resisitivity) It is a bit unsettling that there are still these problems after changing cables, reinforcing contacts, adding lubricant…. Maybe we are seeing effects due to the magnetic field, we could try to block the cables as was done before the connector exchange At any rate, I suspect we have to live with this, since changing the connectors on the A3050 looks like mission impossible Muon Barrel Workshop Alberto Benvenuti

20. Gas Analysis Alberto Benvenuti • A rather painful restart: • Software updates blocked the data storage/acquisition • The refresh rate of the measurements was changed and the time stamp is not useful anymore, still to be fixed • O2 Analysis cells damaged by leak (will be replaced ASAP) • H2O analysis cells to be replaced with a new model (proto in YB0) • Current O2 measurements are 20 to 50 ppm lower than in 2010 probably due to analysis cells • The system is still leaking 1000 liters/hour, ongoing since July • Issues were discussed with gas group (Haider, Guida…) suggestions for a more efficient operation should be implemented ASAP • Online gas chromatograph should be operational by end of march. Muon Barrel Workshop

21. DT shutdown highlights • Readout/Control • New VME PC crate control board: Linco No need to reboot PCs after crate Power cycle • 10->5 DDU configuration Spares! • As a subproduct get faster configuration (30 %) • New ROS FW (delicate intervention) FW reloaded at TTCHardReset • As a subproduct get faster configuration (50 %) • Prototype Secondary link to MC installed • Trigger (see session with specific talks) • Janos installed new VME Control FW in DTTF crates • Common effort: DT/DTTF, Bologna/Padova (linco was very helpful), others Enables optoRX monitoring Enables automatic link synchronization • Infrastructure: • Extensive DSS tests Several mapping errors corrected • HV • LV All connectors lubricated • Gas VDC system installed (following talk) Muon Barrel Workshop

22. The End Muon Barrel Workshop

23. Prototype of the new Secondary Board installed on YB+1 Present mapping Foreseen mapping node1 -the prototype seems to work properly 485 port0 sec1 PIC sec1 Can bus sec2 sec3 node2 sec2 485 port1 PIC -10 boards (+ 5 spares) will be ready at the end of the year node3 sec4 sec5 sec3 485 port2 PIC node4 node9 485 port3 PIC sec6 sec4 node8 node5 485 port4 PIC sec5 Protection circuitry DS1820 temp sensor node0 node6 485 port5 PIC sec6 RS422 link 230400 bps node7 485 port6 Optical link 38400 bps PIC TSB backup link PIC PIC PIC Optical link 230400 bps Fiber2 (new fast mode) RS422 1-Wire With this mapping the board is fully compatible with the old one Fiber1 (compatible mode) Primary link 5 sec3 RS422 VME crate Primary link 5 sec2 Franco Gonella Muon Barrel Workshop