1 / 35

TPC shift guide

TPC shift guide. Run period 10.09-31.12.2008 Updated on 04.09.2008. Foreword. This shift guide is intended for the TPC shifter to be able to operate, run and monitor the TPC: Identify the relevant machines: aldaqacr34, the local TPC linux machine in the Working Room (TPC.501)

gunnar
Download Presentation

TPC shift guide

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. TPC shift guide Run period 10.09-31.12.2008 Updated on 04.09.2008

  2. Foreword This shift guide is intended for the TPC shifter to be able to operate, run and monitor the TPC: • Identify the relevant machines: • aldaqacr34, the local TPC linux machine in the Working Room (TPC.501) • alitpcon001, the Windows PVSS Operator Node • alitpcwn008, the Busy Box linux machine • Power on/off and configure the FEE for any specific run • Configure and initialise the Busy Box • Use the Local Trigger Unit • Start/stop various standalone runs, or get ready for a global run • Periodic pedestal and cal. pulser runs (DAs); laser runs • Use the online monitor to look at event displays, and use AMORE • Monitor the TPC: LV, FEC temperatures, cooling, ROC HV, Drift HV, gas, run status • Use the electronics logbook • Troubleshooting, list of expert

  3. Connecting to the Operator Node • Make sure there is a remote desktop session to alitpcon001; if there is not one, a registered user does: • rdesktop –g1280x1024 –a16 –u username alitpcon001 • Enter your NICE username and password • Start  TPC.bat (or just TPC) • Click on the key icon (top left) to login as tpc tpc • You now have the PVSS panel • On the left there is the tree with all the panels • Bottom left, a synoptic with the colour-coded status of all systems • The top level node, TPC_DCS, will be the most extensively used • Note: if you can't type in a field, go to a linux terminal session, type something there, and go back to PVSS

  4. Functionality of top node • TPC_DCS: Top node of the TPC FSM Tree Take the lock of the FSM Tree from here. If you need to CLOSE the main panel, first release the lock with the option RELEASE ALL FSM Commands: State: OFF Action: GO_STANDBY -> Switches ON LV PS State: XXX Action: CONFIGURE  -> Configuration of the FEC according to the sent parameter (usually RESET and then COSMICS or pp)State: XXX Action: SOR or EOR -> Actual condition values are send to database for Shuttle • If the FSM has to be restarted (it also releases the lock) press 'Start/Restart All'. Wait until the 12 boxes (tpc_ui etc.) on the lower part of the panels are green. • Forced restart: Press 'Stop All' wait ~20 seconds, press 'KILL FSM' wait ~5 seconds, press 'Start/Restart All'

  5. Cooling • TPC_COOL (in Operator Node) (next page) • FEC Cooling plant must be RUNNING • Resistor Rod Cooling plant must be RUNNING • In case of problems ('Standby' or 'Stopped') call expert Experts: Cooling Piquet: CCC 72201 (ask for CV/Detector Cooling, answer ALICE TPC) Rudi Schmidt 16 3962 Chilo Garabatos 16 5327 Danilo Vranic 16 3691 • Note: if the cooling goes to standby, an interlock will turn of the FECs after 2 min, and the LV power supplies after 7 min

  6. Power up and configure FEE • Switch ON LV Power Supplies from PVSS • This operation should be done by an expert • Go to the FED MONITOR, select FSM state and wait for required partitions to be blue • In the (sub)system state tab, select CONFIGURE and enter, e.g., COSMICS as parameter. This will turn on the FECs and configure them for running with cosmic triggers. • Other commands are • GO_IDLE (FEC power off, do not do this for fun) • CONFIGURE with entry RESET (a popular command) • CONFIGURE with entry PEDESTAL_RUN (for pedestal run) • Before each run it is recommended to CONFIGURE with entry RESET and once more with, e.g., entry COSMICS (and reinitialise the BB, see later) • Experts: Christian Lippmann 162430, Luciano Musa 163119, Chilo Garabatos 165327

  7. FEE Monitoring • In TPC_FERO you can select different monitors: • FED Monitor: shows by default the temperature of the FECs, which should be lower than ~30 oC. • In the lower third of the panel, all boxes should be green. If one is red, try Restart. If still red, call expert (CL). • The tab FSM State shows you the state of all readout partitions • LV Monitor: shows by default the analog currents, which should be 2 Amp (FECs off) to 130 Amp (all FECs on). • Cooling Monitor: shows the status of the cooling: it should be OK and RUNNING (all green). • In case of problems call Christian Lippmann 16 2430, Chilo Garabatos 165327, or Luciano Musa 16 3119

  8. FEE Configuration (after power up) • On the operator node (alitpcon001) right-click on TPC_FERO in the FSM tree to the left. • Choose view panel. • Click on the FSM button (just above). • Take the lock by clicking on the lock symbol. • In the System TPC_FERO (on the top) click on the state and choose CONFIGURE and type RESET as run parameter. The states should all be (or change to) STANDBY. • Choose again CONFIGURE ,and type as run parameter the right name from the table of next slide. • Wait for all states to change to ON. This may take ~10 seconds. • In case of ERRORs choose RECOVER and repeat steps 6 and 7.

  9. Configurations

  10. Configuration via DCS of single RCUs • On the operator node (alitpcon001) open the FSM tree and find the desired partition (RCU): TPC_RCU_xxx_p. • Check monitoring values and change states here. • You get the same functionality by clicking on the individual partitions (RCUs) on the FED Monitor panel.

  11. Troubleshooting: FEE Configuration / Monitoring via DCS (1) The FSM tree is locked: • Find out who has the lock, and ask him/her to release it. • From TPC_DCS top node, Restart ALL, Kill FSM, Start/Restart all No response from FEE: • If it is a single Partition (DCS board / RCU), try • Restart feeserver on this DCS board (ask expert), • Reboot this DCS board (ask expert) It takes 20 to 60s for the DCS board to reboot and for the data from this RCU to appear on the monitoring panel. • Restart PVSSdim by clicking on “Kill PVSSdim“ in the FED Monitor panel. • Restart interComLayer(s) (see next slide).

  12. Troubleshooting: FEE Configuration / Monitoring via DCS (2) Restart interComLayers: • Log in to alitpcwn008 (A side) or alitpcwn009 (C side) (user tpc, no password needed from aldaqacr34). • Check if interComLayer is running (type „ps –a“ in a shell). • Restart it by executing „killall interComLayer“. Additional information: • The interComLayer is an intermediate Layer between the FeeServer on the DCS boards and the PVSS and FSM Control Layer you are using. There is one per side. • It runs in the directory /home/tpc/icl-rundir. • In case it exits (or is killed), the interComLayer is automatically restarted by the script /home/tpc/icl-rundir/startInterComLayer.sh.

  13. Busy Box • The busy box has to be reconfigured before a run starts. • It is now done automatically from the central ECS • It can be done from our FSM by sending a SOR command with the option RECONFIGURE • Click on BUSY BOX button of the TPC_FERO node to see the status of each partition • The selected partitions should show green in the setup tab • clicking on a partition toggles inclusion/exclusion • clicking the label of a sector toggles the whole sector • Click APPLY to load a new configuration • You can reconfigure manually –not during a run!- the BB by clicking RECONFIGURE in the STATUS tab; selected partitions should go green

  14. FEE Configuration via PERL scripts • Go to aldaqacr34 and type the command: ./SECTORS_do.pl has the following options: reset/config/on/off/armSYNC choose the appropriate one, the side, and the sector number • Examples of commands: ./SECTORS_do.pl A config 0-17 (to configure all FEE on A side) ./SECTORS_do.pl C reset 0-4 5 6 7 8-17 • The normal sequence is on, reset, config • The option armSYNC is used in stand alone runs only, in case one has just reconfigured a part of the system should not be needed

  15. FEE Configuration via scripts in each LDC • If you log into one LDC, you should go to/local/home/tpc/TPCrdo and type: source TPCrdo_commands.sh • Commands: SECTOR_on SECTOR_off SECTOR_reset SECTOR_configuration SECTOR_armSYNC • If this failed for a partition, you can address it individually: rpx_on rpx_off rpx_reset rpx_config rpx_armSYNC x=1,2,3,4,5,6 should not be needed

  16. FEE Configuration via TPCC (1) 1) Find a desktop with a session in aldaqecs01 (/date/tpcc/Linux) 2) Type: ./tpcc_server (and maybe type after this: ./dumpOracle.tcl) 3) Find a desktop in /local/home/tpc/rundir 4) For each LDC type /date/tpcc/Linux/tpcc_client 5) Move to aldaqpc126 (should be in /local/home/tpc/TPCrdo) -should see configurations (1-15), different commands for FECs(100-103), check the status of server (200) and exit (300) Not yet fully operational

  17. If you don't see the above options, type: /date/tpcc/Linux/tpcc_control To see online the informations online go to ALICE DAQ window, press infoBrowser and another window should appear; in this new window change from Stream (should be TPC and you have to clear this field) to Facility (write TPCC) and then press Online FEE Configuration via TPCC (2) Not yet fully operational

  18. Trigger (Standalone) To run CTP Emulator (LTU, trigger window): • Go to alidcscom026 (tpc tpc145) and type vmecrate tpc • The VME crate window should appear • select Configuration -> TTCinit • (Busy box is used by default) • Open the CTP Emulator window and here • choose sequencel2a.slm • load it by pressing the button Load sequence • select Signal • for pulser trigger choose BC and write the rate, e.g., 1s • for laser runs choose pulser/edge • In case the CTP emulator gets stuck, kill the 'ltu client' process from alidcscom026, and use the command 'xkill' and mouse-click to close the dead windows. In case of doubt or problems, please ask the CTP shifter.

  19. DAQ (Standalone) In DAQ Run Control window (click Date HI button in task bar to open): Select LDCs for Readout make sure you have the lock press left arrows until you get to Configuration press Define go to GDC and choose e.g. aldaqpc046 go to Detector: choose TPC -> A/C side -> choose LDCs save configuration and press ok Select the HLT mode (Mode A: no HLT) Select the equipment (readout partitions) in the equipment button of the DAQ menu) In DCA window (for DAs for example): • Make sure you have the locks • Select the run type: pedestal, laser, cosmics, etc • Start a run

  20. Configuration of Busy Box • Log in to alitpcwn008 (user tpc, no password)- cd /home/tpc/busy- execute ./bbox-conf.pl- select the partitions- save and exit • From here log into busy box dcs board: ssh alidcsdcb1555 - cd /mnt/dcbrw/busy/busybox_files- execute ./bbinit.sh (before each run!)- to check status, execute ./regpoll.sh status If one of the two top registers is zero, execute ./program busybox_fpga1.bit busybox_fpga2.bit Now in DCS

  21. Start/Stop stand-alone run In DAQ Run Control • press right arrows to get from Configuration to Ready to Start • press 'Start process', wait for ready and check for errors • press 'Start' • press 'Start emulation' in LTU window • Note: Each sector (6 RCUs) is readout via optical fibre with a Local Data Concentrator, a DAQ machine. The data is then transferred to one or several Global Data Concentrators which may or may not perform Event Building and Data Recording • To stop the run, quit the emulator (LTU) first, then click stop in Run Control

  22. Run control from ECS (standalone) • Use the ECS panel TPC Detector Control Agent (DCA) to control the run. • get the lock from the „TPC DAQ – Run Control“ (release it there first) • Select the operation of the GDC • No event building • No data recording • mStream • get the LTU, DAQ_RC, DETECOR, (HLT) locks • Select the HLT mode (ask the HLT shifter) • Select the run type (pedestal, pulser, laser, cosmic, ...) • this will start a run • pedestal and pulser will stop automatically • Eventually stop the run Note: if your lock is red and nobody seems to have it, make sure there isn't a second, hidden window in your panel: this one might have the lock!

  23. HV Monitoring • Drift HV (SIM_TPC_FC next page) • check Voltage (100 kV) and Current (355 A) • ROC HV (SIM_TPC_ROC next page) • In FSM state, all sectors must be green • If red (trip) call expert • If electronics is turned off >1 hour • click on the corresponding chambers • Switch ON button to INTERMEDIATE • Check the currents: call expert if > 1 mA • Experts: • Danilo Vranic 16 3691 • Chilo Garabatos 16 5327

  24. Gas • TPC_GAS (Operator Node) • Status must be 'OK' and 'Connected' • Mixer, Exhaust, Pump and Distribution modules should be run or circulating; if stop, call expert. • You may navigate around and look at trends In case of problems, call expert: Chilo Garabatos 16 5327 Gas Piquet: 16 2516

  25. Gating Grid pulsers The Gating Grid Pulser voltages are operated by experts. • Experts: Danilo Vranic 16 3691, Rainer Renfordt 16 3969, Chilo Garabatos 16 5327

  26. Lasers One PVSS panel per side. To turn on and run do: GO_STANDBY LOAD RECIPY GO_ON (this will calibrate and launch the 'tracker' to get optimal intensity) SUTTER_OPEN There is a trigger box which allows one to select the trigger rate –implementation in progress • Experts: Borge Nielsen 16 4221, C. Garabatos 16 5327

  27. Calibration pulser There is one unit per side. In each of them click • Open COM port • INIT • All ON The corresponding automatic run (DA) is called Pulser_Calibration_run • Click on All OFF when you are done!

  28. Online monitoring • Log in asdaquser@aldaqpc081 and go to/local/home/daquser/OM • type root • in root type: .x Monitor.C • another window appears: • select as format (DATE file/stream) • press Selectand OK • select File/Stream -> write LDC or GDC name (e.g. @aldaqpc046:). • Monitoring has to be enabled on that machine • press Next Event button • If root crashed or hangs, open another window in aldaqpc081 and type: killall root root.exe • To take a screenshot, open a terminal on aldaqacr34 and run ksnapshot; paste it in the logbook

  29. How to run AMORE /1 1) Open two terminal windows on aldaqacr33 (as user tpc) 2) In both do: ssh aldaqdqm04, source .balice, cd /tmp 3) Check that you get the monitoring data by doing e.g.eventDump –i –b @aldaqpc046: 4) On the first machine start the amoreAgent:amoreAgent -aTPCQA -s @aldaqpc046: 5) On the second machine start the display:amore  -dTPC -mUIQA - select init - select start - when you receive data, make the window bigger and you should see the different variables

  30. How to run AMORE /2 The variables are:MaxCharge – Qmax of the clusterMeanCharge – Qtot (integral over the 5x5 cell)NlocalMaxima – total number of clusters in this padNPads – average width of cluster in pad directionNTimebins – average width of cluster in time direction (Noisy pads willtypically have a small average since the noise is not shaped, seeexample in guide)NoThreshold – fraction of digits above threshold (ZS reduction) This isthe only variable which is not a cluster variable but shows the averageoccupancy for each pad (between 0 and 1)TimePosition – mean Z (time) position of clusters (if you have manynoise clusters in a pad this will give a peak centered at the middle ofthe time bin interval because the noisy clusters are typically evenlyspread out in time - perhaps a bit counterintuitive). Some more info can be found here:http://www.hep.lu.se/staff/christiansen/tpc_online_monitor_guide.ppt

  31. Logbook • Use it. • For each run add an entry with the run conditions: • FEC configuration • Missing partitions • Gain • Magnet status • Define run quality for TPC (good, bad)

  32. List of on-call experts • DCS, FEE, Busy Box, DAQ, Trigger • U. Frankenfeld 164048 (DCS), C. Lippmann 162430, L. Musa 163119 • FC HV, ROC HV, Cooling • D. Vranic 163691, C. Garabatos 165327 • Gas • C. Garabatos 165327, Gas piquet 162516 • Laser • B. Nielsen 164221, C. Garabatos 165327 • GG Pulser, Calibration Pulser • R. Renfordt 163969, D. Vranic 163691 • Online, Offline • M. Ivanov 77761, P. Christiansen 78956

More Related