260 likes | 267 Views
DCS Instructions. K. Grogg, M. Weinberg, M. Grothe. DCS Monitoring . Check that monitoring is working Data is being recorded Bugs in the code Check that the system is in a good state Top level state machine Histograms for each RMC Enter information in elog What was checked and results
E N D
DCS Instructions K. Grogg, M. Weinberg, M. Grothe
DCS Monitoring • Check that monitoring is working • Data is being recorded • Bugs in the code • Check that the system is in a good state • Top level state machine • Histograms for each RMC • Enter information in elog • What was checked and results • Details on any errors, alarms, warnings
Starting • The following slides have step by step instructions for what to look at and what to expect, along with screen shots • Be sure to read Twiki carefully, it has more information than these slides • Start by • Tunnelling • ssh -Y lxplus.cern.ch -L 60001:pcwiscms05.cern.ch:3389 • Set up putty and pvss (See Twiki for full set up instructions!) • This only needs to be done once if you save you settings
Starting PVSS Click here to start
Finite state machine • Are the FSM states of unmasked RMCs either standby or ok? • Any known errors should be masked • RMC10 might be off or in error, usually ok • If need to mask, take control (click lock icon), and change check mark to X • Does the alarm/alert overview panel report any warnings and faults? • Does the alarm/alert history show any new alarm/alert entries? • Need to take control (lock icon) to see • Record any new or fixed alarms • Be sure you have released control when done!
Finite State Machine Take Control Alarm Overview RMC 10 is masked
After taking control Alarm History Release Control Mask/Unmask Do not leave it like this. Be sure to release control!
RMC GUI • Double click on each RMC to get panel • Click “Send Unprivileged Command” • Is information about the crate sent every minute? • If not, note that there is a monitoring problem, contact an expert • Look at histograms – (time histories) (right click on top of histogram to see actual values) • Is the system healthy? • Are the temperatures, voltages, currents, etc stable? • Look for spikes or slowing rising/falling values • Is the Alarm Status 0, and Online status 1? • Make sure numbers in histograms match those in the “Detailed System Status” • Checks that the values are recorded in the database 1/4/2020 8
RMC GUI • Double click on each RMC to get panel • Click “Send Unprivileged Command” • Is information about the crate sent every minute? • If not, note that there is a monitoring problem, contact an expert • Look at histograms – (time histories) (right click on top of histogram to see actual values) • Is the system healthy? • Are the temperatures, voltages, currents, etc stable? • Look for spikes or slowing rising/falling values • Is the Alarm Status 0, and Online status 1? • Make sure numbers in histograms match those in the “Detailed System Status” • Checks that the values are recorded in the database
Getting to RMC GUI Double click to open RMC GUI
RMC GUI • Double click on each RMC to get panel • Click “Send Unprivileged Command” • Is information about the crate sent every minute? • If not, note that there is a monitoring problem, contact an expert • Look at histograms – (time histories) (right click on top of histogram to see actual values) • Is the system healthy? • Are the temperatures, voltages, currents, etc stable? • Look for spikes or slowing rising/falling values • Is the Alarm Status 0, and Online status 1? • Make sure numbers in histograms match those in the “Detailed System Status” • Checks that the values are recorded in the database 1/4/2020 11
RMC GUI Panel Check the circled buttons
RMC GUI • Double click on each RMC to get panel • Click “Send Unprivileged Command” • Is information about the crate sent every minute? • If not, note that there is a monitoring problem, contact an expert • Look at histograms – (time histories) (right click on top of histogram to see actual values) • Is the system healthy? • Are the temperatures, voltages, currents, etc stable? • Look for spikes or slowing rising/falling values • Is the Alarm Status 0, and Online status 1? • Make sure numbers in histograms match those in the “Detailed System Status” • Checks that the values are recorded in the database 1/4/2020 13
Histograms Check all histograms for any changes/problems
RMC GUI • Double click on each RMC to get panel • Click “Send Unprivileged Command” • Is information about the crate sent every minute? • If not, note that there is a monitoring problem, contact an expert • Look at histograms – (time histories) (right click on top of histogram to see actual values) • Is the system healthy? • Are the temperatures, voltages, currents, etc stable? • Look for spikes or slowing rising/falling values • Is the Alarm Status 0, and Online status 1? • Make sure numbers in histograms match those in the “Detailed System Status” • Checks that the values are recorded in the database 1/4/2020 15
Temperatures Right click here to get this toggle option and select
Temperatures Temperature should be roughly around these values: Temp A ~ 27 ± 3 Temp B ~ 25 ± 3 Temp C ~ 30 ± 3 Temp D ~ 32 ± 3 Axes have been adjusted (dragged) to show all crate temps
Voltage and Analog Temp For most RMCs +5V is usually 5.06 +12V is usually 12.2 -12V is usually -12.3 For RMC 3 +5V is usually 5.7 +12V is usually 14.3 -12V is usually -8.7 Axes have been adjusted to display all voltages Analog Temp should be in the low 20s
Supply Current Current should be between 46-49 A, except for RMC 5 which is usually 50-51 A
Alarm Status Should be zero (0), otherwise there is an alarm!
Online Status Should be one (1) unless there is a known reason otherwise
RMC GUI • Double click on each RMC to get panel • Click “Send Unprivileged Command” • Is information about the crate sent every minute? • If not, note that there is a monitoring problem, contact an expert • Look at histograms – (time histories) (right click on top of histogram to see actual values) • Is the system healthy? • Are the temperatures, voltages, currents, etc stable? • Look for spikes or slowing rising/falling values • Is the Alarm Status 0, and Online status 1? • Make sure numbers in histograms match those in the “Detailed System Status” • Checks that the values are recorded in the database 1/4/2020 22
RMC GUI Panel Check the circled buttons The values here should match those in the histograms 1/4/2020 23
Closing an RMC GUI Use this button to close, the X doesn’t work
What to expect • Temperatures should be fairly stable • Temp A ~ 27 ± 3 • Temp B ~ 25 ± 3 • Temp C ~ 30 ± 3 • Temp D ~ 32 ± 3 • Look for spikes or slowly rising values • Voltages should be very stable • For most RMCs: • +5V is usually 5.06 • +12V is usually 12.2 • -12V is usually -12.3 • Analog Temp is in low 20s • Supply current should be around 50 • Between 46-49 A, except for RMC 5 which is usually 50-51 A • Alarm Status should be zero • Note any changes and when they occurred • Online/Offline should be one • Note any changes and when they occurred • For RMC 3: • +5V is usually 5.7 • +12V is usually 14.3 • -12V is usually -8.7
In case of problems • Check “Detailed Alarm Status” and “Detailed System Status” • Know where alarms are coming from • Diagnose the problem • Call an expert • In case of Alarm, check “Most Recent Fault Record” to see error info • Check time stamp • If rack power is switched off, some information may not be updated • Do not Refresh All! • Do not click POWER ON, POWER OFF, CLEAR ALARM, or Further Expert Action! • Be sure to enter everything you find in the elog