Lessons Learned - SNS Linac PPS Phased Upgrade to Safety PLC

1. Lessons Learned - SNS Linac PPS Phased Upgrade to Safety PLC John Bombard Protection Systems Engineer Spallation Neutron Source

2. Overview • Summary of SNS Linac Safety PLC Upgrade • Design Lessons • Installation Lessons • Operational Lessons • Questions or Comments?

3. SNS Linac Safety PLC Upgrade • The SNS Linac PPS mitigates radiation hazards from beam or other radiation sources in the Front-End “Ion Source” as well as the linear accelerator tunnel sections • 5 Monitored Entry/Exit Doors • 15 Emergency Stop / Sweep Stations with Status Lights • 43 Controlled Devices: 3 Ion Source, 15 RF Modulators, 25 RF Amplifiers • Upgrade effort converts radiation safety controls to SIL rated safety class equipment • Get ahead of obsolescence issues using non-safety controllers in a safety application • First phase is the conversion of the Front End Ion Source

4. SNS Linac Safety PLC Upgrade • Historically used Allen-Bradley ControlLogix programmable logic controllers • Two independent systems, we call “PLC-A” and “PLC-B” • Software code maintained by two engineer/programmers • Follows A-B recommendations for using ControlLogix in a SIL 2 configuration

5. SNS Linac Safety PLC Upgrade • Upgrade replaces ControlLogix with its safety-rated equivalent GuardLogix • Two GuardLogix systems, maintaining PLC-A/PLC-B overall architecture • Safety field devices in a “Category 2” (“SIL 1” or “PL b/c”) configuration • One Input Device and One Output Device per PLC, with diagnostic monitoring • The duplex SIL 1 systems can meet SIL 2 reliability requirements

6. SNS Linac Safety PLC Upgrade • Upgrade is a “phased install” • Processor modules and one of five large control racks was updated with GuardLogix safety hardware • Racks were a complete remove and replace project with all-new hardware • Remaining 4 racks still using all ControlLogix hardware to control safety devices and will be upgraded in a future effort • Upgrade required new network infrastructure • The GuardLogix safety hardware only works over Ethernet, but can still control non-safety ControlLogix hardware over other busses • Lesson: Complete remove/replace allows for extended off-line testing and verification, allowing more efficient use of accelerator downtime to install with most of the intrusive testing already done.

7. Design Lessons - Hybrid Network Types • New GuardLogix safety hardware requires Ethernet communications • Existing ControlLogix hardware installed using ControlNet communications • Original design had Ethernet between new GuardLogix processor and the new field rack, but kept ControlNet between upgraded field rack and the remaining 4 non-upgraded racks Source: Rockwell Automation Knowledge Base

8. Design Lessons - Hybrid Network Types • Later identified this topology would not work at all in practice due to incompatibilities • Caught early during design process • Decision made to convert all racks, including those not being upgraded, to Ethernet • Added an additional ~$70k in unexpected equipment cost • Lesson: Validate new network architecture with manufacturer support engineering or with internal testing

9. Design Lessons – Integrating Pulse Testing • Safety field devices in a “Category 2” configuration for GuardLogix requires the use of “Pulse Testing” • The voltage supply feeding a switch or actuator is replaced with a pulse train to constantly verify the input’s ability to sense transitions • In an ideal configuration, each device should have a different pulse

10. Design Lessons – Integrating Pulse Testing • Most existing field devices at SNS used intermediate junction boxes to fan out DC power for actuators • Example: 4 conductor cable was pulled for 3 signals, however proper pulse test implementation requires 6 conductors • Replacing these cables and boxes was outside the scope of this phase Lesson: Proper implementation of diagnostic pulse testing will require design changes to field devices, and likely require new cabling

11. Design Lessons – Software Coding Methods • GuardLogix at a basic level uses two user programs, a “Standard Task” and a “Safety Task” • The “Safety Task” is logic run on the redundant high reliability hardware • This task can only use designated safety memory and safety I/O directly. • Standard I/O may be used with a ”mapping” procedure • The “Standard Task” is logic run like a standard ControlLogix • At SNS, the PPS logic was already a standard ControlLogix program

12. Design Lessons – Software Coding Methods • Two different programmers are used for the “PLC-A” and “PLC-B Systems • The PLC-A programmer chose to re-use the existing ControlLogix program running entirely in the Standard Task, only using the Safety Task to perform the mapping to the Safety I/O that is only accessible from the Safety Task • The PLC-B programmer chose to totally re-code the logic from the ground up, correctly isolating Standard and Safety functions to their respective tasks • However, similar mapping had to occur in reverse, such that the 4 un-upgraded control racks still controlling safety devices with standard I/O could have safety logic control standard I/O

13. Design Lessons – Software Coding Methods • Lesson: Either way, a hybrid system will never fully meet the intended use of safety/standard tasks; so neither implementation is correct or incorrect. • Which way it should be done depends on how much time you have during a particular upgrade • PLC-A worked exactly as it previously did with little debugging; but will be harder to make compliant with the manufacturer's “correct” implementation of the Safety PLC when all the racks have been upgraded • PLC-B required a lot of debugging and tweaking in the field during this upgrade since the code was all-new, but will be easier to maintain and integrate with additional future rack upgrades • Additionally, rewriting one program at a time minimized risk of introducing systemic errors to both programs.

14. Installation Issues – Operability of Other Systems • One of the refurbished cabinets contained a component that was part of a separate Oxygen Deficiency Hazard monitoring/alarm system for cryogenic equipment in the Linac tunnel • This system is a required Credited Control when the Linac tunnel is occupied by workers • The refurbished cabinets each had their own electrical feed, so all power would need to be under lockout/tagout to safely work in these racks • Upgrade work was done during a maintenance outage, locking workers out of the Linac tunnel for several weeks was not possible if the ODH system was down

15. Installation Issues – Operability of Other Systems • A small control box was constructed to house this equipment, and was swapped over to during a shorter outage • This temporary box needed safety-class redundant backup power such that it was equivalent to what was removed, making it rather expensive and complex for something that was only needed for a few weeks • Lesson:Systems that operate independently should have their own independent enclosures and power systems, such that one may be removed from service without affecting the other

16. Installation Issues – Field Cable Lengths • When the upgraded racks were designed they were laid out to separate standard and safety wiring/power, in a different fashion than the existing racks • The I/O was moved one cabinet closer to the field, intended to mitigate any cable length problems pulling the existing cables to the new I/O • This wasn’t enough, and many of the “PLC-A” cables were too short to reach their intended termination point • Additional rows of intermediate terminal blocks had to be added to extend these connections to their proper locations • Lesson: During construction, leave adequate service loops for 1-2 major rack changes during the life of a facility. When doing an upgrade, take the existing layout and cable length into account when designing a new rack.

17. Operational Issues – Default Configurations • Since the GuardLogix hardware requires creating a new software development project for the logic, there are many underlying settings that will be reset to defaults • One of these values is a “watchdog timer” that detects when the program gets stuck in a loop or otherwise hangs up or crashes • If the program doesn’t finish its scan in the specified time, the controller faults and shuts everything down • The default is set to a conservative 20ms, but it needs to be tweaked depending on how much logic needs to execute, how long it takes to communicate with the I/O, etc • Shortly after commissioning, one of the systems faulted due to a watchdog over-run, since this value was not changed from the default to something better suited for the communications speed and amount of logic being evaluated • Lesson: Ensure that you are familiar with all of the underlying configuration settings that need to be updated before releasing a system to operations.

18. Final Questions or Comments ?