Software QA

1. Software QA For Active CECs at SNS

2. Motivation “As Scott Jerome-Parks lay dying, he clung to this wish: that his fatal radiation overdose — which left him deaf, struggling to see, unable to swallow, burned, with his teeth falling out, with ulcers in his mouth and throat, nauseated, in severe pain and finally unable to breathe — be studied and talked about publicly so that others might not have to live his nightmare.” Radiation Offers New Cures, and Ways to Do Harm NYTIMES.COM, Jan. 23, 2010

3. Observations and Questions • Software QA is a BHAG • Fortunately our scope is limited

4. Observations and Questions • Software QA is a BHAG • Fortunately our scope is limited • Are we the “before” picture or the “after” picture?

5. Observations and Questions • Software QA is a BHAG • Fortunately our scope is limited • Are we the “before” picture or the “after” picture? • How much software QA is enough? • How many CEC software engineers does it take to screw in a light bulb? • Seven. One to write the specification program, two to screw it in, one to check if they screwed it in, one to validate that it was screwed in correctly and two to explain why the project was late.

6. Observations and Questions • Girls just want to have fun • It takes a village • If it ain’t broke don’t fix it • Change may lead to a learning experience at 3:00 am • KISS, maybe • Just one more feature….

7. Objective • Review current status of QA related activities for SNS CEC software • Establish a framework based on a consensus national standard(s) • Come up with a comprehensive roadmap for CEC software QA at SNS

8. Things to Keep in Mind • SNS has a large PLC based CEC for the accelerator • But most of the work now is centered around new instruments • These new systems are based on “safety” PLCs • Limited Variability Language • Aimed at users to create their safety application functionality. Typical languages used are Ladder Diagram and Function Block Diagram

9. The times they are a changin’ • SNS started out with the “standard” two redundant industrial PLC model with two programmers • Transitioning from two box, two programmer model to one box, two programmer model • One programmer writes the non-safety task and ½ the safety task while the other programmer writes the other ½ of the safety task • AB safety PLCs have certified code modules plus diagnostics built into hardware • Use of these tools it not exactly leaping out at us

10. Current status • Things we do per procedure • Things we do but not proceduralized • Things we intend to start doing

11. The following activities are required by the listed procedure:

12. The following features have been partially incorporated: • Expanded code comments • Reference to safety function • Pulse Test • Complimentary inputs

13. Standard, standard, whose got the standard • Target standard is ISO 13849-1:2006(E), Safety of machinery-Safety related parts of control systems- Part 1: General principles for design • Most applicable standard to current projects (Instrument PPS equipment) • Addresses software QA

14. Software lifecycle

15. Software related software specification: The SSRS document is unique to each PPS and provides information necessary to generate the PLC program. Included are safety functions to be accomplished, guidelines such as two programmer rule, system fault definition, particular functional systems details, etc. The SSRS addresses the safety functions from the SRD. The SSRS is identified as a requirement in document SNS-ASD-IC-P03.

16. System design: program configuration – tasks, safety tasks, routines, I/O routing, communications Should this be standardized? • Module Design: Based on the system design and SSRS. a module refers to pieces of field hardware that require software to collect inputs, process data, and provide outputs. A module could be a beam shut down station, a RAD detector, a trap key sequence, etc...

17. Independent review? • Coding: Is the process of writing program modules, as a standard each module may have its own routine, of be combined with other modules that contribute to similar functions. Documentation is added to identify each module and it’s components.

18. How formal? • Module Testing - Each module is tested individually and with the system to ensure a proper and excepted outcome.

19. Ideally will use an independent programmer in addition to two person programming team • Can review code to structure test plan • Has more latitude to change PPS equipment to facilitate testing • Semi-formal documentation- placed in Projectwise

20. Periodically testing software or just hardware faults? • Formal OPM procedure • Structured such that “A cave man can do it” • Performed per work control process annually and tracked (ASE requirement) • Written by an independent person • Intent is to perform “black-box” testing • Focus on testing interlocks that do not have built in diagnostics

21. Improvement initiative • Standardize format, test progression, signatures, etc. • Track safety functions per SSRS • Coordinating testing with other systems (TPPS) • Compartmentalized to facilitate testing • Including warnings and cautions in lieu of a JHA • Include brief operational description for reference

22. Future Improvements • Use resources from vendor’s PLC safety manual • Independent code review/ testing • Tracking of safety functions throughout documentation • Separate ACL for software items • Development of standard hardware/ software layouts for instruments

23. Are we there yet? MANAGEMENT PERFORMANCE ASSESSMENT • PROGRAM • PERSONNEL TRAINING & QUALIFICATION • QUALITY IMPROVEMENT • DOCUMENTS & RECORDS • WORK PROCESSES • DESIGN • PROCUREMENT • INSPECTION & ACCEPTANCE TESTING • MANAGEMENT ASSESSMENT • INDEPENDENT ASSESSMENT