Columbia Generating Station INPO AFI Recovery Dave Weber, Energy Northwest CMBG June, 2015

1. Columbia Generating Station INPO AFI Recovery Dave Weber, Energy Northwest CMBG June, 2015

2. CGS - INPO Area For Improvement (AFIs) September 2014 • AFI CM.2.1 - Design Calculation Backlog and Quality • AFI CM.3.1 - Engineering Service Provider Design Quality

3. AFI CM.2.1 - Design Calculation Backlog and Quality

4. CGS - INPO Area For Improvement (AFI) September 2014 • AFI CM.2.1 - Design Calculation Backlog and Quality. Some plant critical design bases calculations have not been maintained and some are of poor quality. This has resulted in some design bases without updated supporting calculations and technical evaluations, a large backlog of revisions to low-voltage calculations, and documentation control issues with calculations. Contributing, design basis calculation update standards are not reinforced.

5. INPO 12-013Performance Objectives and Criteria Operational Configuration Control • Performance Objective (CM.2) - Plant operation, maintenance, and testing activities are conducted in a manner consistent with the licensing and design bases and maintain configuration control. • Criteria - 1. Conditionsthat could result in structures, systems, or components being outside of operating and design requirements are evaluated and resolved on a schedule commensurate with the importance to safety and reliability. Special consideration is given to conditions that may challenge design bases or safety analyses.

6. Background History • January 2014 • Identified Design Calculations and > 5 Calculation Modification Records (CMRs) as a negative trend • Created Excellence Plan Focus Area, “Calculation Backlog Improvement” • January – September 2014 • Worked on calculation backlog as level of effort • July 2014 • Created Calculation Backlog Project • September 2014 • AFI in CM.2-1, Design Calculation Backlog and Quality • CR 315070 Severity Level A (RCE) on 9/27/2014

7. Root and Contributing Causes • Root Cause • RC1: Design Engineering did not recognize the vulnerability to configuration management with regards to calculations. • The AFI is based on three aspects: • Four legacy calculation discrepancies were identified during the 2013 CDBI Inspection • The design calculation backlog is higher than the industry standard • Approximately 150 calculations were reported during the plant evaluation as missing, illegible, or incomplete.

8. Root and Contributing Causes • Aspect one: Four legacy issues involving plant design documents without supporting calculations or technical evaluations. These issues were not self-identified, not corrected, and therefore challenge the design organization when addressing nonconforming conditions. • CR 292758 – Independence requirements IAW industry standards (IE/Non-1E Loads) • CR 292764 – No technical basis for turbulent mixing of diesel combustion air • CR 292760 – Operation of under voltage relays did not consider all design inputs • CR 292762 – Emergency core cooling system pumps’ limits were not included in emergency operating procedures.

9. Root and Contributing Causes • Aspect two: Three critical electrical calculations have a Calculation Modification Request (CMR) backlog of greater than 100: • Calculation E/I-02-95-01 Calculation for Overcurrent Protective Device Settings and Coordination for 480V Distribution System (146 CMRs), • Calculation 2.06.20 Calculation for Cable Ampacity Verification for Scheduled Power Cables (144 CMRs), • Calculation E/I-02-90-01 Calculation for Low Voltage System Loading (102 CMRs). • At the time of initial investigation, there were 37 calculations that had greater than 5 CMRs (efforts during this RCE reduced this number to 29), 30 calculations that have 5 CMRs, and approximately 3425 calculations that have one to three CMRs. This large backlog challenges the retrieval of the design basis of the affected systems when addressing nonconforming or degraded plant conditions or the development of design changes.

10. Root and Contributing Causes • Aspect three: Approximately 150 design bases calculations were identified as either missing in their entirety (hard and microfiche copies), incomplete (missing pages, appendices, or attachments) or illegible. During the evaluation, the missing calculations were located: however, the issue with missing pages, appendices, or attachments remains. • The plant evaluation team stated “Some plant critical design bases calculations have not been maintained and some are of poor quality.” Based on the examples given and the Design Engineering Manager’s discussions with the evaluator it was determined that ‘poor quality’ relates to the incomplete or illegible calculations listed on the spreadsheet and not technical quality. Therefore the poor quality piece is addressed under documentation control issues. Additionally technical quality issues related to design basis documents is being addressed under Root Cause Evaluation, CR 315069.

11. Root and Contributing Causes • The team identified that there has been a lack of priority applied to upkeep of the design bases calculations • With emphasis placed on issuing design modifications and/or emergent plant needs, configuration management suffers with a culture that has justified the Calculation Modification Request (CMR) process as meeting minimum standards.

12. Corrective Actions • Immediate and Interim Corrective Actions • Revised calculation procedure to clarify timeliness requirements of revising a calculation • Developed a work down curve for the outstanding calculation backlog • RC1/CC1/CC2/CC3 corrective actions • Dedicate Design Engineering resources that are focused on completing calculation updates • Revised calculation procedure to clarify timeliness and corrective action requirements for revising a calculation

13. RCE Corrective Actions • RC1/CC1/CC2/CC3 corrective actions • Create performance evaluation SMART goals for each supervisor • Create calculation backlog project plan • Create a calculation workdown curve performance indicator • Create a schedule addendum that will include calculations and milestone due dates • Drive the calculation backlog >5 CMRs to zero – due 12/31/2015

14. Effectiveness Review • Perform three Semi-annual effectiveness reviews • Review Supervisory observations, Condition reports and Performance indicators for validation – due 06/30/2016 Actions • AE hired to revise calculations with greater than 5 plant implemented CMRs • Weekly challenge calls with the AE to discuss calculation work focused on quality, cost and schedule

15. Design Calculation Backlog

16. Calculations > 5 CMRs PI

17. Sustainability • Set standards and expectations • Performance Appraisals • Technical Conscience • Procedure Improvements • Time based calculation updates not to exceed 1 cycle • Modifications developed to include calculation/ drawing update • Condition reports generated when CMRs >5 • Monitor Calculation Backlog with Performance Metrics • Annual funding for Configuration Management maintenance • Perform periodic effectiveness reviews

18. INPO Interviews - Drawing Backlog • January 2014 • Drawing backlogs for those categorized as “Update Upon Request” identified as a negative trend • January – September 2014 • Backlog reduction efforts were worked as level of effort • September 2014 • Drawing Backlog entered into the Design Engineering Excellence Plan • RCE for calculation backlog identified an extent of cause and condition issue regarding drawing backlog • Condition Report generated Background History

19. Actions Completed to Date • Develop Project Plan for resolving the drawing backlog • Funds obtained for FY15 • Project plan has been developed with schedule, resources and funding identified • Work down curves have been established • 7 contractors have been hired to work on drawing backlog • Additional AutoCAD and Raster Design licenses purchased to support new contract Drafters • Improved Performance Indicator • Funds obtained for FY16 • Monitor progress using work down curve and performance indicators metrics on a weekly and monthly basis

20. Current status • As of May, 2015 • 1886 of 10,200 drawings have been drafted • 1869 of 10,200 drawings have been checked • 1813 of 10,200 drawings are approved • Prioritization of System Drawings (Based on Maintenance Department) • Heater Drain 442 total drawings 97% Comp • Condensate 674 total drawings 95% Comp • Bleed Steam 320 total drawings 83% Comp • Seal Steam 111 total drawings 50% Comp • Heater Vents 225 total drawings 29% Comp • Misc Drains 175 total drawings 3% Comp

21. Drawing Backlog PI

22. AFI CM.3.1 – Engineering Service Provider Design Quality

23. CGS - INPO Area For Improvement (AFI) September 2014 • AFI CM.3.1 - Engineering Service Provider Design Quality. From November 2012 to September 2014, engineering service provider design packages and products have contained technical errors such as use of incorrect or incomplete design inputs, specifying use of obsolete equipment or components, missing affected drawings and calculations, and frequently omitting critical construction and installation details. Contributing to this, engineering supervisors do not consistently hold engineering service providers accountable to department standards for quality and schedule adherence.

24. INPO 12-013Performance Objectives and Criteria DESIGN CHANGE PROCESSES • Performance Objective (CM.3) - Changes to plant configuration, design, and licensing bases are evaluated, controlled, tested, and implemented while consistency is maintained among the physical plant configuration, design and licensing requirements, and the documented plant configuration. • Criteria - 1. The design authority is clearly identified and authorizes all plant design changes through a system of controls. Roles, responsibilities, and processes for developing, reviewing, and approving design and license bases changes are clearly defined and followed.

25. Background History • 2005 -Decision to use AEs • 2008 – AFI in CM.3-1, Weaknesses in oversight of external engineering products • 2010 -Self Assessment, AE Oversight • January 2014 • Identified Engineering Change Product Quality as a negative trend • Created Excellence Plan Focus Area, “Effectiveness of Design Engineering Changes” • April 2014 - INPO IER L1-14-20 • July 2014 - Created Procedure Upgrade Project • September 2014 • AFI in CM.3-1, Engineering Service Provider Design Quality • CR 315069 Severity Level A Root Cause (RCE) 9/27/14

26. Root and Contributing Causes • Root Cause • RC1: Design Engineering organization does not consistently demonstrate ownership of design quality to provide early engagement and consistent, accountable resources for AE oversight. • Contributing Causes • CC1: Engineering Management is not enforcing priorities to ensure the quality of design products prepared by the AEs. • CC2: AEs are not consistently holding themselves accountable for providing design products that meet EN standards.

27. Root and Contributing Causes • In 2005, EN transitioned to the use of external AEs to supplement in-house engineering resources(not in same time zone, not on site) • In October 2008, a plant evaluation determined that CGS had a weakness in the oversight of external engineering products (AR-CR 190612). • This ACE resulted in multiple corrective actions that included revising the EN Project Manager qualification, enhancement of design procedures to clarify EN expectations for product in-progress reviews and condition reporting expectations for the AE, better defined roles and responsibilities in design engineering procedures and contracts, and implementation of a Comment Control Form for formal communication between EN and AE firms • Transition AE to on-site presence

28. Root and Contributing Causes • In 2010, a Self-Assessment was completed to determine the effectiveness of the corrective actions from ACE 190612 (AR-SA 222381). The Self-Assessment determined that there were improvements made from implementing the administrative tools such as the Division of Responsibility form, Comment Control Form, and the Columbia Design Guidance Tool. However, the self-assessment also revealed there were still areas for improvement. The key findings determined that EN was still not holding the AEs accountable for product quality and EN had not effectively developed a resource loaded schedule to adequately develop design changes

29. Root and Contributing Causes • In 2014, the first quarter Engineering Department Roll-Up Meeting (DRUM) report identified an adverse trend in the number of field changes. The report indicated the Design Engineering Excellence Plan would address the causes and corrective actions going forward. The Design Engineering Excellence Plan identified six focus areas for the department, one of which was Effectiveness of Design Engineering Changes, identifying that AE design products do not consistently meet EN engineering standards for effectiveness regarding quality, cost, and timeliness

30. Root and Contributing Causes • In April 2014, a Level 1 event report was published, IER L1-14-20, Integrated Risk – Healthy Technical Conscience, relating to integrated risk, technical conscience principles and engineering fundamentals. The report discusses an adverse trend in technical errors and risk management that have resulted in serious organizational and operational impacts. The report shares lessons learned regarding insufficient technical expertise, critical design requirements and loss of margin, and insights from engineering vendors. The recommendation for Engineering Vendor Oversight and Support focuses on adhering to sound project management fundamentals. The attributes of the recommendation include communicating achievable and understandable objectives, assessing technical and project management proficiencies, and ensuring prompt reporting of events during project execution. • EN has approximately 50 assignments to address IER L1-14-20; 24 are complete and the others are scheduled to be completed by 9/25/2015

31. Root and Contributing Causes • EN uses AEs to develop engineering products such as design changes, calculations, and drawings for CGS. The AEs work under a Basic Ordering Agreement (BOA) contract with individual contract releases for each specific project scope. The AEs are expected to develop engineering products in accordance with EN procedures with oversight from EN project managers and design engineers. Design Engineering reviews and evaluates (also known as an owner acceptance review) in-progress and final AE engineering products using score sheets in accordance with DES-3-3, Formal Design Reviews. Once the AEs have completed the engineering products, they are approved by EN and implemented in accordance with applicable procedures. • Any issues or errors that are discovered after EN’s approval are generally resolved by Design Engineering as a design change revision or field change (not the AE). Each revision and field change is designated with one of the following cause codes for trending purposes; planned, scoping, material substitution, design error, construction convenience, installation error, administrative only, and user defined

32. Root and Contributing Causes • Specific Mods • Multiple design change revisions and field changes related to the Radwaste Building Exhaust Air (WEA) and Turbine Building Exhaust Air (TEA) modifications, • Installation difficulties and omissions, translation errors within design documents, Old mod sat on the shelf • Fast track modifications for Motor Control Center (MCC) Buckets and 50DHP-350 3000A (N/S) Breakers resulting in critical path delays

33. Corrective Actions • Immediate and Interim Corrective Actions • Briefed design engineers and AEs on gap and importance to EN • Interim corrective action – Perform self-assessment of R22 designs for technical errors • RC1/CC1/CC2 Corrective Actions • Change Roles Responsibilities and Accountabilities • Assign a single Energy Northwest (EN) engineer foreach AE design change to follow projects from cradle to grave • Modify design engineering and project management procedures to clarify roles and responsibilities

34. RCE Corrective Actions • RC1/CC1/CC2 corrective actions • Ensure future Basic Ordering Agreements (BOA)/ Contract Releases for the AEs reflect EN standards and quality, schedule, and cost • Engineering VP to provide briefing to Engineering Department to reinforce priorities for Design Engineering and emergent plant response expectations (after procedures revised) • Incorporate a SMART goal in Performance Appraisals for Design Engineering supervision • Onsite Engineering Service Providers to develop and implement a product quality recovery plan that meet EN standards

35. Effectiveness Review • Perform an interim corrective action effectiveness review following R22 refueling outage. • Review Engineering Management focused observations, corrective action program, performance indicators, product quality score sheets, and Comment Control Forms (CCFs) - 06/30/2016 • Perform a final corrective action effectiveness review. • Review performance indicators, quality score sheets for design changes, and field changes for implemented design changes - 12/31/2017

36. Procedure Upgrade Project • Procedures Upgrade Project • Three month effort to make large scale improvements to over 30 Design Engineering Procedures and 20 forms • Significant Upgrades/ Improvements • Implemented a 30% Conceptual Design Review Board • Implemented required quorums for all stakeholders • Implemented scorecards following both the Design Preparation (post 100%) and the modification implementation • Added performance at risk options based on product quality, timeliness, and cost behaviors • Addressed Enterprise Risk • Developed an engineering walkdown procedure • Training on these procedure changes is complete

37. Completed Excellence Plan Actions • Obtained FY15 funding to allow early start for high impact designs in first quarter 2015 • Bi-weekly Design Engineering schedule review meeting • Weekly meetings between DE manager and AE’s managers to discuss performance • Developed PI metrics on ECs • Engineering Field Changes (including design errors) • Root Cause Evaluation reviewed and approved on 01/06/2015

38. Actions Remaining • RCE (CR 315069) • Implement corrective actions • Perform effectiveness reviews • Monitor progress through PI Metrics • Engineering Change Field Changes (including design errors) • Product Quality Grading scores • Excellence Plan • Improve accountability through contract management • Perform snapshot assessment to ensure success

39. INPO 90-009, Guidelines for the Conduct of Design Engineering, Rev 3 • Field change requests are monitored and causes for field changes are identified and analyzed to provide a basis for improving the design change process. Implementing appropriate corrective actions should decrease the number of field changes needed for future modifications. Experience has shown that large numbers of field changes are frequently caused by inadequate walkdowns, inaccurate or incomplete as-built drawings, or incomplete designs resulting from tight scheduling or insufficient planning. • Field change requests for ease of construction (Construction Convenience) should be minimized, because the cost of analysis, documentation (e.g. rework to the EC package, WO task package, QC reviews, etc.), and work delays related to the field change often will exceed the expected benefit

40. CGS - Cause Code DefinitionsConstruction Convenience USA KPI Cause Categories

41. CGS - Cause Code DefinitionsDesign Error

42. CGS - Cause Code DefinitionsMaterial Substitution/Scoping

43. Functional Design Documents ‑ Documents which portray design criteria and schematic arrangements for the purpose of establishing and conveying the operational (functional) aspects of Structures, Systems, and Components. Examples: Elementary Diagrams, Flow Diagrams, Logic Diagrams, Safety Related Equipment Lists, Design and Safety Calculations, One‑line Diagrams, Engineering Standards that implement Criteria, and Procurement Specifications. Physical Design Documents ‑ Documents which convey physical layout (configuration, organization of elements) and arrangement requirements. Physical Design Documents interpret the functional requirements given in the Functional Design Documents into configuration requirements of Structures, Systems and Components. Examples: Connection Diagrams, Piping ISOs, Mechanical Layout Drawings, Cable Schedules, Raceway Plan Drawings. Functional/Physical Definition

44. EC Field Changes PI (Month to Month)

45. EC Field Changes PI (Month to Month)

46. EC Field Changes PI (6 Months Rolling)

47. Sustainability • Set standards and expectations • Performance Appraisals • Technical Conscience • Procedure Improvements • Conceptual Design Review Board implementation • Scorecard Improvements establishing performance at risk based on quality, cost and schedule performance • Design Walkdown implementation • Monitor Field Changes through Performance Metrics • Perform periodic effectiveness reviews (self-assessments)

48. Excellence Plan • Calculation Improvement • No calculations >5 CMR’s 12/31/2015 • Margin Management • Technical Conscience Self Assessment • Regulatory Margin • CDBI Preparation/ Downgraded systems • Effectiveness of Design Engineering Changes • Procedures upgraded/ Self Assessments on schedule • Design Engineering Backlog • Drawing backlog Reduction 08/31/2017