Harris Plant – Loss of Main Feed Pump Lube Oil

1. Harris Plant – Loss of Main Feed Pump Lube Oil INPO ICES #305656 HNP Root Cause Evaluation 598302

2. Presentation Outline • Main Feedwater Pump Details • Loss of Lube Oil Event Description • Lube Oil System Overview • Related History • Event Causes • Lessons Learned

3. Main Feedwater Pump Details • Each pump provides 50% capacity. • Flowserve (Ingersoll-Dresser) 16X19 CA-2 • 2 Stage - Barrel Type • 9000 HP Motor Driven • Flow 14,836 GPM • Developed Head 2,216

4. Event Desciption • On Friday afternoon 3/29/2013 around 13:00, a member of the housekeeping staff called the control room and reported an unusual amount of oil around the base of the ‘B’ Main Feed Pump. • Operations found a substantial amount of oil in the catch basin under the pump and motor, no level indicated in the reservoir, but normal flow indicated in the bearing sight glasses. • Operations then entered into Abnormal Operating Procedure for a rapid downpower, reduced power to approximately 55%, and secured the ‘B’ MFP. • HNP declared a Unit Threat and the Outage Control Center was staffed.

9. Motor Bearings (2) 7 8 5 Orifice (2) Sight Glass (2) 6 Sight Glass (2) PS 2143 Pump Bearings (2) PS 2142 12 PS 2141 Shaft Driven Pump 4 PI 1 9 12 LIC Aux Pump 3 V 2 CUNO Filter 10 11 MFP Lube Oil System Schematic

10. Lube Oil System Illustration (not including MFP)

11. Inboard Motor Bearing with Casing Disassembled

12. Oil Slinger Ring Arrangement

13. Outboard Motor Bearing Showing Connected Hoses

14. Related History • Neoprene stoppers installed in 1991 in MFP Motor bearing inspection ports to stop weepage and associated housekeeping problem without any formal evaluation. • Breathers were first installed on the MFP’s in 2004 to provide better filtration of contaminants. Subsequently it was determined these new breathers are subject to clogging from oil mist. • Several historical WO’s described oil leaks, which were cleaned up, with no cause of leak determined. • Mechanics began to slit the paper element in MFP breather in 2006 to eliminate oil leaks on the MFP Motors. There was no engineering evaluation or WO identified to direct this practice. • Site had grown tolerant of small amounts of leakage from motor and did not pursue cause of leakage. • Other new uses of breathers on site were also not trouble free. • Type of breather changed from paper element to desiccant type about a month before event.

15. Event Causes • Root Cause: • Inadequate breather installed on ‘B’ MFP oil reservoir • Contributing Causes: • Specific design information for MFP lube oil sub-system not obtained. • MFP oil system pressure control valve return line dumps oil above the liquid surface in the oil reservoir resulting in aeration. • Failure to effectively use available OE/CAP (clogged breathers can cause oil leaks). • Lack of ownership of oil reservoir breathers within the Predictive Maintenance Lube Oil program. • Management follow-up or monitoring of activities did not identify problem.

16. Lessons Learned • Do not make changes to equipment without involving OEM. • Things can be more complex than they first appear. • Fully understand “what is the worst thing that can happen” before making changes. • Remember the old adage “If it ain’t broke …………………. • Aggressively pursue small signs that something is not quite right. • Don’t let time demands/pressure cause you to accept good news or explanations because they relieve you of adding another thing to your plate.