OMSAP Presentation
Download
1 / 24

Agenda - PowerPoint PPT Presentation


  • 323 Views
  • Updated On :

OMSAP Presentation April 2004 Power Outage Events. David F. Duest Massachusetts Water Resources Authority Deer Island Treatment Plant Sept 2004. Agenda. Introduction DITP Power outage events – What happened? - April 3, 2004 - April 11, 2004 Impacts Corrective Actions.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Agenda' - Michelle


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Slide1 l.jpg

OMSAP PresentationApril 2004 Power Outage Events

David F. Duest

Massachusetts Water Resources AuthorityDeer Island Treatment Plant

Sept 2004


Agenda l.jpg
Agenda

  • Introduction

  • DITP Power outage events – What happened?

    - April 3, 2004

    - April 11, 2004

  • Impacts

  • Corrective Actions


Deer island wastewater treatment plant l.jpg
Deer Island Wastewater Treatment Plant

  • $3.8 Billion Construction Project

  • 2nd Largest in US

  • Treatment Capacity:

    • Maximum

      • 1.27 Billion Gal/Daycombined sewer system

      • 810 MGD by Secondary

    • Average Daily Flow:

      • 360 Million Gal/Day

  • Built on 200 Acres

    • includes 60 acre public access area

  • Removes 250 TPD of solids with 93 TPD for beneficial reuse

  • Consumes 18.5 MW daily with Peaks to 40 MW


Slide5 l.jpg

NStar Building

On Deer Island

NStar (South Boston)

K-Street Substation

115,000 Volts

Cross-Harbor Cable

Main Breaker

13,800 Volts

Main Breaker

MSB

A-Bus

B-Bus

CTG A

CTG B

STG

Deer Island Power Distribution

Thermal Power Plant

Support Facilities

North Pumping

South Pumping

Primary Treatment A & B

Primary Treatment C & D

Secondary A & B

Secondary C

Cryogenic Oxygen Facility

Residuals Treatment


Deer island thermal power plant including nstar building main switchgear building nmps wt headworks l.jpg
Deer Island Thermal Power Plantincluding NStar Building, Main Switchgear Building, NMPS & WT Headworks


What happened april 2004 l.jpg
What Happened – April 2004

  • Event 1: April 3, 2004 07:54 (Saturday)

    • NStar fault causes loss of “B-Bus”

      • Most significant DITP effect:

        • loss of North Pumping (NMPS) for approximately 2 hours 20 minutes

  • Event 2: April 11, 2004 19:23 (Easter Sunday)

    • NStar line-to-line fault at K-Street Substation causes complete power loss

      • Most significant DITP effects:

        - loss of all systems on DITP for approximately 2 hours 50 minutes


Event 1 april 3 2004 07 54 l.jpg
Event 1: April 3, 2004 07:54

  • Event – NStar fault causes loss of “B-Bus”

    • Leak in roof at NStar building caused the B-Bus circuit breaker to trip open.

  • DITP Effect –

    • lost 50% of most systems on DITP

      • Partial impacts on Pumping, Primary & Secondary

      • No impacts to disinfection/dechlorination

    • Lost 100% of pumping operations at NMPS


Slide9 l.jpg

April 3 Power Outage

NStar Building

On Deer Island

NStar (South Boston)

K-Street Substation

115,000 Volts

Main Breaker

13,800 Volts

Main Breaker

MSB

A-Bus

B-Bus

CTG A

CTG B

STG

Deer Island Power Distribution

Thermal Power Plant

Support Facilities

North Pumping

South Pumping

Primary Treatment A & B

Primary Treatment C & D

Secondary A & B

Secondary C

Cryogenic Oxygen Facility

Residuals Treatment


Slide10 l.jpg

April 3 Power Outage

NStar Building

On Deer Island

NStar (South Boston)

K-Street Substation

115,000 Volts

Main Breaker

13,800 Volts

Main Breaker

MSB

A-Bus

B-Bus

Correction Action

- Cross-tie A-Bus into B-Bus

- Power restored to B-Bus within minutes

CTG A

CTG B

STG

Deer Island Power Distribution

Thermal Power Plant

Support Facilities

North Pumping

South Pumping

Primary Treatment A & B

Primary Treatment C & D

Secondary A & B

Secondary C

Cryogenic Oxygen Facility

Residuals Treatment


Event 1 april 3 2004 07 5411 l.jpg
Event 1: April 3, 2004 07:54

  • B-Bus breakers reset at all pump stations

    • Pumps re-started at South System Pump Station, NMPS & Winthrop Headworks

  • North Main Pump Station – Pumps still unavailable

    • Staff reset breakers in pump station

    • Started Pumps / received pump fail alarms

    • Staff investigated alarms on lube oil systems

    • ID’d double failure in UPS at “Force Main Control Panel” (FMCP)

    • FMCP bypassed manually to re-energize VFD’s

  • Pumps started (phased startup) at 10:15

  • Station fully operational by 12:15

  • NStar switch replaced. B-Bus restored onto NStar at 20:00


Slide12 l.jpg

North Main Pump Station

VFD

Typical Permissives

VFD – bearing/winding temp, overload, vibration, FMCP

Pumps – Flow path, seal water, bearing temp, vibration, lube oil

Pump Shaft – bearing temp, vibration, lube oil

B-Bus

A-Bus

VFD

Motor Control Center (MCC)

VFD

VFD

VFD

Bypass

VFD

VFD

Force Main

Control Panel

UPS

VFD

VFD

Batteries

VFD

VFD = Variable Frequency Drive (controls speed of pumps through variable power input)


Event 1 april 3 2004 07 5413 l.jpg
Event 1: April 3, 2004 07:54

  • Overall Effect:

    • Collection system decisions made to PREVENT:

      • Flooding

      • Loss of life / injury

      • Property damage

      • Infrastructure damage

    • Remote Headworks isolated

    • Staff deployed to monitor levels in collection system

    • Limited storage in collection system (due to rain event)

    • Cottage Farm CSO facility re-activated. ~22 Mgal to Charles River

    • Other storm sewer overflow sites activated

    • All flow sent through DITP received full treatment


Event 2 april 11 2004 19 23 l.jpg
Event 2: April 11, 2004 19:23

  • Event – NStar line-to-line fault at K-Street Substation

    • NStar power unavailable to DITP until after 22:15 pm

    • 1st event of this type in the history of DITP

  • DITP Effect –

    • loss of all systems on DITP for approximately 2 hours 50 minutes


Slide16 l.jpg

April 11 Power Outage

NStar Building

On Deer Island

NStar (South Boston)

K-Street Substation

115,000 Volts

Main Breaker

13,800 Volts

Main Breaker

MSB

A-Bus

B-Bus

CTG A

CTG B

STG

Deer Island Power Distribution

Thermal Power Plant

Support Facilities

North Pumping

South Pumping

Primary Treatment A & B

Primary Treatment C & D

Secondary A & B

Secondary C

Cryogenic Oxygen Facility

Residuals Treatment


Slide17 l.jpg

April 11 Power Outage

NStar Building

On Deer Island

NStar (South Boston)

K-Street Substation

115,000 Volts

Main Breaker

13,800 Volts

Main Breaker

MSB

A-Bus

B-Bus

  • Correction Action

  • - Start CTG

  • DC power supply failure kept CTG from powering up DITP grid

  • Swithgear breakers needed to be closed manually (very hazardous)

  • Power restored to A-Bus ~22:00

  • 1st Pumps started 22:15

CTG A

CTG B

STG

Deer Island Power Distribution

Thermal Power Plant

Support Facilities

North Pumping

South Pumping

Primary Treatment A & B

Primary Treatment C & D

Secondary A & B

Secondary C

Cryogenic Oxygen Facility

Residuals Treatment


Event 2 april 11 2004 19 2318 l.jpg
Event 2: April 11, 2004 19:23

  • On CTG power from 22:15 until midnight

  • Midnight – while switching power back to NStar CTG tripped off-line.

  • Plant power / pumping restored (with phased startup) on NStar at 00:12

  • All systems reset / plant operating normally as of 01:30


Event 2 april 11 2004 19 2319 l.jpg
Event 2: April 11, 2004 19:23

  • Overall Effect:

    • All Headworks isolated

    • Staff deployed to monitor levels in collection system

    • Wastewater diverted to prevent flooding/backups (~3hrs)

      • North flows - Cottage Farm CSO (4.7 Mgal) to Charles River

      • South flows – NI Emergency outfalls opened – 17 Mgal raw sewage into Quincy Bay

    • All flow sent through DITP received full treatment


Slide21 l.jpg

Station number

Location Description

Enterococcus/100ml

E. coli/100 ml

4/13/04

4/13/04

4/12/04

4/12/04

10

<5

117

Mid Hingham Bay

90

40

20

<5

080

Quincy Yacht Club

210

110

5

<5

079

Outfall 103

150

105

5

<5

082

Outfall 101

95

35

45

45

081

Outfall 102

125

55

Water Quality Monitoring in Quincy and Hingham Bays after NI Emergency Discharge on April 11, 2004

Enterococcus limits (for swimming, marine water): geometric mean 35, single sample maximum of 104E. coli limits (for swimming, freshwater): geometric mean of 126, single sample maximum of 235


Slide22 l.jpg

Date

Time

Station

Location

Enterococcus/

100 ml

E. coli/100ml

04/12/04

11:53

005

Magazine Beach,

upstream of Cottage Farm

70

110

04/12/04

11:46

006

BU Bridge,

downstream of Cottage Farm

50

60

04/12/04

11:31

007

MIT boathouse

100

120

04/12/04

11:25

008

Harvard Bridge

20

40

04/12/04

11:16

009

Between Harvard and Longfellow

20

20

04/12/04

11:08

010

Downstream of Longfellow

10

50

04/12/04

10:56

011

Downstream of Science Museum

10

50

04/12/04

11:37

145

Stony Brook outlet

3700

2800

Water Quality Monitoring in Charles River after Cottage Farm Emergency Discharge on April 11, 2004

Enterococcus limits (for swimming, freshwater): geometric mean of 33, single sample maximum of 62E. coli limits (for swimming, freshwater): geometric mean of 126, single sample maximum of 235


Corrective actions l.jpg
Corrective Actions

  • MWRA retained two electrical engineering consulting firms to audit and review the incidents ($100K)

  • Retained an electrical testing firm to test all battery backup systems on DITP

  • Staff performed a thorough investigation of all systems involved in the events, independent of the consulting firms

  • Repairs made to major equipment that either failed during the events or were damaged as a result of the failure ($387K)

    • UPS at FMCP (secondary failure for April 3rd event)

    • DC Power system in Thermal Power Plant (secondary failure for April 11 event)

  • Facility cleanup ($57K)

  • NStar replaced roof (root cause of April 3rd event) on their building

  • Changed maintenance procedures for testing battery backup systems on DITP (UPS units) to identify issues before they become problems


Corrective actions continued l.jpg
Corrective Actions (continued)

  • MWRA has performed a follow-up black-start test for the thermal power plant systems (on August 10, 2004.) This test did not include any of the treatment systems and the thermal power plant was electrically isolated from the treatment plant systems and NStar. The new DC control system operated as designed.

  • Staff performed a detailed analysis of all potential Pump Station single-point of failure mechanisms that may occur.

  • Continue to work on implementing recommendations by the consulting firms. 18 out of 61 are fully implemented (as of 8/13).

  • Continue to work with DEP/EPA on all follow up actions

  • …more…


ad