Tropical storm allison recovery
Download
1 / 45

Time Line - PowerPoint PPT Presentation


  • 242 Views
  • Updated On :

Tropical Storm Allison Recovery Charles Chambers Information Technology 2001 Time Line Saturday flooding begins (1 A.M.) – IT team on campus Additional IT staff arrive – pumping and power are critical Institutional priorities are focused – Non-IT Sunday

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Time Line' - paul2


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Tropical storm allison recovery l.jpg
Tropical Storm AllisonRecovery

Charles Chambers

Information Technology

2001


Time line l.jpg
Time Line

  • Saturday

    • flooding begins (1 A.M.) – IT team on campus

    • Additional IT staff arrive – pumping and power are critical

    • Institutional priorities are focused – Non-IT

  • Sunday

    • Pumping of network node facilities begin

    • Power restored to UHS data center

    • Internet connection reestablished

    • 713-743-1000 call center established

    • Systems and servers returned to service


Time line3 l.jpg
Time Line

  • Monday, June 11th

    • WAN services restored except UHV

    • IP telephones activated in critical areas

    • Relocated some servers to Data Center

  • Tuesday, June 12th

    • Internet connectivity unstable due to ISP

    • Servers in Data Center grow by 20

    • UHV network connection restored

    • Desktop assessment team inventories campus

    • Network teams inspect buildings for damage


Time line4 l.jpg
Time Line

  • Wednesday, June 13th

    • UH-CVN services restored, classes held.

    • Payroll processed

    • 50% of phone system returned to service

  • Thursday, June 14th

    • Data backbone restored

    • Data connectivity to opened buildings

    • Order Equipment for Storm Solutions for Hofheinz Pavillion


Time line5 l.jpg
Time Line

  • Friday, June 15th

    • Majority of phone system restored

  • Saturday, June 16th

    • Equipment for Hofheinz Pavillion arrives, installed


Time line6 l.jpg
Time Line

  • Sunday, June 17th

    • Hofheinz Pavillion available

      • 270 IP telephones

      • 314 data ports

      • IT support booth created

  • Monday, June 18th

    • University Re-Opens

    • First occupants moves into Hofheinz Pavillion


Time line7 l.jpg
Time Line

  • Tuesday

    • Voice Mail system restored

    • Touch Tone Registration restored

    • Remainder of phone system restored

  • Wednesday

    • Majority of SWB trunking problems corrected


Time line8 l.jpg
Time Line

  • Thursday

  • Friday

    • Reconfiguration of Hofheinz Pavillion network


Technology priorities l.jpg
Technology Priorities

Infrastructure

  • Phone System

  • Internet Connectivity

  • Inter-campus Network

  • Campus Data Network

  • Interactive Video Network

  • Data Center Operation


Technology priorities10 l.jpg
Technology Priorities

Services

  • E-mail

  • Web Site

  • WebCT

  • IVR for Registration/Records

  • Payroll

  • Financial Application


Technology priorities11 l.jpg
Technology Priorities

Desktop Computers

  • Inventory

  • Repair

  • Replacement Computers

  • Data restores


Technology priorities12 l.jpg
Technology Priorities

Temporary Space - Holfheinz

  • Used basketball pavilion

  • Boxes became offices

  • Basketball floor

    • became labs and cubicles


Holfheinz pavillion l.jpg
Holfheinz Pavillion

From Basketball to Academics

  • 27 boxes

  • Cisco IP telephony – 270 phones

  • Switched data network

    • 270 ports in offices

    • 144 ports on basketball court

    • Over 100,000 feet of copper and fiber

  • IT support booth

    • Voice and data network support

    • Desktop computer support


Recovery resources l.jpg
Recovery Resources

  • Over xxx person hours (UH Staff)

  • Loss and Business Resumption

    • Costs estimated in millions

  • Over 250 IT related staff involved


Lessons learned l.jpg
Lessons Learned

  • Move IT facilities above ground

  • Move generators above ground

  • Flood doors on underground entrances

  • Established plans helped . . .

    • The right people at the right places, making the right decisions was the key.

  • Mobilize support efforts early

    • Food

    • Sleeping Quarters


Heroic efforts l.jpg
Heroic Efforts

  • Pumps/Pumping

  • Saving the Animals

  • Getting Primary DID in service

  • Internet Connectivity

  • Holfheinz Pavillion


How the speedy recovery l.jpg
How the Speedy Recovery

  • Temporary fixes

  • Great minds and enormous goals

  • Dedicated staff

  • Very long hours

  • Multiple plans

  • Quick vendor response

  • Elimination of bureaucracy

  • Communication of status and goals

  • Central coordination point


Starting recovery l.jpg
Starting Recovery

  • Pumping and Power

  • Starting pumping PGH on Saturday

  • Moved pumps to animal lab (SR2)

  • Sunday power restored to Computing Center

  • Internet Connection

    • Reconfigured to computing center

    • Web and e-mail up first


Recovery partners l.jpg
Recovery Partners

  • Herculean efforts by all!

    • Pumping water

    • Removing damaged equipment

    • Designing and installing new systems

    • Power restoration

  • UH Support

    • College volunteers

    • Facilities

  • Vendor response was great!

    • Staff on site (over 100)

    • Rapid deployment of equipment


Initial impact l.jpg
Initial Impact

  • Primary Technology Areas Affected

    • UHS Data Center

    • Network Node in PGH

    • Network Node in UCU

    • Headend Facility in Moody


Uhs data center l.jpg
UHS Data Center

  • Lost power

  • No flooding or other damage


Network node pgh l.jpg
Network Node (PGH)

  • Services lost

    • Voice, data, video for UH

    • Internet and interactive video for all UHS

  • Damage

    • Water 18 inches above raised floor

    • Power systems total loss


Network node pgh23 l.jpg
Network Node (PGH)

Telephone and Voice Mail Systems


Network node ucu l.jpg
Network Node (UCU)

  • Services lost

    • Voice network for 1/3 of campus

    • Trunk lines from SWB

    • High capacity circuits from SWB

  • Damage

    • Telephone Switch corroded

    • Wire center destroyed

      • 3600 stations


Network node ucu25 l.jpg
Network Node (UCU)

Telephone system


Network node ucu26 l.jpg
Network Node (UCU)

SWB

Interface


Network node ucu27 l.jpg
Network Node (UCU)

Wire Center Frame


Headend facility moody l.jpg
Headend Facility (Moody)

  • Services Lost

    • CATV Network


Hofheinz pavillion l.jpg
Hofheinz Pavillion

Home of “Phi Slamma Jama”


Hofheinz pavillion30 l.jpg
Hofheinz Pavillion

Wiring was pretty open…
















Here s who did it l.jpg
Here’s Who did it! Sanford Street

CTS “Crew” at Storm Allison Luncheon


ad