1 / 27

September 11

September 11. What Worked, What Didn’t Sean Donelan Donelan.COM Critical Infrastructure Design. Introduction. Impact on the Internet Rumors Causes What worked What didn’t work Duct tape solutions Recommendations. Names Omitted.

dane-riddle
Download Presentation

September 11

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. September 11 What Worked, What Didn’t Sean Donelan Donelan.COM Critical Infrastructure Design

  2. Introduction • Impact on the Internet • Rumors • Causes • What worked • What didn’t work • Duct tape solutions • Recommendations

  3. Names Omitted • Individual company names omitted, unless there is only one company • Building addresses used if well-known location • General description of problems or vulnerabilities

  4. Killed and Missing • World Trade Center: 445 people confirmed killed, 4500 to 5000 people missing • Pentagon: 125 killed • American Flight 11: 92 killed • United Flight 175: 65 killed • American Flight 77: 64 killed • United Flight 93: 44 killed • Estimated 2,600 citizens from 80 countries included in above numbers

  5. Impact on the Internet • The Internet wasn’t a target • You aren’t a Tier-1 provider if you weren’t affected by something • Limited network partitioning US/Europe • Local impact ranged from complete destruction to no impact • Most network disruptions happened hours after the initial attack • Most service disruptions due to problems in edge networks

  6. Rumors • 60 Hudson structurally unsound • FBI seizing ISP equipment “supporting” terrorist web sites • Military taking over satellite transponders shutting down ISPs • Carrier/Ryder trucks missing/stolen • Carnivore slowing down the Internet • Terrorists knew the code name for Air Force One

  7. Yogi Berra It ain’t over, till its over.

  8. Causes • “Normal” disruptions like maintenance, fiber cuts, tropical storms, and crackers continue • Loss of third-party infrastructure • Operator errors & omissions • Exceeded environmental design • Direct damage due to the attack • Software bugs/Hardware failures • Lack of coordination/planning/information • Lack of auto-start/auto-boot

  9. Gross Performance

  10. What WorkedInternet • Undamaged portions of the Internet continued to function (mostly) • TCP/IP worked (best-effort delivery) • BGP routing worked • Multicast routing worked • Core application protocols (DNS, E-mail) worked • VOIP (excess capacity, NMC bypass) • Packet wireless, Blackberry, Richochet, 802.11b • Carrier Hotels/Colo’s

  11. What WorkedContent • IRC used to feed live news captions • Instant Messenger usage increased by and estimated 20% • Mirroring/Local caches • Corporate web sites distributed updated information. Non-Internet companies seemed to use the web more effectively immediately after attack • Charity fundraising from web sites with help from some e-commerce sites • SPAM, SPAM, SPAM

  12. O’Toole’s Commentary on Murphy’s Law Murphy was an optimist.

  13. What Didn’t WorkComplex Services • Load-balancing products replaced with DNS round-robin • Generated web pages replaced with direct load pages • Software disk mirroring product didn’t automatically recover after power failures • Analog lines repaired first

  14. What Didn’t WorkSecurity & Authentication • Dialup authentication problems • Connect, but couldn’t login • Central authentication servers were located in other regions • Several register/pay news web sites suspended authentication checks (public service, improved performance) • Difficulties verifying authenticity of requests from the “government” (possible social engineering or just FUD)

  15. What Didn’t WorkCongestion • Its so crowded, no one goes there anymore • Well-known news web sites initially overloaded (cached by other sources) • Government web site overloaded (FBI tip site) • NANOG and other mailing lists posting delays, but did deliver • Unicast (distributed and single source) streaming news sources overloaded • Generally a point-source problem • Not a backbone capacity issue (yet)

  16. What Didn’t WorkPOTS/Voice • “Worked” but did calls get through? • Carrier 1-800 call problems • Cell sites depend on landlines • ILEC versus CLEC access • ISPs established new dialup numbers replacing out of service numbers • Call centers were evacuated, who answered the phones

  17. What Didn’t WorkNew York City • Network-wide effects • Physical damage in New York City • Network problems in New York City • Pentagon and Western Pennsylvania are not major public Internet hubs

  18. NYC Damage Map

  19. What Didn’t WorkThe net needs electricity • Electric substations and grid damaged • Outside plant carrier equipment not connected to the best available backup power source • Batteries don’t last a week • Generator failures • Operator turned off generator to save fuel • Fuel delivery problems • Lack of maintenance • Environment exceeded design conditions • Cooling (HVAC) equipment power supply

  20. What Didn’t WorkRedundancy & Spares • If only a single circuit exists and it is destroyed, no IP traffic • Most end-users connected by a single circuit • Multi-homing versus a second circuit • Limited spare parts stored locally, rely on overnight couriers for replacement parts from central parts depots • Non-revenue generating equipment

  21. What Didn’t WorkDiversity & Avoidance • Equipment in the World Trade Center primarily served tenants in complex (shared fate) • SONET ring through WTC tower 1 and alternate path through WTC tower 2 • Damage to 140 West Street central office and surrounding underground infrastructure • Backup circuit routed through same facility • “Advanced” data circuits (ISDN/DSL) concentrated in a few central offices

  22. Duct Tape Solutions • Cables out windows and manholes and along streets • Carriers shared working facilities in telco hotels to restore service, more carriers generally means more facilities • Carrier provided emergency transit to ISPs in Europe to heal breaks in NYC • ConEd organized generators and fuel truck route for many buildings • Lots of offers of assistance

  23. Blaise Pascal People are generally better persuaded by the reasons which they have themselves discovered than by those which have come in to the mind of others.

  24. Recommendations • Rumors will happen, must actively share information to combat it • Update government response plans to include the Internet and post-1982 telecommunication carriers • Automatic/Remote operation of backup systems in case of evacuation • Plan for customer service during evacuation of call centers

  25. More Recommendations • Pre-plan emergency access with authorities, building owner, etc • Pre-plan load shedding procedures to prevent shutting off critical equipment (Note specify “critical equipment”) • “Outside plant” network transport equipment should be connected to building generator(s)

  26. Net Recommendations • Operators are dangerous, do nothing? • Weakest link, know your circuits • Centralized login can create a denial of service vulnerability during a crisis • Using ISDN for out-of-band access may delay recovery • Simple services work best in a crisis • Diversity, Diversity, Diversity

  27. What WorkedWhat Didn’t Work Questions??? Sean Donelan

More Related