disaster recovery business continuance sept 11 lessons learned l.
Skip this Video
Loading SlideShow in 5 Seconds..
Disaster Recovery / Business Continuance Sept 11 Lessons Learned PowerPoint Presentation
Download Presentation
Disaster Recovery / Business Continuance Sept 11 Lessons Learned

Loading in 2 Seconds...

play fullscreen
1 / 16

Disaster Recovery / Business Continuance Sept 11 Lessons Learned - PowerPoint PPT Presentation

  • Uploaded on

Disaster Recovery / Business Continuance Sept 11 Lessons Learned. Introduction. Material comes from first hand experiences 25 EMC Customers Affected on Sept. 11 All references to people and companies have been removed except: Citigroup, Deutsche Bank, JP Morgan, Commerz Bank

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Disaster Recovery / Business Continuance Sept 11 Lessons Learned' - albert

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
  • Material comes from first hand experiences
    • 25 EMC Customers Affected on Sept. 11
  • All references to people and companies have been removed except:
    • Citigroup, Deutsche Bank, JP Morgan, Commerz Bank
  • Lessons Are Grouped By:
    • Business Resumption
    • Application Recovery
    • Data Recovery
  • Conclude with summary of EMC’s emergency response
recent lessons learned summary
Recent Lessons Learned-Summary
  • Family First then Business!
  • Data Mirroring Technology worked!
    • SRDF failover was 100% successful
    • BUT…people and processes were not always in place to ensure application recovery and business resumption
  • Vendor support personnel and capabilities are extremely important to a successful recovery
    • In a crisis vendor support staff are critical
    • Be sure to include them in building and testing your recovery plans
summary cont d
  • Don’t forget to plan for ALL three major types of threats:
    • Physical
    • Natural disaster
    • Electronic
  • Be sure your plans cover all three major aspects:
    • Data Recovery
    • Application Recovery
    • Business Resumption
  • Never Enough Testing and Documentation
business resumption
Business Resumption
  • Process for declaring Disaster
    • Need clear chain of command (plus offsite alternatives)
    • Predetermine rules and goals
  • Press Relations
    • Assign someone to speak on behalf of the company. People will need information about their family, friends, jobs and state of the company.
business resumption6
Business Resumption
  • Communications
    • Bandwidth will become especially taxed. Cell phones clogged, landlines busy. Don't depend on solid communications during a disaster.
    • Two way radio communications - the walkie-talkie cell phones would assist DR operations.
    • Satellite/cell phones may support communications outside the effected area
    • Backup web site for employee communications
  • Command Post
    • The availability, location and access by people to the operations command post is critical.
business resumption7
Business Resumption
  • Airports
    • Any part of your DR process that includes access to air travel will be effected. People and tapes expecting to catch flights bound for DR sites were delayed by days. This will also effect the Supply Chain as well.
  • Intellectual Capital
    • Protect your people and information.
    • Executive management has become acutely aware of their fiduciary obligations to protect assets on behalf of the shareholders and employees.
    • Corporate insurance will be reviewed to include coverage for violent acts and coverage for anyone else on the company premises.
business resumption8
Business Resumption
  • Disperse your DR support teams
    • 2 companies lost their entire DR staff in the attack and were less able to recover operations elsewhere as a result
  • Loss of Worksite
    • People need a place to contribute to the recovery and they will also need a command post too. Size of the facility is important too.
business resumption9
Business Resumption
  • Supply Chain Impact
    • Airports and transportation systems will impact suppliers, vendors and delivery
    • Supply Chain process must be re-adjusted for something other than JIT delivery.
  • Payroll
    • People will need access to their wage to cover very immediate needs. Food, Shelter, clothing etc.
  • Automate the business processes
    • If the people aren't able to operate the business, then the more automation built into the business infrastructure the more able the company will be able to recover, re-synchronize and continue operations. The intellectual capital to run the business may not be available.
application recovery
Application Recovery
  • Business Continuance and Disaster Recovery (BCP) should be a part of the Applications Development and Deployment process.
    • If left to the LOB manager or IT staff then the application may not have been included in the DR plan or possess any continuity attributes. BCP should added to the Project Life Cycle process.
  • E-mail is critical
    • With the loss of communications and the workplace, workers will need to work from home. Without e-mail systems they were idle.
    • Important for both internal AND external communications to customers, suppliers and even the press.
data recovery
Data Recovery
  • Replacement of the Primary Site
    • Make arrangements for a new recovery or primary facility. Running unprotected during this time is a big risk factor
  • Back-up the lap-tops and PCs
    • Many production jobs are in fact run on PCs (large spreadsheets) for many financial/actuarial applications.
    • These were also needed for the DR procedures to be able to recover.
    • And they weren't backed-up
data recovery12
Data Recovery
  • All of the Paper Records have disappeared
    • Strong visual for me on this one, dust clouds swirling around tons of paper. In addition to the loss of many electronic forms of data, the paper source/back-up will also be effected.
  • Mirror more data
    • The cost to add the "other" applications is in retrospect, minimal and well worth the investment.
    • While many companies replicated data in real time and it was very successful. The impact of relying on tape (other sources too) to restore the "lesser" applications negatively impacted the time to recover.
    • "Lesser" Applications that were integrated into larger Enterprise Business Applications are now by default mission critical too.
    • Sharing information among applications, platforms and networks is an important consideration for recovery
data recovery13
Data Recovery
  • First Come First Served Access to Vendor Recovery Centers
    • Lack of access to the favorite local, primary Recovery Sites will force an unexpected travel event. Any specialized equipment may also need to be re-located to new recovery site, further delaying the recovery
  • Test Frequently
    • An excellent infrastructure mismanaged or ill operated is not much better than a poorly designed process.
    • Operational Familiarity is key, the more people trained and dispersed the better the plan
data recovery14
Data Recovery
  • A Change management system integrated into the DR process will ensure that recovery included the last change.
  • Asset management becomes critical because you'll need to know what you have installed and how it's configured. Changing configurations should be included in change management.
emc emergency response
EMC Emergency Response
  • Established dedicated crisis support center
    • 24 X 7 in Hopkinton, affected areas got immediate escalation
    • Extra people and parts dispatched to NJ
  • EMC facilities offered for customer recovery
  • Emergency product shipments
    • Used customer config DB to configure replacement units
    • 1st replacement shipped and installed night of Sept 11!
    • Additional 13 units shipped and delivered on Sept 12