Disaster Recovery / Business Continuance Sept 11 Lessons Learned

1. Disaster Recovery / Business ContinuanceSept 11 Lessons Learned

2. Introduction • Material comes from first hand experiences • 25 EMC Customers Affected on Sept. 11 • All references to people and companies have been removed except: • Citigroup, Deutsche Bank, JP Morgan, Commerz Bank • Lessons Are Grouped By: • Business Resumption • Application Recovery • Data Recovery • Conclude with summary of EMC’s emergency response

3. Recent Lessons Learned-Summary • Family First then Business! • Data Mirroring Technology worked! • SRDF failover was 100% successful • BUT…people and processes were not always in place to ensure application recovery and business resumption • Vendor support personnel and capabilities are extremely important to a successful recovery • In a crisis vendor support staff are critical • Be sure to include them in building and testing your recovery plans

4. Summary-cont’d • Don’t forget to plan for ALL three major types of threats: • Physical • Natural disaster • Electronic • Be sure your plans cover all three major aspects: • Data Recovery • Application Recovery • Business Resumption • Never Enough Testing and Documentation

5. Business Resumption • Process for declaring Disaster • Need clear chain of command (plus offsite alternatives) • Predetermine rules and goals • Press Relations • Assign someone to speak on behalf of the company. People will need information about their family, friends, jobs and state of the company.

6. Business Resumption • Communications • Bandwidth will become especially taxed. Cell phones clogged, landlines busy. Don't depend on solid communications during a disaster. • Two way radio communications - the walkie-talkie cell phones would assist DR operations. • Satellite/cell phones may support communications outside the effected area • Backup web site for employee communications • Command Post • The availability, location and access by people to the operations command post is critical.

7. Business Resumption • Airports • Any part of your DR process that includes access to air travel will be effected. People and tapes expecting to catch flights bound for DR sites were delayed by days. This will also effect the Supply Chain as well. • Intellectual Capital • Protect your people and information. • Executive management has become acutely aware of their fiduciary obligations to protect assets on behalf of the shareholders and employees. • Corporate insurance will be reviewed to include coverage for violent acts and coverage for anyone else on the company premises.

8. Business Resumption • Disperse your DR support teams • 2 companies lost their entire DR staff in the attack and were less able to recover operations elsewhere as a result • Loss of Worksite • People need a place to contribute to the recovery and they will also need a command post too. Size of the facility is important too.

9. Business Resumption • Supply Chain Impact • Airports and transportation systems will impact suppliers, vendors and delivery • Supply Chain process must be re-adjusted for something other than JIT delivery. • Payroll • People will need access to their wage to cover very immediate needs. Food, Shelter, clothing etc. • Automate the business processes • If the people aren't able to operate the business, then the more automation built into the business infrastructure the more able the company will be able to recover, re-synchronize and continue operations. The intellectual capital to run the business may not be available.

10. Application Recovery • Business Continuance and Disaster Recovery (BCP) should be a part of the Applications Development and Deployment process. • If left to the LOB manager or IT staff then the application may not have been included in the DR plan or possess any continuity attributes. BCP should added to the Project Life Cycle process. • E-mail is critical • With the loss of communications and the workplace, workers will need to work from home. Without e-mail systems they were idle. • Important for both internal AND external communications to customers, suppliers and even the press.

11. Data Recovery • Replacement of the Primary Site • Make arrangements for a new recovery or primary facility. Running unprotected during this time is a big risk factor • Back-up the lap-tops and PCs • Many production jobs are in fact run on PCs (large spreadsheets) for many financial/actuarial applications. • These were also needed for the DR procedures to be able to recover. • And they weren't backed-up

12. Data Recovery • All of the Paper Records have disappeared • Strong visual for me on this one, dust clouds swirling around tons of paper. In addition to the loss of many electronic forms of data, the paper source/back-up will also be effected. • Mirror more data • The cost to add the "other" applications is in retrospect, minimal and well worth the investment. • While many companies replicated data in real time and it was very successful. The impact of relying on tape (other sources too) to restore the "lesser" applications negatively impacted the time to recover. • "Lesser" Applications that were integrated into larger Enterprise Business Applications are now by default mission critical too. • Sharing information among applications, platforms and networks is an important consideration for recovery

13. Data Recovery • First Come First Served Access to Vendor Recovery Centers • Lack of access to the favorite local, primary Recovery Sites will force an unexpected travel event. Any specialized equipment may also need to be re-located to new recovery site, further delaying the recovery • Test Frequently • An excellent infrastructure mismanaged or ill operated is not much better than a poorly designed process. • Operational Familiarity is key, the more people trained and dispersed the better the plan

14. Data Recovery • A Change management system integrated into the DR process will ensure that recovery included the last change. • Asset management becomes critical because you'll need to know what you have installed and how it's configured. Changing configurations should be included in change management.

15. EMC Emergency Response • Established dedicated crisis support center • 24 X 7 in Hopkinton, affected areas got immediate escalation • Extra people and parts dispatched to NJ • EMC facilities offered for customer recovery • Emergency product shipments • Used customer config DB to configure replacement units • 1st replacement shipped and installed night of Sept 11! • Additional 13 units shipped and delivered on Sept 12

16. Closing Slide