Contingency Processing Strategy Report for Information Systems Services March 1988

1. Contingency Processing Strategy Report for Information Systems Services March 1988

2. CONTENTS Background Objectives and Scope Approach and Methodology Findings and Recommendations Data Processing Item Processing Data Communications Summary of Recommendations

3. Background (cont’d) BIA recommended that: • Deposit processing be normalized within 3 days. • Loan and other key operations be normalized within 8-10 days. • Other production operations be normalized within 14 days.

4. Estimated Deposit Erosion

5. Background Business impact Analysis (BIA) concluded that without Mainframe Data Processing — • Depositor Base would start to erode at Day 3 after disaster because of customer frustration caused by: — lengthened service queues — degraded service and Information availability • Reduced deposit levels would increase cost of funds by 200 to 250 basis points for the eroded deposit amounts. — cost= $700 million over 10 years • The difficulty of manual operations/controls and lack of current balances would: — paralyze deposit operations before two weeks — accelerate rate of depositor erosion.

6. Background (cont’d) BIA Conclusions (cont’d) • Loan imitation activity could continue for a month at reduced levels of efficiency. Where current balance information is required, loan operations would break down more rapidly: — portfolio packaging — loan payoffs — servicing • All accounting operations would stop, except for manual processing in problem areas. • After two weeks, dunning activities would stop. • Estimated losses from an outage are estimated at: — 3-day outage—nominal — 2-week outage — $25,000,000 — 2-month outage—Association Is no longer a viable economic entity

7. Objectives and Scope of Strategy Objectives of Disaster Recovery Strategy* • Determine alternative emergency compute operations scenarios that would permit bank to fulfill Business Impact Analysis (BIA) recommendations. • Assist bank management in selecting the alternative emergency computer operations approach that best permits rapid and effective Implementation. • Prepare an Implementation Plan for the selected emergency computer operations approach. * Disaster Recovery Project Task Ill. A deliverable.

8. Approach and Methodology • Developed jointly with bank team feasible alternative scenarios that meet BIA recommendations for emergency computer operations and telecommunications after a disaster. • Assisted bank team in selecting a strategy that effectively and efficiently permits rapid implementation of contingency processing capability. • Prepared Draft Disaster Recovery Strategy Report and reviewed with Project Team.

9. Findings: Data Processing Alternatives Available Data Processina backup resource alternatives include: Meets BIA Requirements A. Dual Data Centers • Capable of resuming processing within 1-2 shifts B. Vendor provided hot/cold site - Capable of resuming deposit processing within 3-4 days Does Not Meet BIA Requirements C. Vendor provided cold site - Capable of resuming processing within 3-4 weeks D. Identified/Unprepared site - Capable of resuming processing within 6-8 weeks

10. Findings: Data Processing Alternatives Alternatives that Meet BIA Requirements I. Dual Data Centers (California and Florida) Pro: - Permit resumption of Deposit and Loan processing within 1 day and other production processing within 2 weeks - Complete protection against local and area-wide disaster Con: - Additional costs can run In the area of millions of dollars per year - Requires 3-5 years to Implement • Vendor Hot/Cold Site Pro: - Permit resumption of Deposit processing within 3 days, loan processing within 7 days and other processing within 2 weeks • Complete protection against local and partial protection against area-wide disaster Con: - Additional costs can run in the area of millions of dollars per year - Significant exposure In event of major area-wide disaster* (e.g. 8.0 on San Andreas near LA) *Sungard has 3 and Comdisco has 5 feasible sites

11. Findings: Data Processing Alternatives (cont’d) Data Processing Alternatives that meet BIA Requirements • Dual Data Center (California & Florida) • Capable of restoring critical deposit and loan processing within a shift • Capable of restoring other production processing within two weeks. Dual Data Centers with Electronic Archiving

12. Findings: Data Processing Alternatives (cont’d) Dual Data Center (continued) • Normally cost justified because of growing East Coast business base and requirement for complete protection from area-wide disaster* • Processing architecture typically involves: — West Coast production and development — East Coast production and cold site • A long-term alternative — 2-3 years to Implement — Additional millions in annual cost In redundant operations *An 8.0 earthquake on San Andreas near Los Angeles

13. Findings: Data Processing Alternatives (cont’d) B. Vendor Provided Hot/Cold Site • Capable of restoring critical processing within 3-4 days after a localized disaster (assumes fully tested recovery plan, and access to backup data files and personnel) Hot Site Restoration Schedule- Most Likely

14. Findings: Data Processing Alternatives (cont’d) Vendor Hot/Cold Site (cont’d) • Capable of moving all production processing to the Cold Site from the Hot Site within 2 weeks after a localized disaster Cold Site ActivatIon Schedule

15. Vendor Hot/Cold Site (cont’d) • Complete protection for localized disaster (fire, bomb, sabotage, small/close earthquake*) • Partial protection from area -wide disaster - multiple vendor clients in northern LA basin - vendor sites (3 for Sunguard and 5 for Comdisco) are allocated on a first come/first served basis with rapid response, would probably get a 3090-200, but might not get the second 3090-200 needed to catch up loan system • A short-term and/or long-term alternative - 90-120 days to Implement - annual cost of $ * A 6.0 earthquake on Verdugo Fault

16. Findings: Data Processing Alternatives (cont’d) Alternatives That Do Not Meet BIA Requirements C. Vendor Provided Cold Site • Capable of resuming processing within 3 weeks • Small annual cost but no capability to test D. Identified/Unprepared Site • Capable of resuming processing within 6-8 weeks. • Little initial or annual costs.

17. Findings: Cost of Data Processing Alternatives Annual Outlays(preliminary) Owned Dual Vender Centers Hot/Cold Site $ 0 $ 0 $ 0 $ $ $ 0 0 $ $ $ Data Center Facility Data Center Operations Computer: 3090-200 3090-400 Extensions Remote Printing Operations Staff Testing * TOTAL

18. Recommendations: Data processing Utilization of Vendor Provided Hot/Cold Site is Recommended • Only feasible short term alternative • Start negotiations with Comdisco with intent to sign contract by April 1, 1988 • Start preparation of Disaster Recovery Plan and Initial Testing immediately to gain protection from disaster within 90-120 days • Level of Protection provided to operations outside area affected by disaster Localized Disaster Good Area Wide Disaster Fair (may have limited access to backup data and personnel)

19. Findings: Item Processing Alternatives Available Item Processing Backup Approaches: • Local Mutual Aid Arrangement with another Los Angeles area Financial Institution — Typical 12 hour delay in capture processing — Good protection from localized disasters • Dual Bank Capture Centers Backup Arrangement (Florida backs up Los Angeles and Los Angeles Backs up Florida) — Typical 24 hour delay In capture processing — Good protection from Area-Wide disaster

20. Findings: Item Processing Alternatives Local Mutual Aid Arrangement • Several financial institutions have available time on compatible systems • Utilize time during off-shift and weekends • Protection level: Localized Disaster Good Area-Wide Disaster Poor • Minimal annual costs for administration & testing

21. Findings: Item Processing Alternatives California & Florida Capture Centers Backup Each Other • Both centers have capacity to backup each other • Vendor Hot/Cold Site has 1 channel extender which will support 1 sorter • Fed would forward all items to alternate capture center causing a 24 hour delay • Protection Level: Localized Disaster Good Area-Wide Disaster Good • Cost Estimates: Full Service Level (2 channel extenders) $ initial cost plus $ per year Reduced Service Level (1 channel extender) $ per year

22. Recommendations: Item Processing Utilize both alternatives: • Mutual Aid • For a Local disaster- Mutual Aid provides full service levels with 12 hour delay • Dual Capture Centers with Reduced Service Levels - For an Area-Wide disaster- Dual Centers providing partial service levels with a 2448 hour delay - Annual Fee $

23. Findings: Data Communication Alternatives Available data Communication Alternatives Include: I. Line switching (passive links) from Los Angeles PacBell office to Cypress Recovery Center and from Northern California to San Ramon Center. Switch 56 connection of Florida to hot site. • Poor protection from Local Area Disaster, since LA PacBell office 12 miles from Burbank Data Center • Significant monthly cost for contingency capability • 90-120 days to Implement Il. Line switching (passive links) from AT&T LATA offices to San Ramon Recovery Center. Switch 56 connection of Florida to hot site • Major monthly cost for contingency capability • Individual lines subject to outage from damage to any TELCO facility utilized by the Line • 90-120 days to implement

24. Findings: Data Communication Alternatives(cont’d) • Line Switching to Cypress and Sam Ramon Recovery Centers • Switch Southern California lines to Cypress Backup Center from LA TELCO offices and Northern California Lines to the San Ramon Backup Center • Significant monthly cost for Contingency Capability • Protection of Operations Outside Area Affected by Disaster Localized Outage Good Area-Wide Outage Poor (since LA Pac Bell Office only 12 miles from Burbank Data Center) • Implementation Requirements 90-120 days

25. Findings: Data Communication Alternatives(cont’d) II. Line Switching to San Ramon Recovery center Only • Switch Northern and Southern California lines to San Ramon backup center • Major monthly cost for contingency capability • Protection of Operations Outside Area Affected by Disaster Localized Outage Good Area-Wide Outage Fair (may have limited access to backup data and to personnel) • Implementation Requirements 90-120 days

26. Findings: Data Communication Service Levels Available Data Communication Service Levels: A.Full Service • ability to activate all multidrop branch support lines through line switching within 24 hours • Ability to activate administrative point4o-point lines with critical application requirements through Dial Backup within 24 hours. • Ability to activate ATM lines and non-critical administrative lines through emergency ordering within approximately 10-15 days B. Partial Service • Ability to activate approximately 50% of branch support and critical administrative support lines within 24 hours • Ability to activate remaining lines through emergency ordering within approximately 10-15 days

27. FINDINGS: DATA COMMUNICATION ALTERNATIVES COSTS Estimated Annualized Costs (5 Year Write-off) A: Full B: Partial Service Service I. Line switching of NC to San Ramon, SC to Cypress, and Florida to Hot Site $ $ II. Line switching of NC 8 SC to San Ramon, and Florida to Hot Sit. $ $

28. RECOMMENDATIONS: DATA COMMUNICATIONS As a Short Term Solution: • Partial Service Level and line switching to both Cypress and San Ramon Is a cost/effectivetemporary strategy • On a permanent basis a revised network architecture can offer improved service levels and Improved level of protection from area-wide disasters. For A Long Term Solution: • Evaluate as a separate business case alternatives such as: - redesigned network using concentrators - reconfigured network to better distribute immediately supported locations

29. Summary of Recommendations Data Processing • Implement vendor provided Hot/Cold site with Initial test of processing capability during May, 1988 Item Processing • Implement Both Mutual Aid and Dual Data Capture Center approaches with initial test of alternate capture capability during May, 1988 Data Communications • Implement, over the short term, Partial Service Level and Local Disaster Protection approach with initial test during July, 1988

30. Summary of Recommendations

31. Impact of Recommendations Protection Level of Recommended Strategies Localized Area-Wide DisasterDisaster Data Processing Good Fair Item Processing Good Good Data Communications Good Fair Overall Good Fair Service Level of Recommended Strategies to Operations Outside Area of Disaster Localized Area-Wide Disaster Disaster Data Processing Good Fair Item Processing Good Fair Data Communications Fair* Fair Overall Fair* Fair * Long Term Network redesign of configuration can raise this level

32. Impact of Recommendations Estimated Annualized Costs(5 year write oft) Testing $ /year Data Processing $ /year Item Processing $ /year Data Communications $ /year Total $ 500,000/year