Business Continuity andthe Disaster Recovery Center The agency’s goal was to build a Disaster Recovery Center with sufficient capacity to support all OIC mission critical applications. The OIC IT Disaster Recovery site located in OB2 supports the OIC Disaster Recovery Center.
DR capability has been provided for the following systems in priority order : • Priority 1: By the next business day • Active Directory • Citrix (30 active connections) • SIMBA (including data manager, document repository and credit card web service) • CATS • Email/Web Email • Employee files and data • Internet access (NAIC, SERFF, HRISD, I-Site, HRMS)
DR capability by priority order continued: • Priority 2: Within 5 business days (or as servers are provisioned) • Image document retrieval (retrieval of existing images only) • SourceSafe/Build Machine • Teammate • Antivirus (WUS/SUS), Spam filter, PC Help • HCIS, online filing search, LAA, Annual Reports and Contribute • Citrix (100+ active connections) • Priority 3: Within 31 days (or as servers and facilities are provisioned) • Imaging scanning and indexing
Steps included in setting up the OIC Disaster Recovery Center • Server Justification and Consolidation • Server Virtualization • Clone Virtual Machines to DR Center • Test the Disaster Recovery Center • Set up DR Center update routines and procedures • Decommission Physical Servers
Benefits of Virtualization • Reduce hardware costs • Reduce maintenance costs • Reduce energy costs • Reduce the time it takes to provision new servers • Reduce the time it takes to restore a server during a disaster • Reduce server management time
The OIC uses a Cluster of Virtual Machine Hosts to provide high availability for the Virtual Machines and maximize host utilization
Clone Virtual Machines to DR Center • The Virtual Machines that are used for Production have been copied to the Disaster Recovery Center. • The rest of the Physical Servers that are used for Production have been converted to Virtual Machines and copied to the Disaster Recovery Center. • Updates to these servers are also copied to the Disaster Recovery Center.
Virtual Machine use at the OIC Disaster Recovery Center • Some of the Virtual Machines at the DR site are used for Application Development and User Approval Testing. • Other Virtual Machines are on all of the time so we can copy current data to the Disaster Recovery Center • SQL databases are mirrored or backed up to the DR site. • Exchange and the File servers are mirrored with HP Storage Works • The rest of the Virtual Machines will only be turned on again when they are needed.
Testing the Disaster Recovery Center • All of the DR systems were tested and re-tested as they were created. • Exchange failover, database failover, document recovery and image retrieval were all tested during several weekends. • During a planned upgrade of our core production application, SIMBA, the DR Center was used for read only access. • We continue to use the Virtual Machines to test planned upgrades to our production systems.
Results of the OIC Virtualization We have 31 physical servers. 23 at 5000 Capitol Boulevard, 5 at the DR/DEV site and 3 more in Seattle, SIU and the Capitol Campus. There are 100 virtual servers. 42 at 5000 Capitol Boulevard and 58 at the DR/DEV site. Total Number of Servers 131 Virtualized 100 Percent Virtualized 76% We also plan to virtualize 9 more of the physical servers at 5000 Capitol Boulevard in 2010. (We will need 1 more physical server to host the additional virtual servers) We will then have 23 physical servers. 15 at 5000 Capitol Boulevard, 5 at the DR/DEV site and 3 more in Seattle, SIU and the Capitol Campus. There will be around 109 virtual servers. 51 at 5000 Capitol Boulevard and 58 at the DR/DEV site. Total Number of Servers 132 Virtualized 109 Percent Virtualized 83%