Municipal League Joint Venture Disaster Recovery Project

1. Municipal League Joint Venture Disaster Recovery Project Lee M. Mandell, Ph.D. Director of Information Technology and Research/CIO North Carolina League of Municipalities

2. What is Disaster Recovery? Disaster Recovery is the ability to recover from a loss of the IT facilities (or access to those facilities) and infrastructure used to conduct daily business.

3. Why Would You Need It? • The loss of the use of IT facilities can occur for many possible reasons: • hurricanes, floods, fires, tornados, chemical spills,earthquakes, power grid failures, HVAC failures, ice and snow storms, pandemics, terrorism, computer viruses, structural failures, vandalism, and the list can probably go on

4. DR Is Not BC A disaster recovery solution should be part of a comprehensive business continuity plan Having a hardware/software/ communications alternative to a data center is not enough This presentation is only about one piece of the DR/BC puzzle and needs to be put into the context of the broader problem and solution

5. The DR Center Project The NC League of Municipalities, in partnership with the Municipal Association of SC, and its technical partner VC3, has developed a jointly-owned Disaster Recovery facility for Wintel and compatible systems Our thanks to HP for technical and financial assistance on this project

6. Project Overview Although initially serving only two municipal organizations, the project determined that a methodology and solution existed that would allow multiple Leagues to share the initial cost and overhead of a scalable, centralized solution and allow recovery of business functions in the event of a disaster.

7. Project Overview The DR solution developed for this joint project leverages: • technology changes that have occurred and matured in the past few years (server virtualization and imaging software) • the public Internet, and • sweat equity to dramatically reduce the cost and increase the operational effectiveness of a DR facility

8. Deploy multiple virtual machines on a single physical server Virtual Machines Unmodified Application Unmodified OS ESX Server Virtual Hardware Physical Server Server Virtualization Basics Benefits • Increase hardware utilization by sharing hardware resources across a large number of virtual machines. Each Virtual Machine is a complete system encapsulated in a set of software files

9. Physical to Virtual • Reduces cost, improves time to recovery, and increases flexibility • Virtual to Virtual • Greatest flexibility, lowest cost, best time to recovery Options for Disaster Recovery Physical to Physical • Today’s scenario without Virtual Infrastructure • Requires identical DR site

10. Activities to Date Two sub-committees formed: • Technology Committee • Business Plan Committee Technology Committee: • Documented initial technical requirements • Designed a technical framework • Completed the Proof of Concept Phase • Completed the Pilot Phase • Entered the Production Phase

11. Technology Committee Assumptions Support for Windows NT 4.0, 2000 & 2003 Servers DR Site is a “very warm” standby; NOT a live HOT site Data synchronized on a daily basis (in some cases more frequently) Images transferred and access to restored servers over public Internet DR Site initially sized to support one organization at a time Full recovery achieved within 24 hours remotely; targeted goal and expectation is less than 12 hours DR site must accommodate firewall needs DR site must accommodate backups after production services have been transferred Two tests per year per organization at the DR site

12. Phase Objectives Proof of Concept Phase • Validate the technology • Verify data transfer and conversion timings • Refine budget and processes Pilot Phase • Utilize live production systems • Build a functional environment with a sub-set of production servers from each League • Validate sustained functionality and reliability Production Phase • Implement final solution with production services from all Leagues

13. The History of the Solution August 2005: MASC and NCLM decide to form a joint venture to solve their own DR needs September 2005: High level solution architected November 2005: Proof of concept phase successfully completed January 2006: Initial pilot phase successfully completed April 2006: Hardware and software refined and orders placed May 2006: Data Center space identified in Charlotte, NC June 2006: Data Center space secured July 2006: MASC went live with basic DR Solution; recovery test completed successfully

14. Key Technology Components Hardware • Three HP dual processor servers (shared) • 2.6 Ghz • 16 GB • 1.8 TB • VPN concentrator (shared) • Firewall appliance (shared) • Gigabit switch (shared) • Console KVM Switch (shared) • Image Server (low cost/high capacitydata storage) Software • VMWare – Server Virtualization Suite • Symantec LiveState – Imaging Software

15. DR Server Replication

16. DR Process Nightly transfers/replication of incremental data and system configuration images to off-site facility: Secure FTP with 3DES encryption Image production and transfer process automated Images validated after creation and transfer Monthly Disk Transports of full images (base size 300 GB): 128 bit AES encryption with encryption at the data and file system level Remote connection to restored servers at DR Center through Citrix

17. DR Center Metrics Creating LiveState images of 11 MASC core virtualized servers: • Initial Image: 2 to 2.5 hours  300 GB • Daily Incremental Image: 2 to 2.5 hours  3 to 4 GB Transferring incremental images nightly to DR Site: Incremental Image : 2 Mb/sec (10 Mbs burst)  3.5 to 4 hours Remote recovery of all 11 systems from incremental images: 6 to 8 hours (parallel recovery)

18. Traditional Vendor-Provided DR Services Costs:$3,000 to $4,000 per month Costs per League for five-year planning horizon:$180,000 to $240,000 DR Center Costs:initial shared setup costs at $40,800; hosting charges of $1,150 per month; pre-funded replacement charges of $680 per month; local setup costs at $20,400 to $51,900 Costs per League (2) for five-year planning horizon:$95,700 to $127,200 DR Cost Comparisons

19. Key Benefits of Basic DR Center Solution Captures all data and all system state information Allows remote restoration of all servers Uses public Internet for both image transfer and access to restored systems Scales more cost effectively as data grows as compared to other solutions Shares costs and resources for little-used but crucial capability Easy to plan and budget the continuing costs Demonstrates that a partnership among like organizations to support joint DR solution can work (formal working/ownership agreement developed and signed)

20. NCLM Enhanced DR Solution Reduce the cost and complexity and increase the flexibility and robustness of business continuity by storing entire database and system files on a storage area network (SAN) that can be replicated and restored at DR Center Provide near real-time, byte-level replication and point-in-time rollback recovery Provide additional level of data recovery beyond nightly backups at the file/email message level Direct broadband connection to DR Center Locate server room appliances at DR Center Would add around $92,000 to five-year costs, net

21. Disaster Recovery Using SAN/iQ Remote Copy Perform disaster recovery using Remote Copy • No need to upgrade secondary site server hardware in lock-step with the primary site • Easy to automate and no need for bare metal recovery tools • Virtual image and data all protected under one DR plan Primary Site Recovery Site

22. The Rest of the Future Expand basic or enhanced version to other southern municipal leagues The basic DR solution also has the promise of being able to provide shared DR services to cities and towns in NC and SC that have historically been cost prohibitive for them to purchase, allowing the recovery of their critical business functions in the event of a disaster

23. Thank You!