Live CS1000 Failover Exercise

1. Live CS1000 Failover Exercise April 12, 2012 Jessica Mosher School Employees Credit Union of Washington

2. Topics Covered • Disaster Modeled in the Exercise • Planning Tasks • Set the Date & Obtain Executive Sign-off • Develop Exercise Timeline & Materials • Post-exercise • Some Discoveries & Action Items

3. Disaster Modeled in The Exercise Normal Operating Conditions at School Employees Credit Union of Washington

4. Loss of Phone Services Only at the Seattle Office Disaster Modeled in The Exercise

5. Disaster Modeled in the Exercise Failover Operating Conditions at School Employees Credit Union of Washington

6. Planning Tasks • Start with a To-Do List that will be dynamically updated • Define the Benefits of the Exercise & Share with Call Center Managers Staff (trainers) will know exactly what incoming calls will be like for agents We’ll learn what hotkeys, shortcuts & sub-queues will no longer work Can our system at the failover site handle the volume? How will staff change call handling without a queue display? How will member calls change without automated telephone banking? We will see if BTN re-routing to the failover site actually works • Schedule a meeting with technical providers to determine their requirements In our case: Arrow S3 Nortel services provider XO Communications telecommunications provider • Set Basic Expectations for the exercise Redirect our main 206-, 509-, and toll-free numbers to our failover system in Spokane Effect open/close hours on the Spokane PBX (Normally handled in Contact Center) Exercise our call overflow/closed automated message in the failover system Recover to normal operations prior to the start of the next business day

7. Planning Tasks • Meeting with Technical Players Agenda (shared with invitees in advance): • Introductions • Outlined disaster scenario & expectations • Explain School Employees Credit Union’s Disaster Policy • Time to implement? • Security requirements – individuals authorized to declare disaster and verify with Arrow S3 and XO Communications • Recovery (fail back to normal operations) procedure • How will we know main circuits are ready to receive calls? • What information needs to be retained to restore Spokane’s original configuration? • What coordination needs to occur between XO and Arrow S3? • Next steps • We held a few conference calls over the next few months, to plan the BTN failover and recovery with XO & define coordination points & condition tests

8. Set the Date & Obtain Executive Sign-off • November 12, 2010 – a Friday • Our phones would have an unannounced “extension” of customer service hours from 6:00 – 6:30pm, 30 minutes after our published 5:30pm closing time • Participants: • School Employees Credit Union IT Resources • XO Resources & Arrow S3 Resources (one at each location) • Member Services Manager • Phone staffing requirements: • One agent in each office and a telecommuter. At least one on [our back office queue]. • At least one Supervisor • Must be able to handle “cold” calls • Must have a cellphone to use during the exercise

9. Develop Exercise Timeline & Materials • Questions asked frequently while planning: • Then what happens? • How long will that take? • Is this a checkpoint? • Information to Include in Planning: • “Need to Know” list, Tester Contact Info, Tests • Customer Service Rep (CSR) expectations & scripts

10. Develop Exercise Timeline & Materials Our Exercise Timeline:

11. Develop Exercise Timeline & Materials

12. Develop Exercise Timeline & Materials

13. Post-Exercise • Supervisors & Volunteers provided with checklist reference materials – added in Business Continuity plan materials in their final form • IT team members phoned in test calls during failover window to ensure call routing was exercised • CSRs kept a call activity & “hurdle” log • Four Real Member calls (one successfully routed to back office queue) • 1 Wrong Number • Wrote down member callback information for next business day in the absence of voicemail

14. Post Exercise • New lists: For Follow-up, Staff Feedback • Final Report titled “Lessons Learned” and only listed points for these questions: • What did we do well? • What went as expected? • What did not occur as expected? • What would we do differently? • References: • Materials Created for Future use:

15. Some Discoveries & Action Items • Follow-up items:

16. Some Discoveries & Action Items From the final report (Business & Member perspective): • What did not occur as expected? • The Lynnwood tester’s phone was not built correctly in the Spokane PBX & didn’t come up • Calls routed to any available agent or Supervisor—it was expected that only [ main queue ] phones would receive incoming calls • Phone system behavior when a queue was unavailable was different than expected • Calls did not “ring” on Agent phones—they connected with just a buzz • Calls could not be transferred or conferenced to unstaffed queues • Agents were not trained on how to reboot phones without disconnecting the powered Ethernet cord

17. Final Thoughts • Start small – we proved our failover system would take live calls with a minimum of risk to our equipment • Plan early, especially if there is customer exposure • Prepare phone staff with polished, well-written materials & a meeting – if IT/Telecom seems prepared, they will feel prepared – including possible problems they may experience • Mitigate live monitoring system alarms • Test backup components that are normally “dormant” to the fullest extent possible • Detail the recovery to normal operations as much as the failover • Build in documentation time into the project & repurpose into overall Business Continuity Plan Questions?