FUTURES INDUSTRY BUSINESS CONTINUITY AND DISASTER RECOVERY2007 INDUSTRY TEST RESULTS“DR IV”November 5, 2007Compiled and Developed byTellefsen and Company, L.L.C.
TABLE OF CONTENTS • Background 3 • Executive Summary 8 • Overall Test Results 12 • Problems Encountered 27 • Lessons Learned 29 • Suggested Next Steps 33 Page Number:
I. BACKGROUND • The FIA Information Technology Division conducted the first annual Disaster Recovery (DR) Test in October 2004. • Starting in Q2, 2007, the FIA Business Continuity Committee began detailed preparations for this year’s third annual industry-wide test. • A Working Group was convened to discuss and agree on a date, goals, objectives, etc.
BACKGROUND (Cont’d) … • Two futures Industry BC/DR symposiums were held in September in New York and Chicago. Panelists included representatives from the major exchanges and clearinghouses: • CBOE Futures Exchange • Chicago Board of Trade • Chicago Mercantile Exchange/CME Clearing • The Clearing Corp. • Eurex AG • ICE Futures EU • ICE Futures US/ICE Clear US • Liffe • Kansas City Board of Trade/KCBT Clearing Corp. • MEFF • New York Mercantile Exchange/NYMEX Clearing • Philadelphia Board of Trade • Winnipeg Commodity Exchange
BACKGROUND (CONT’D) … • The scope of this year’s initiative was designed to test disaster recovery back-up connectivity and functionality between exchanges, clearinghouses and member firms: • Test firm back-up to exchange back-up sites (DR-DR). • Verify connectivity. • Test round-trip communications capabilities. • The symposium sessions were well attended by numerous operations managers and BC/DR representatives from various clearing and non-clearing firms. • Regular Committee conference calls were held between May and October (bi-weekly and weekly).
BACKGROUND (Cont’d) … • The Committee including representatives from: • Clearinghouses • Exchanges • FCMs • Clearing firms • Non-clearing firms • Key service providers • Independent software vendors
BACKGROUND (Cont’d) … • To encourage participation, the FIA agreed that all firm names would be anonymous and test results would be without attribution to any firm. • Post-test interviews with firms and survey forms were used to fine tune the results.
II. EXECUTIVE SUMMARY • The fourth annual industry-wide disaster recovery test in the U.S. financial services sector was highly successful, largely in part to the good working relationship between exchanges and the firms. • Major U.S. and international futures exchanges, clearinghouses, FCMs and clearing firms participated in this year’s test: • 103 FCMs, clearing firms and non-clearing firms participated - an increase over last year • 98% of firms tested successfully • Participant firms represent a significant critical mass of futures order flow and liquidity: • ~88% of overall futures exchanges’ volume.
EXECUTIVE SUMMARY (CONT’D) … • This year’s test initiative was expanded to include several domestic and international exchanges: • ICE Futures EU • MEFF • Philadelphia Board of Trade • Exchanges and clearinghouses demonstrated that their systems, processes and procedures simultaneously worked very well.
EXECUTIVE SUMMARY (CONT’D) … • Feedback from the firms, exchanges and clearing houses indicated that this was a valuable and worthwhile exercise and that the industry should continue to do more common testing in the future. • Firms indicated that the test helped them: • Exercise their Business Continuity/Disaster Recovery plans • Identify internal single points of failure • Tighten up and document their business continuity procedures • Better understand the need for cross-training • Test connectivity with exchanges’ DR sites
TEST PARTICIPANTS REPRESENT A SIGNFICANT % OF EXCHANGES’ VOLUME Source: Exchanges and Clearing Houses
III. OVERALL TEST RESULTS • The major domestic and international futures exchanges and clearinghouses and 103 firms* participated in the test: • CBOE Futures 6 • CME Group 54 • Clearing Corp. 23 • Eurex 66 • ICE 87 • ICE Futures US/ICE Clear US 29 • Kansas City Board of Trade 17 • Liffe 19 • MEFF 2 • NYMEX/COMEX 40 • Winnipeg Commodity Exchange 14 * Some firms are common members of multiple exchanges and clearing houses. PBOT testing did not involve members.
OVERALL TEST RESULTS (CONT’D)… • Test participants included both clearing firms and non-clearing firms. • Approximately 85%of all futures clearing firms participated. • 98% of firms tested successfully. • 92% tested from their Disaster Recovery sites and/or via back up communications facilities. • The remainder tested from their primary sites.
CBOE FUTURES EXCHANGE • Tested via member firms’ back up connectivity to production CBOEdirect trading platform. • Scripted trade entry for VIX futures contracts. • Transmitted trades to the Clearing Corp’s. back- up systems.
CHICAGO BOARD OF TRADE • Members tested connectivity to eCBOT trading system. • Participated in post-trade testing and generated a subset of October 26th trades to member firms from CME Clearing. • Injected trades in trade handling system via back up site. • Transmitted trade confirmations from CME Clearing. • Accepted EFP, change records, PCS and large trader records.
CHICAGO MERCANTILE EXCHANGE • Tested member firms back-up connectivity to production GLOBEX trading platform via CME remote back-up site. • Entered a script of orders for various products. • Received ex-pit, block trade information via CME remote site portal URL. • Generated trade reporting and SPAN files from 10/26 trade date. • Received PCS and large trader information from member firms. • Transmitted trade registry data and SPAN files via FTP.
THE CLEARING CORP. • Tested member firms’ back-up connectivity to back-up clearing system. • Confirmed system connectivity via successful system login message – no scripted trade entry. • Transmitted previous business day clearing reports to member firms: • Match off files • Report files • SPAN files • PCS, ex-pit and large trader information was not accepted.
EUREX AG • Tested member firms’ back up connectivity to production XCEED electronic trading platform. • Connectivity was confirmed via successful system log in message – no scripted order entry.
ICE FUTURES US / ICE CLEAR US • Tested member firms’ back-up connectivity to the ICE Trader electronic trading system. • Scripted order entry for Cotton and Sugar futures contracts. • Trade entry via TIPS system for test trades, ex-pit, large trader and PCS information. • ICE Clear produced end-of-day files: • Match off files, report files and SPAN files.
ICE FUTURES EU • Tested member firms’ back-up connectivity to the ICE Exchange electronic trading system DR site. • Scripted order entry for Cotton and Sugar futures contracts. • Tested ICE Block and other non trading functionality from DR site. • ICE Web reporting via Internet portal for deal reporting, position reports, etc. • Futures trades submitted to the ICE Clear DR site.
KANSAS CITY BOARD OF TRADE • Members tested connectivity to eCBOT trading system. • Participated in post-trade testing. • Generated a subset of October 26th trades to member firms. • Transmitted trade confirmations from KCBT Clearing Corp. • Accepted EFP, change records, PCS and large trader records.
LIFFE • Tested member firms’ back up connectivity to production CONNECT trading platform. • Connectivity was confirmed via successful system log in message to central gateways– no scripted order entry.
MEFF • Tested via the MEFF production ETS environment. • Simulated a failure of their London POP and successfully failed over to the backup system during the test. • Members entered trades and received reports. • Transfer files were delivered from clearing, but member firms’ back office systems did not receive any test data.
NEW YORK MERCANTILE EXCHANGE • Tested member firms’ back-up connectivity to GLOBEX trading platform, ClearPort TMS and ClearPort Clearing systems. • Order entry via scripted trade entry in crude oil, natural gas, heating oil, gasoline, silver and gold futures contracts. • Reported trade fill information to member firms. • Transmitted end-of-day clearing reports via NYMEX Clearing: • Ex-pit, PCS, large trader information • 591 Member Trade Register report
PHILADELPHIA BOARD OF TRADE • PBOT conducted an ‘internal’ test • Used the PBOT/XL trading system • Trade Orders received at PBOT/XL Help Desk. • Fill reports were generated. • PBOT trade file processed for clearing and settlement.
WINNIPEG COMMODITY EXCHANGE • Members tested connectivity to eCBOT trading system. • Participated in post-trade testing. • Generated a subset of October 26th trades to member firms from Clearing and transmitted trade confirmations. • Accepted EFP, change records, PCS and large trader records.
THE MAJORITY OF PARTICIPANTS WERE CLEARING FIRMS Number of Firms Source: Exchanges and Clearing Houses
IV. PROBLEMS ENCOUNTERED • A number of problems were encountered, including: • Incorrect FIX tags impacted routing logic • Configuration parameter not set properly on back up servers • Configuration file issues with ISV systems • MQ connectivity/could not generate MQ messages • Incorrect IP addresses/incorrect network configuration • Firewall configuration parameters • Member firm hotline was not working initially • Exchange trading system not accepting logins • Exchange web portal connectivity problems • Remote login to clearing system did not work • Most were resolved quickly, although some caused an unexpected delay to test progress.
PROBLEMS ENCOUNTERED (CONT’D)… • Some firms experienced procedural problems with invalid or expired passwords, looking at the wrong system screens, etc. • One exchange had problems with firm’s access to their backup systems, issues with SFTP encryption certification. • Some firms did not verify that their back up servers and equipment was properly functioning prior to the test.
V. LESSONS LEARNED • The futures industry proved that it is capable of successfully orchestrating an industry-wide disaster recovery test. • Most problems that were encountered were rectified quickly, although some caused an unexpected delay to the restoration of full processing capabilities: • Under real life situations, most problems could probably be resolved within hours or by the next business day. • The exchanges and clearinghouses’ internal support processes and procedures worked well.
LESSONS LEARNED (CONT’D)… • Firms indicated that the test helped them: • Identify some internal single points of failure • Tighten up and document their business continuity and system fail over procedures • Better understand the need for cross-training • Test connectivity with exchanges’ DR sites • Pre-test connectivity testing was valuable and should be a pre-requisite for future tests.
LESSONS LEARNED (CONT’D)… • Firms must be aware of any environmental impact on their network fire walls caused by test requirements: • Have proper network staffing support actively engaged during the test. • Be aware of any firewall impact and make changes accordingly to accommodate testing. • There is a need to better coordinate testing activities: • More time should be allocated for this during future tests. • Need to identify and corroborate all test contacts prior to the tests. • Try to standardize scripts across exchanges.
LESSONS LEARNED (CONT’D)… • Need clear support desk procedures and sufficient staffing at exchanges to properly support the test. • A number of firms apparently did not pre-register and “showed up” for the test at several exchanges: • They were set up and most tested successfully, however, this placed an additional demand on exchange and clearing house support staffs. • Additionally, several firms that registered at 1-2 exchanges were “no shows”
VI. SUGGESTED NEXT STEPS • Survey firms and exchanges for input to future testing. • Announce the details of the 2008 DR test: • Define the scope of next year’s scenario test exercise • Establish the goals and objectives of test initiatives • Announce dates early in advance • Broadly “market” future test initiatives to firms’ IT, operations, trading and senior management, to ensure a wider dissemination of test information within all firms.