IDS 11.5 MACH Your Business Continuity

1. IDS 11.5 MACH Your Business Continuity Boycho Velkov NDB Ltd.

2. Content Summary • AVAILABILITY CONCEPTS • IDS AVAILABILITY OFFERINGS OVERVIEW • CONFIGURING IDS AVAILABILITY COMPONENTS • BUSINESS SCENARIOS BY INDUSTRY • RETAIL • TELCO • INTERACTIVE ENTERTAINMENT • SUMMARY • RESOURCES

3. Content Summary • AVAILABILITY CONCEPTS • IDS AVAILABILITY OFFERINGS OVERVIEW • CONFIGURING IDS AVAILABILITY COMPONENTS • BUSINESS SCENARIOS BY INDUSTRY • RETAIL • TELCO • INTERACTIVE ENTERTAINMENT • SUMMARY • RESOURCES

4. TERMINOLOGY • High Availability – The proportion of time that an application can be used for productive work, measured against the time that it must be functional • Continuous Availability – Online capabilities that make the system available all the time (near zero downtime) • Continuous Availability maybe achieved by using High Availability capabilities • Global Availability – Online capabilities that allow your businesses to be available across geographic boundaries • Failover – Transfer of processing or services to an identical system for operational continuity • HA is generally achieved using Failover technology • Cluster – A group of tightly coupled systems working together in a way that appears as one system to users

5. THINGS TO CONSIDER • Define and understand requirements • Uptime needs (planned and unplanned outages) • Tolerance for data and/or transaction loss • Automatic vs manual failover • Geographic distribution and related network availability • Database Design – single vs multi-partition database • Nature and size of transactions

6. THINGS TO CONSIDER (Cont’d) • Understand system capabilities • Understand “what and “where” of system bottlenecks • Network • Hardware • Application • Database • Understand cost of achieving High Availability for your business

7. Availability - Defined Availability • The time that the application must be functional or available to users is called "mission time," which may be quite different than 7×24 or 24×365 operation • The proportion of time that an application can be used for productive work, measured against the time that it must be functional Factors Determining Availability • The reliability of the components that comprise the application: namely, how often they fail • How long it takes for the application to be restored once a failure has occurred Source:“High Availability: A Perspective,” Jane Wright, Ann Katan (Gartner Group), 11/24/2004

8. Availability - Business Impact Availability • Loss of data access can stop business processing and operations • Loss of Customer Confidence • Loss of Employee Productivity • Loss of Company or Share Value • Loss of Market Share and Revenue • Penalties, Fines and Regulatory Fee At 99% Uptime, a financial market would lose around $540 million per year

9. Content Summary • AVAILABILITY CONCEPTS • IDS AVAILABILITY OFFERINGS OVERVIEW • CONFIGURING IDS AVAILABILITY COMPONENTS • BUSINESS SCENARIOS BY INDUSTRY • RETAIL • TELCO • INTERACTIVE ENTERTAINMENT • SUMMARY • RESOURCES

10. HIGH AVAILABILITY DATA REPLICATION (HDR) • Two node configuration for failover – one primary and one secondary • Fully active-active configuration with updatable secondaries • Supports asynchronous and synchronous replication • Supports manual or automatic failover • Supports automatic client re-connect • Simple to setup and administer • Can be migrated to RSS (discussed shortly)

11. IDS HDR X LOG TRANSFER PRIMARY HDR SECONDARY

12. SHARED DISK SECONDARY (SDS) • Multi-node configuration – one primary and any number of additional secondary servers • Nodes share disk subsystem, with Primary as master of disk • Ideal in a cluster environment • Very fast failover, any secondary can be promoted to primary when needed • Additional nodes can be used for load balancing queries and transaction updates • Can be configured with other IDS availability components

13. Primary with Multiple SDS nodes on a blade server Primary SDS #1 Shared Disk SDS #2 SDS #3 Blade Server Shared Disk Mirror Shared Disk Secondary Primary SDS 1 SDS 2

14. Remote Standalone Secondary (RSS) • Similar to HDR: • Maintains a full disk copy of the database. • Created by performing a backup/restore of the instance • Can be used for: Additional Backup, Load Balancing. • Distinct from HDR: • Uses full duplex communication (SMX)– better throughput over slower lines. • Does not support SYNC mode, not even for checkpoints. • Can not currently be ‘promoted’ to primary – but can be promoted to HDR secondary (Focus is on Disaster Recovery, not HA). • There can be any number of RSS instances • Requires Index Page Logging be turned on. • RSS can be used in combination with HDR secondary: • RSS can be converted into HDR secondary • HDR secondary can be converted into RSS.

15. Enterprise Data Replication (ER) • Used for Workload partitioning, capacity relief and data distribution • Can be used to replicate subset of data (rows, columns) • Supports update anywhere with very low latency • Integrated to be compatible with all other IDS availability solutions with the option of secure data communications • Applies Parallelism in update of target tables reducing response latency back to the source • Built-in conflict resolution • Works in a variety of topologies for varying business needs Cont’d …

16. IDS Enterprise Replication Replication models • Update anywhere (workload sharing) • Primary-target (consolidate or distribute data Routing topologies • Fully Connected • Hierarchical Tree • Forest of Trees

17. Primary Continuous Log Restore (CLR) • Provides value as part of Disaster Recovery solution • Provides a secondary instance with ‘log file granularity’ • Asynchronous update of remote site through log files replay • Does not impact the primary server • Can co-exist with “the cluster” (HDR/RSS/SDS) as well as ER • Useful in unreliable, intermittent or non-existent network environments Cont’d …

18. IDS Availability Options – Solutions that fits your need Continuous Availability ER ScaleOut HA Availability • High Availability (HA) HDR HDR+RSS • Scale Out HDR+SDS HA with Scale Out HDR+SDS+RSS (The IDS Cluster) • Enterprise Data Replication (ER) IDS ER • Not one size fits all

19. Continuous Availability IDS Global Availability ER Scale Out HA HDR HDR + SDS ER or HDR + RSS Availability IDS Availability Options – Solution that fit your need • High Availability (HA) HDR HDR+RSS • Scale Out HDR+SDS HA with Scale Out HDR+SDS+RSS (The IDS Cluster) • Enterprise Data Replication (ER) IDS ER • Not one size fits all

20. Continuous Availability IDS Global Availability Scale Out HA IDS Cluster Data Replication HDR HDR + SDS HDR + SDS + RSS ER or HDR + RSS Availability IDS Availability Options – Solution that fits your need • High Availability (HA) HDR HDR+RSS • Scale Out HDR+SDS • HA with Scale Out HDR+SDS+RSS (The IDS Cluster) • Enterprise Data Replication (ER) IDS ER • Not one size fits all Any combination of HDR along with SDS, RSS and ER or just ER alone can be used to meet your requirements

21. IDS Availability Options – Solution that fits your need Do you need to Protect yourself from Node failure? Use HDR Yes No Yes Do you need to Multilevel site failure protection? Yes Do you need to protect yourself from Site failure? Use RSS Use SDS No END Do you need geographically disperse processing? Yes Use ER No

22. Content Summary • AVAILABILITY CONCEPTS • IDS AVAILABILITY OFFERINGS OVERVIEW • CONFIGURING IDS AVAILABILITY COMPONENTS • BUSINESS SCENARIOS BY INDUSTRY • RETAIL • TELCO • INTERACTIVE ENTERTAINMENT • SUMMARY • RESOURCES

23. Integrated Online Reservation System (IORS) • Consider an online reservation system that will provide integrated airline, hotel and car reservations with the following model: • Will maintain 5-7 years of customer data online for highly customized service • Expected data size around tens of terabytes • Will have several lookup applications to provide an integrated view to customer: • Customer preferences • Airline, car and hotels • Travel advisories • Corporate discounts • Employee verification • Previous travel/reservation listings

24. System Infrastructure and Deployment • Operations will be managed out of service/support and data centers: • Continued use of agent center for customer service and support • IDS and web servers at data center for website operations • To handle peak workloads, IORS will have: • Load balanced web servers • Multiple IDS instances running on blade servers configured in SDS mode • Shared disk subsystem used by IDS instances

25. Application Architecture • Partner Application Interface – airlines, hotel, car rental • Reservation module • Verification module • Customer Preferences module • Corporate Discounts module

26. Database Architecture • One Primary IDS instance and 3 SDS instances • All updates will be directed to the primary • SDS nodes will be used to load-balance queries • Data resides on shared disk: • Reservations Database • Customer Database • History Database

27. Configuring SDS • Set SDS_TIMEOUT on Primary • Make Primary active in SDS environment • onmode –d set SDS primary server_name • On secondary set SDS_ENABLE, SDS_PAGING, SDS_TEMPDBS • Start secondary using oninit • Monitor SDS stats using onstat –g sds

28. IORS: Application Logic and System Design Achieving Availability and Scalability with IDS SDS Airline Systems Partner App Interface Hotel Systems Car Rental Systems Updates to Primary Updates to New Primary Reservation App Verification Process App Reads configured On 3 SDS nodes Primary X Customer Preferences App SDS_1 Corporate Discounts Process SDS_2 Customer History App SDS_3 Shared Disk Subsystem Blade Server hosting IDS SDS instances

29. Application Availability with SDS • If the primary fails: • promote an SDS node to primary • onmode –d make primary sds_servername • restart old primary later as new SDS node • Primary can also be brought down for maintenance • issue onmode –d make primary to promote SDS to primary • this will also shutdown the primary

30. Availability: Incremental improvements • As application continues to service more users, stakeholders look to improved availability: • The shared disk itself is not protected from failure • The blade server itself can be a single point of failure • Two approaches: • Add a disk mirror and additional blade server with SDS nodes • Add HDR secondary • Useful if secondary needs to be hosted at a different facility within same vicinity to account for site failover (with n/w ping rates under 50 ms)

31. Redundant SDS for Additional Failover Redundant SDS nodes SDS 4 Primary SDS 5 SDS 1 SDS 6 SDS 2 SDS 7 SDS 3 Disk Mirror

32. Site Failover Protection with HDR HDR Secondary Primary SDS 1 SDS 2 SDS 3 SERVICE CENTER BLDG 2 DATA CENTER BLDG 1

33. Additional site failover with HDR – with existing SDS nodes capable of handling growth requirements, site failure protections is critical IORS team decides on Secondary can also be used for scale as needed The HDR secondary will be housed at the support center building

34. Secondary IORS : Improved Availability with IDS HDR Improving Business Continuity with IDS SDS and HDR Airline Systems Partner App Interface Hotel Systems Car Rental Systems Updates to Primary Reservation App HDR Verification Process App Reads configured On 3 SDS nodes X Primary Customer Preferences App Primary SDS_1 SDS_2 Corporate Discount Process SDS_3 Shared Disk Subsystem Blade Server hosting IDS SDS instances

35. Secondary IORS : Improved Availability with IDS HDR Improving Business Continuity with IDS SDS and HDR Airline Systems Partner App Interface Hotel Systems Car Rental Systems Reservation App Updates re-routed to new Primary Verification Process App Primary Queries re-routed to new Primary Customer Preferences App Primary SDS_1 SDS_2 Corporate Discount Process SDS_3 Shared Disk Subsystem Blade Server hosting IDS SDS instances

36. HANDLING SUCCESS MANAGING IORS GROWTH

37. IORS SYSTEM - REVIEW • Application interface to third party systems • Application manages load balancing with writes on primary and reads on SDS nodes • Additional HDR node for site failover With this existing configuration, the IORS team were successfully able to manage scale and availability per business requirements

38. NEW BUSINESS AND NEW REQUIREMENTS • IORS stakeholders have signed up three new airline and three new hotel partners essentially doubling current accounts • Two additional service centers will be opened one additional in the United States and another in India • As part of improving customer service and overall system, data will be collected on customer visits to the site

39. REVISITING SYSTEM REQUIREMENTS The IORS team has made the following recommendation of the infrastructure to handle new requirements: • Additional IDS SDS cluster and a disk mirror • Deployment of IDS Connection Manager to dynamically load balance and arbitrate failover • Additionally configured IDS ER node at each customer service site • Deployment of the Open Admin Tool to remotely monitor the cluster The IORS team was also able to present the additional SDS cluster as incremental improvement to availability as it provides node failover for current blade server

40. IDS CONNECTION MANAGER The benefits of IDS Connection Manager: • Dynamically routes connection requests to most appropriate server in HA cluster • Connection Manager Arbitrator enables automatic failover logic for cluster • Works by using a well-defined Service Level Agreement (SLA) that allows different services levels such as OLTP (update intensive) or Reports (read only)

41. IDS Connection Manager Clients Connection Redirection Reports db_server OLTP OTHER db_sds_rw db_sds_ro

42. CHANGE IS GOOD! Since IORS modules have implemented some custom load-balancing, using the Connection Manager to its best potential requires some change to IORS modules. However, this change is for the better: • Higher level of separation of business logic • modules no longer have to know or manage the server connectivity of individual servers • lower application maintenance • Dynamic management of load-balancing including read/write scaling and failover

43. IDS Connection Manager (CM) • Connection Manager is installed with IDS CSDK • CM can run on any machine independent of the IDS server instances • Defined as named “SLAs” in a configuration file or through command line • Clients connect to SLAs in CM, so CM SLA itself must have entry in sqlhosts

44. IDS CM – configuring for load balancing Using the default configuration file $INFORMIXDIR/etc/cmsm.cfg : Tells CM to try sds_2_ro first, then sds_4_ro next and finally sds_3_rw if 2 and 4 are unavailable Parenthesis tells CM to treat ids_primary and sds_5_rw with equal priority but use one with the least load

45. IDS CM – SQLHOSTS for CM The CM entries must be entered into sqlhosts: CM requires one server definition to go about it’s business

46. IDS CM – SQLHOSTS for IDS Server

47. IDS CM – SQLHOSTS for Client Client only needs entries for Connection Manager SLA definitions: Applications now connect to their databases via SLA: ‘reserve_db@oltp’ or ‘service_db@oltp’

48. Secondary IORS – The New Look Airline Systems Partner App Interface Hotel Systems Car Rental Systems Connection Manager Reservation App HDR Mirror Verification Process App Customer Preferences App Corporate Discount Process Primary SDS_4 SDS_1 SDS_5 SDS_2 = Reads SDS_6 Shared Disk = writes SDS_3 SDS_7

49. Monitoring the IDS Connection Manager With the default configuration file, CM can be instantiated from the command line : $prompt$ oncmsm Monitoring the CM can be done using: $prompt$ onstat –g cmsm IBM Informix Dynamic Server Version 11.50.UC1 -- OnLine -- Up 1 days 00:29:38 -- 166296 Kbytes CM name host sla define foc flag connections ids_db iors.dc reports sds_2_ro+sds_4_ro+sds_3_rw 3 30 ids_db iors.dcoltp ids_primary+sds_5_rw 3 130 Ok, but what is foc ?

50. IDS CM Arbitrator – configuring for failover (contd.) • FOC – Fail Over Configuration parameter • Another SLA in CM SQLHOSTS for failover model • Defaults to SDS+HDR+RSS, 0 • Can be disabled using keyword DISABLE Timeout in seconds DISABLE