1 / 46

In RAC We Trust

In RAC We Trust. O RAC LE - From Dream To Production. BGOUG – Gabrovo 22.04.2005. ,, Let someone k n o w ”. Plamen Zyumbyulev. Presentation Goals. Describe the major steps in RAC implementation Show and explain the main problems and obstacles Show and explain solution and workarounds

aqua
Download Presentation

In RAC We Trust

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. In RAC We Trust ORACLE - From Dream To Production BGOUG – Gabrovo 22.04.2005 ,, Let someone k n o w ” Plamen Zyumbyulev

  2. Presentation Goals • Describe the major steps in RAC implementation • Show and explain the main problems and obstacles • Show and explain solution and workarounds • Give some practical ideas and variations • Try to look at the problems from different angles Non-Goals • Explain RAC concepts and fundamentals • Show ALL aspects of RAC

  3. Agenda • Introduction • RAC Installation • High Availability Configuring • Testing and Tuning RAC • Implementing RAC in production

  4. Introduction

  5. RAC Installation Problems Solutions

  6. Problems • Need to evaluate and purchase cluster hardware • Need of knowledgeable OS, High Availability, Network and Storage professionals • All this takes time

  7. Solutions • RAC on Single Node • RAC on Single VMware Node • RAC on Multiple VMware Nodes • RAC and Network Block Device • Other Solutions All of the solutions presented here are for testing purposesONLY. These configurations are not certified or supported by Oracle Support Services

  8. RAC on Single Node • Why not? • Metalink Note:241114.1 - Step-By-Step Installation of RAC on Linux - Single Node (Oracle9i 9.2.0 with OCFS) • Key Points • No need of “fencing” configuration • No need of using clustered file system or raw devices • No need of multiple oracle homes (ORACLE_HOME)

  9. RAC on Single Node(cont’d) • Key Points (cont’d) • One oracle user with 2 or more profiles one for every instance. (e.g. .rac1, .rac2, …) zyumix:/# su - oracle oracle@zyumix:~$ . rac1oracle@zyumix:~$ echo $ORACLE_SID rac1 zyumix:/# su - oracle oracle@zyumix:~$ . rac2oracle@zyumix:~$ echo $ORACLE_SID rac2

  10. RAC on Single Node(cont’d) • Key Points (cont’d) • Oracle Universal Installer needs Clusterware software in order to install RAC option. • Disadvantages

  11. RAC on Single Node(cont’d) Client side load balancing Listener ‘A’ Listener ‘B’ Server side load balancing Server A Instance ‘A’ Server B Instance ‘B’ Database

  12. RAC on Single VMware Node • Even easier !!! • The Oracle-on-Linux VMware Cookbook -http://www.oracle.com/technology/tech/linux/vmware/cookbook/index.html An easy, hands-on, step-by-step guide describing how-to install VMware, Linux (RHEL/SLES) and Oracle RAC (again on single node) VMware Workstation (90-day free eval; registration required) RHEL3 (not free) , SLES8 (not free) • Disadvantages

  13. RAC on Multiple VMware Nodes • VMware GSX/ESX Server permits the sharing of plain disks with multiple virtual machines running on the same host, provided the disk in question is a SCSI disk. • This approach is very powerful but complex. You can create very complex environments – multiple NICs, switches, disks etc. • Now there are a number of nodes although virtual • Disadvantages

  14. RAC and Network Block Device • This solution allows you to build scalable and high available database system only with common Intel PCs connected into Ethernet network. • In this solution, a standard shared disk subsystem is replaced by a native Linux technology - Network Block Device (NBD) that maps remote files to local block devices (e.g. /dev/nb0) via TCP/IP network. One computer (not necessarily Linux machine) serves as data storage for all cluster nodes (Linux machines) instead of expensive disk array.

  15. RAC and Network Block Device (cont’d) • With this thing compiled into the kernel, Linux can use a remote server as one of its block devices. Every time the client computer wants to read /dev/nd0, it will send a request to the server via TCP, which will reply with the data requested. • The remote resource doesn't need to be a whole disk or evena partition. It can be a file.

  16. RAC and Network Block Device (cont’d) Typical configuration Simple NBD configuration

  17. RAC and Network Block Device(cont’d) • Installation Both client and server machines are with RHEL3 • Download source from http://nbd.sourceforge.net/ • As root do • bunzip2 nbd-2.7.3.tar.bz2 • tar -xvf nbd-2.7.3.tar • cd nbd-2.7.3 • ./configure • make • make install

  18. RAC and Network Block Device(cont’d) • Creating new empty files at NBD server [root@rac2 root]# dd if=/dev/zero of=/u01/oradata/rac/system.01 count=300 bs=1M 300+0 records in 300+0 records out 314572800 bytes transferred in 1.683993 seconds (186801738 bytes/sec) [root@rac2 root]# • Running NBD server Syntax: nbd-server <port> <filename> [root@rac2 root]# nbd-server 4101 /u01/oradata/rac/system.01 [root@rac2 root]#

  19. RAC and Network Block Device(cont’d) • NBD client NBD client must be run as root (because of kernel parts of NBD). Before starting NBD client you would have to install Linux kernel NBD module • Installing ndb module – RHEL3 [root@rac3 root]# rpm -Uvh kernel-unsupported-2.4.21-4.EL.i686.rpm warning: kernel-unsupported-2.4.21-4.EL.i686.rpm: V3 DSA signature: NOKEY, key ID db42a60e Preparing... ########################################### [100%] 1:kernel-unsupported ########################################### [100%] [root@rac3 root]#

  20. RAC and Network Block Device(cont’d) • NBD client (cont’d) • Loading nbd module [root@rac3 dev]# lsmod | grep nbd [root@rac3 dev]# modprobe nbd [root@rac3 dev]# lsmod | grep nbd nbd 16388 0 (unused) [root@rac3 dev]# • running nbd client Syntax: nbd-client <data server> <port> /dev/nb<n> [root@rac3 dev]# nbd-client rac2 4101 /dev/nb0 Negotiation: ..size = 307200KB bs=1024, sz=307200 [root@rac3 dev]#

  21. RAC and Network Block Device(cont’d) • Now block devices are configured and it is possible to access remote data. Oracle Real application clusters need raw access to shared disk subsystem so mapping raw devices to block devices is needed. This could by done with standard rawcommand. Syntax: raw /dev/raw/raw<N> /dev/<blockdev> [root@rac3 root]# raw /dev/raw/raw1 /dev/nb0 /dev/raw/raw1: bound to major 43, minor 0 [root@rac3 root]#

  22. Other Solutions • RAC and FireWire • Build Your Own Oracle RAC 10g Cluster on Linux and FireWire http://www.oracle.com/technology/pub/articles/hunter_rac10g.html • RAC and NFS • Locking • Caching • Write through cache

  23. HA Configuration

  24. Simplified RAC schema NET Database

  25. HA System

  26. Dedicated Network Maximum Availability Architecture Application Server Application Server Data Guard Primary Site Secondary Site

  27. Dedicated Network Extended Distance Clusters Application Server Application Server RAC Virtualization storage layer What about the Quorum Server??? Primary Site Secondary Site

  28. Extended Distance Clusters(cont’d) • Resolving Distance Problems • Application partitioning • gc_files_to_locks • Wavelength Division Multiplexing • Dense Wavelength Division Multiplexing DWDM • Coarse Wavelength Division Multiplexing CWDM • ACTIVE_INSTANCE_COUNT • *.active_instance_count = 1 • *.cluster_database_instances = 2

  29. Testing and Tuning RAC Introduction RAC testing steps Functional Application Tests RAC High Availability tests Scalability tests Digging into RAC performance problems

  30. Introduction • Testing isn’t trivial !!! • Classical testing/tuning methods. • Always tune single instance first! • Specific RAC issues • RAC aware tools

  31. RAC testing steps • Functional Application Tests • RAC High Availability tests • Be aware about the timeouts!!! • Scalability tests

  32. RAC testing steps (cont’d) • Scalability tests (cont’d) • Patterns of application scalability Performance (TPS, 1/response time) of one user nearly static linear constrained exponential Load ( # users, size of tables)

  33. RAC testing steps (cont’d) • Scalability tests (cont’d) • Good scalability Performance -TPS (for all users) 2 node RAC Single node # of concurrent users

  34. RAC testing steps (cont’d) • Scalability tests (cont’d) • Problem!!! Possible disk bottleneck Performance -TPS (for all users) 2 node RAC Single node # of concurrent users

  35. RAC testing steps (cont’d) • Scalability tests (cont’d) • Problem!!! Possible interconnect bottleneck Performance -TPS (for all users) Single node 2 node RAC # of concurrent users

  36. Digging into RAC performance problems • Interconnect and shared storage are the two most possible performance problem areas in RAC • Interconnect “speed” • Throughput • Latency • average latency of a consistent block request. AVG CR BLOCK RECEIVE TIME should typically be about 15 milliseconds depending -- on your system configuration and volume

  37. Digging into RAC performance problems(cont’d) • Interconnect types

  38. Digging into RAC performance problems(cont’d) • cluster_interconnectsparameter It provides Oracle with information about additional cluster interconnects available for use and can be used to load balance the interconnect traffic to different physical interconnects thus increasing interconnect bandwith. When you set CLUSTER_INTERCONNECTS in cluster configurations, the interconnect high availability features are not available. In other words, an interconnect failure that is normally unnoticeable would instead cause an Oracle cluster failure as Oracle still attempts to access the network interface which has gone down.

  39. Digging into RAC performance problems (cont’d) • STATSPACK reports • The STATSPACK report show statistics ONLY for the node or instance on which it was run • Top 5 Timed Events • Global Cache Service and Global Enqueue Service • Note:135714.1 Script to Collect RAC Diagnostic Information (racdiag.sql)

  40. Digging into RAC performance problems (cont’d) BAD PERFORMANCE Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time --------------------------------------- ------------ ----------- -------- global cache cr request 34,568 958 31.44 buffer busy global CR 6,513 620 20.35 db file sequential read 64,214 455 14.92 latch free 13,542 453 14.88 buffer busy waits 10,971 295 9.69 GOOD PERFORMANCE Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time --------------------------------------- ------------ ----------- -------- latch free 10,969 666 51.28 buffer busy waits 15,379 159 12.23 CPU time 149 11.47 PL/SQL lock timer 51 105 8.05 db file sequential read 25,163 96 7.36

  41. Digging into RAC performance problems(cont’d) Global Cache Service - Workload CharacteristicsBADGOOD ----------------------------------------------- Ave global cache get time (ms): 11.8 2.2 Ave global cache convert time (ms): 51.7 11.2 Ave build time for CR block (ms): 0.7 0.0 Ave flush time for CR block (ms): 0.2 0.2 Ave send time for CR block (ms): 0.0 0.2 Ave time to process CR block request (ms): 0.9 0.4 Ave receive time for CR block (ms): 1.6 0.4 Ave pin time for current block (ms): 0.2 0.2 Ave flush time for current block (ms): 0.0 0.0 Ave send time for current block (ms): 0.1 0.1 Ave time to process current block request (ms): 0.3 0.3 Ave receive time for current block (ms): 33.4 7.5 Global cache hit ratio: 9.5 3.9 Ratio of current block defers: 0.0 0.0 % of messages sent for buffer gets: 6.7 2.5 % of remote buffer gets: 1.8 0.7 Ratio of I/O for coherence: 1.2 1.3 Ratio of local vs remote work: 4.3 4.4 Ratio of fusion vs physical writes: 0.0 0.0

  42. Digging into RAC performance problems(cont’d) Global Enqueue Service StatisticsBAD GOOD --------------------------------- Ave global lock get time (ms): 0.2 0.0 Ave global lock convert time (ms): 0.0 0.0 Ratio of global lock gets vs global lock releases: 1.2 1.1 GCS and GES Messaging statistics -------------------------------- Ave message sent queue time (ms): 16.5 1.7 Ave message sent queue time on ksxp (ms): 29.4 2.5 Ave message received queue time (ms): 1.9 0.3 Ave GCS message process time (ms): 0.1 0.1 Ave GES message process time (ms): 0.1 0.0 % of direct sent messages: 49.7 63.4 % of indirect sent messages: 50.3 36.6 % of flow controlled messages: 0.0 0.0

  43. Implementing RAC in production Smooth transition from single instance to RAC Change ORACLE_HOME Relinking the RAC Option ON/OFF CLUSTER_DATABASE = TRUE/FALSE Start/Stop the second instance Start gradual movement of clients from one instance to another

  44. Relinking the RAC Option • Login as the Oracle software owner and shutdown all database instances on all nodes in the cluster. 2. cd $ORACLE_HOME/rdbms/lib 3. make -f ins_rdbms.mk rac_on (rac_off) If this step did not fail with fatal errors then proceed to step 4. 4. make -f ins_rdbms.mk ioracle

  45. Reference • Metalink Note:211177.1 RAC Survival Kit: Rac On / Rac Off - Relinking the RAC Option • Metalink Note:183340.1Frequently Asked Questions About the CLUSTER_INTERCONNECTS Parameter in 9i. • http://www.fi.muni.cz/~kripac/orac-nbd/ • The Oracle-on-Linux VMware Cookbook http://www.oracle.com/technology/tech/linux/vmware/cookbook/index.html • Build Your Own Oracle RAC 10g Cluster on Linux and FireWire http://www.oracle.com/technology/pub/articles/hunter_rac10g.html • Note:135714.1 Script to Collect RAC Diagnostic Information (racdiag.sql)

  46. Thank You Q&A zyumbyulev@mobiltel.bg

More Related