Implementing Oracle
1 / 62

Implementing Oracle Real Application ClusterBy Charles Pfeiffer888 - PowerPoint PPT Presentation

  • Uploaded on

Implementing Oracle Real Application Cluster By Charles Pfeiffer (888) 235-8916 [email protected] Who Are You?. DBA Experience? RAC Experience? ASM Experience? Linux Server Experience? Firewire Experience? Disk Subsystem IO Tuning Experience? Production v. Dev/Test.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Implementing Oracle Real Application ClusterBy Charles Pfeiffer888' - Leo

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Slide1 l.jpg

Implementing Oracle

Real Application Cluster

By Charles Pfeiffer

(888) 235-8916

[email protected]

Who are you l.jpg
Who Are You?

  • DBA Experience?

  • RAC Experience?

  • ASM Experience?

  • Linux Server Experience?

  • Firewire Experience?

  • Disk Subsystem IO Tuning Experience?

    • Production v. Dev/Test

What is this guy talking about l.jpg
What Is This Guy Talking About?

  • RAC

    • Done with Firewire on Linux

    • Something you can do without a huge budget

  • Why Linux?

    • Firewire Share Locking only works on Linux

  • Why Not Linux?

    • Disk IO. More to come…

What is rac l.jpg
What is RAC?

  • Multiple Instances (memory) accessing one Database (disk)

  • Fault-tolerance and Transparency

  • Performance and Scalability

  • Cheaper to grow out instead of up

    • Google Success Story

Rac done cheap l.jpg
RAC Done Cheap

  • Low-end RAC deployments cost around $20,000 for the hardware alone

  • You can build your own test system for 1/10th the cost, using Linux and Firewire

  • This is not a production-ready solution, but it is good for proof-of-concept and testing

Rac shared storage l.jpg
RAC Shared Storage

  • Fiber is the best

    • Costs more, but you get what you pay for

    • Fiber speed is up to 4 Gbps now

    • Supports traditional SCSI protocol

    • Supports up to 127 nodes

Rac shared storage continued l.jpg
RAC Shared Storage (continued)

  • SCSI is about ½ the cost of Fiber

    • Less nodes, but likely enough

    • Less performance, may not be good enough


    • Danger, Danger! Who comes up with this?

    • Better distance, poor performance and reliability

  • Stay away from NAS/NFS

    • Reliability and Performance Standards

    • Who says this is OK?

Rac shared storage continued8 l.jpg
RAC Shared Storage (continued)

  • Firewire

    • 1/30th the cost of SCSI

    • Not supported, not ready for production

    • Works well for testing and proof-of-concept

    • Only works with Linux

      • Special share locking programmed by Wim Coekaerts “Oracle’s Mr. Linux” and his team.

Rac shared storage continued9 l.jpg
RAC Shared Storage (continued)

  • Transfer Speeds (Max spec in MBps)

    • You’ll never get Max. You’re lucky to get half. Expect 1/3rd. More information in another presentation.

    • Serial = 0.014

    • IDE = 16.7, Ultra IDE = 33

    • USB1 = 1.5, USB2 = 60

    • Firewire = 50

    • ATA/100 = 12.5, SATA = 150, Ultra SATA = 187.5

    • FW SCSI = 20, Ultra SCSI = 40, Ultra3 SCSI = 80, Ultra 160 SCSI = 160, Ultra 320 SCSI = 320

    • Gb Fiber = 120, 2Gb Fiber = 240, 4 Gb Fiber = 480

Hardware requirements l.jpg
Hardware Requirements

  • Two Intel-based PCs

    • AMD is OK if you have it, Intel is cheaper

    • Cheap, old, used, whatever.

    • Pre-Pentium is fine, but will be slower

    • VMware is OK. Performance is not as good

      • Ask about running Linux and Oracle on VMware

Hardware requirements continued l.jpg
Hardware Requirements (continued)

  • 512 MB RAM each. Try hard for 1 GB.

  • 10 GB internal Hard Drive each

  • Two network cards each

  • CD-ROM, monitor, keyboard

Hardware requirements continued12 l.jpg
Hardware Requirements (continued)

  • Firewire card each

  • Dual-port Firewire external HDD

    • Do not connect the firewire drive until the firewire modules are installed and configured correctly!

  • Cabling for all

Install linux l.jpg
Install Linux

  • Oracle is tightly intertwined with RedHat

    • Jeffrey Hunter did this with WBEL

    • SuSE and some others work well

  • Typical/default setup for the most part

    • Manual Partitioning

      • /boot = 100 MB

      • Swap = 150% RAM or 200% if < 1 GB RAM

      • / gets the rest

      • Flexibility here, but /usr should not be a mount point

Install linux continued l.jpg
Install Linux (continued)

  • Typical/default setup for the most part (continued)

    • Separate networks for the two NICs

      • Interconnect cannot be the same network as the public

      • Don’t use DHCP

    • Time sync

Install linux continued15 l.jpg
Install Linux (continued)

  • Typical/default setup for the most part (continued)

    • Go ahead, install all packages!

      • If you don’t use/need it you can always disable it

    • up2date, up2date, up2date!!!

      • Oracle has specific package requirements

      • Don’t do this the hard way

Verify packages l.jpg
Verify Packages

  • Advanced Server doesn’t install everything

  • You may have chosen not to install everything

  • Find your requirements

    • There are minimum (and in some cases exact) package version requirements for RAC. Be sure to follow carefully

    • See the links to RAC documentation on the slide titled “More Information”

Firewire modules l.jpg
Firewire Modules


    • Choose the correct module for your system

    • Download and install it (rpm –ivh)

  • Add to /etc/modprobe.conf

    options sbp2 exclusive_login=0

    alias scsi_hostadapter sbp2

  • Once this is done on both machines you can power them down, connect the firewire device and restart them

Firewire modules continued l.jpg
Firewire Modules (continued)

  • lspci |grep –i firewire should return one line

  • lsmod |egrep “ohci1394”|sbp2|ieee1394|sd_mod|scsi_mod” should return one line each

  • fdisk

    • /CRS = 300 MB

    • /oradata (usually like to have multiples)

    • /orabackup

Hangcheck timer module l.jpg
Hangcheck-timer Module


  • Hangcheck tick setting

    • Time between system health checks

    • Default is 60 (seconds)

    • Oracle recommends 30

    • Maybe 10 or 15 (overhead v. discovery)

Hangcheck timer module continued l.jpg
Hangcheck-timer Module (continued)

  • Hangcheck margin setting

    • Time of tolerated delay before node reset

    • Default and Oracle recommendation is 180 (seconds)

    • Maybe 60 (quick reset v. delayed reset)

  • Add to /etc/modprobe.conf

    options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180

More kernel settings l.jpg
More Kernel Settings

  • Set via sysctl (sysctl –w net.core.)

    • rmem_default, rmem_max, wmem_default and wmem_max = 262144

  • sysctl –w kernel.

    • shmmax=2147483648 (2 GB)

    • sem=“250 32000 100 128”

    • sysrq = 0

    • core_uses_pid = 1

More kernel settings continued l.jpg
More Kernel Settings (continued)

  • sysctl –w fs.

    • file-max = 65536

  • Sysctl –w net.ipv4

    • ip_forward = 0

  • Sysctl –w net.ipv4.conf.default

    • rp_filter = 1

Oracle users and groups l.jpg
Oracle Users and Groups

  • Create the oracle user and the dba and oinstall groups

  • UIDs and GIDs must be the same on both machines

  • Create a software install directory on both machines’ internal drives (e.g. /opt/oracle)

Oracle users and groups continued l.jpg
Oracle Users and Groups (continued)

  • Profile



  • ORACLE_SID (each node is unique)

  • PATH=$ORACLE_HOME/bin:$ORACLE_BASE/common/oracle/bin:$PATH:$/usr/bin:/bin:/usr/X11/bin:/usr/local/bin

  • LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/oracm/lib:/lib:/usr/lib:/usr/local/lib


  • THREAD_FLAG=native (use OS Java version)

Introducing cache fusion l.jpg
Introducing Cache Fusion

  • Oracle Parallel Server Disk Writes

    • RAC High Speed Interconnect

  • Performance Gains

    • Outward Scalability

  • Performance Losses

    • Overhead for Cache Fusion

The network l.jpg
The Network

  • Hard-set, do not use auto-negotiate, auto-detect, auto-sense, auto-*, auto-bad

  • Gigabit Hub for the interconnect

    • We use crossover in the test.

    • Don’t mess with Cache Fusion!

    • Not supported

The network continued l.jpg
The Network (continued)

  • Set up a virtual IP address in hosts and the name server

    • Only need name/IP

    • Must be the same network as the public interface

  • Remove the linux name alias on the loopback line and put it on the right line

Firewall configuration l.jpg
Firewall Configuration

  • Allow incoming SSH

    • Key-based access only?

    • Don’t allow root/oracle remotely?

  • Allow incoming TNS

  • Allow all outgoing

  • Allow traffic related to established sessions

Firewall configuration continued l.jpg
Firewall Configuration (continued)

  • Log

    • All dropped?

    • All allowed?

    • Port specific?

  • SE Linux

    • Start with warn mode to see what you want to do

    • More specific than the firewall, but the firewall is usually good enough

Configuring ssh l.jpg
Configuring SSH

  • Used for

    • Remote administration

    • RAC software deployment

  • Create a key pair for root and oracle on each node

    • ssh-keygen –t dsa –N “”

    • Accept the defaults

    • cd ~/.ssh

    • scp <other_node>:./.ssh/<this_node>.pubkey

      • NOTE: Password is required for this scp because the key is not in place yet

Configure ssh continued l.jpg
Configure SSH (continued)

  • On the other node

    • cd ~/.ssh

    • cat <other_node>.pubkey >> authorized_keys

  • Make the initial connection to accept the key familiarity question

    • Software installation will error if it is presented with unexpected text

Introducing ocfs l.jpg
Introducing OCFS

  • Oracle Cluster File System

  • Used for RAC on Linux and Windows

  • Eliminates the need for third-party clusters

  • Can store all RAC-related and Database files

Install ocfs l.jpg
Install OCFS



  • OCFS v. ASM

    • ASM can store datafiles, OCFS can store everything

    • OCFS uses DBA-friendly, filesystem-style datafiles

  • Just download and install the RPMs

  • ocfs2console (this is the ocfs2 version)

    • OCFS2/RHEL4 not supported, but it works!

  • Select cluster > configure nodes

    • Add the name/IP of the public interfaces

Configuring ocfs for autostart l.jpg
Configuring OCFS for Autostart

  • Fix the chkconfig bug (OCFS2)

    • Remove the lines between (and including) ‘### Begin Init Info’ and ‘### End Init Info’ in /etc/init.d/o2cb

  • Reregister

    • chkconfig --del o2cb

    • chkconfig --add o2cb

  • Reload

    • /etc/init.d/o2cb offline ocfs2

    • /etc/init.d/o2cb unload

    • /etc/init.d/o2cb configure

Configure ocfs file systems l.jpg
Configure OCFS File Systems

  • Back to ocfs2console

    • Tasks > format

    • 4K Block Size. Get it right now, it can’t be changed.

    • Cluster Size Setting

      • 32K or 64K for OCR.

      • 128K or 256K for OLTP Data Files.

      • 512K or 1M for Data Warehouse Data Files.

      • 1M for backups

  • Reboot both nodes

    • OCFS FS should be mounted on both

Get the rac software l.jpg
Get the RAC Software

  • Oracle Database Server

  • Oracle Companion Software

  • Oracle Cluster (version dependent)

    • Oracle Clusterware

    • Oracle Cluster Ready Services

Install oracle cluster software l.jpg
Install Oracle Cluster Software

  • Unset environment variables



  • runInstaller

    • Use the ORA_CRS_HOME

    • Use a cluster name that doesn’t match either SID

    • Mirror the OCR and voting Disks! (10gR2)

Install oracle cluster software continued l.jpg
Install Oracle Cluster Software (continued)

  • Update CSS miscount

    • Timeout before evicting a node

    • Can cause installs to hang

    • Set it higher for slower disks

    • Recommend 60 for most disks, 480 for firewire

    • Edit before running it on each node

Install oracle cluster software continued41 l.jpg
Install Oracle Cluster Software (continued)

  • Run

    • Run on each node, one at a time

    • Takes a while to complete

    • Get several warnings. Ignore them.

    • May get a critical alert regarding virtual IP addresses at the end on the last node

      • Run the vipca GUI and set the interface information

  • Verify

    • $ORA_CRS_HOME/bin/olsnodes –n

      • Should see all nodes in the cluster

Install the db server l.jpg
Install the DB Server

  • Unset environment variables



  • runInstaller


    • After specifying the ORACLE_HOME the next screen should be a clustered install screen

      • Do not proceed if this is not the case. Something is wrong with the cluster. Fix it first!!!

Install the db server continued l.jpg
Install the DB Server(continued)

  • runInstaller (continued)

    • Select all nodes

    • Choose install Software only. Use dbca later, not as part of the initial installation

  • Run on all nodes, one at a time

Create the listener service l.jpg
Create the Listener Service

  • Run this from only one node in the cluster

  • Netca

    • Choose cluster configuration

    • Select all nodes

    • Use different names for each node’s service

Install companion software l.jpg
Install Companion Software

  • NCOMP is required or you will hit a bug ( –

  • runInstaller


    • Specify Hardware Cluster Installation

Transparent application failover taf l.jpg
Transparent Application Failover (TAF)

  • Available in 10g. Use it!!!

  • Automatically migrate sessions from a failed instance to a working instace

    • Session continues, uninterrupted to the user

  • Requires a simple modification to your configuration

  • Should be selected during DB creation

Create the db l.jpg
Create the DB

  • Run dbca on ONE node

    • Should be presented with a RAC welcome screen

    • Stop! If you don’t get the RAC welcome your cluster is not running or configured properly. Fix the cluster before you create the DB

    • Install on all nodes

    • Choose a “custom” installation unless you have already created a RAC template that uses TAF

    • Add a TAF Service

Configure taf l.jpg
Configure TAF

  • Just update the tnsnames.ora files

    ORCL1 =


    (ADDRESS = (PROTOCOL = TCP)(HOST = vip-rhcl1)(PORT = 1521))

    (ADDRESS = (PROTOCOL = TCP)(HOST = vip-rhcl2)(PORT = 1521))

    (LOAD_BALANCE = yes)




    (METHOD = BASIC) (RETRIES = 180) (DELAY = 5)

    ) ) )

Connecting to the db l.jpg
Connecting to the DB

  • You may connect to

    • An instance on either node

      • Transaction aborted when that instance dies

    • The DB service

      • Oracle chooses an instance to connect you to

      • Transaction aborted when that instance dies

    • The TAF DB service

      • Oracle chooses an instance to connect you to

      • Instance failure is transparent

Test taf l.jpg
Test TAF

  • Connect to the TAF Service with SQL

  • Select instance_name from v$instance;

  • Shut that instance down

  • Your session is still connected

  • Select instance_name from v$instance;

Verify your configuration l.jpg
Verify Your Configuration

  • Status of all instances

    srvctl status database –d <db name>

  • Status of a global service

    srvctl status service –d <db name> -s <TAF service name>

  • Status of all cluster applications

    srvctl status nodeapps –n <server name>

Verify your configuration continued l.jpg
Verify Your Configuration (continued)

  • List of configured DBs

    srvctl config database

  • List of services for a DB

    srvctl config service –d <db name>

  • List of applications for a server

    srvctl config nodeapps –n <server name> -a -g -s -l

Verify your configuration continued53 l.jpg
Verify Your Configuration (continued)

  • DB’s perspective of all running instances

    Select status, database_status dbstat,

    active_state state, host_name host from



    ----------- ------------ ------------- --------



Shutdown the cluster l.jpg
Shutdown the Cluster

  • Stop an instance

    srvctl stop instance –d <db name> -i <instance name>

  • Stop all instances

    srvctl stop database –d <db name>

  • Stop the server applications on a server

    srvctl stop nodeapps –n <server name>

Start the cluster l.jpg
Start the Cluster

  • Start all instances

    srvctl start database –d <db name>

  • Start an instance

    srvctl start instance –d <db name> -i <instance name>

  • Start the server applications on a server

    srvctl start nodeapps –n <server name>

Linux disk io why do i keep bringing it up l.jpg
Linux Disk IO: Why Do I Keep Bringing It Up?

  • The transfer size penalty

    • Ext3 maxes out at 512 KB, Ext2 at 256 KB

    • UFS/VxFS/NTFS all do 1 MB

    • Latest version of NTFS is actually prepared to do 2 MB

  • But what does it all mean?

    • The IOps threshold is creeping up fast

    • Be careful of what you put in production

  • People Say Linux Performs well

    • What are you using it for?

    • Corvettes v. Dump Trucks

Summary l.jpg

  • You can do RAC cheap

    • Use Linux and Firewire

    • Not meant for production

  • You can do RAC easy

    • More effort is spent in prepping the host server than in actually configuring RAC

    • Don’t relax your efforts there. It is crucial that the host servers are configured correctly

    • OCFS Makes It Simple

    • Don’t skip steps and “get back to them later”. Do everything in sequence and make sure it works.

This presentation l.jpg
This Presentation

  • This document is not for commercial re-use or distribution without the consent of the author

  • The author does not guarantee this document to be error free

  • Submit questions/corrections/comments to the author:

    Charles Pfeiffer, [email protected]

Take a bow l.jpg
Take a Bow

  • Jeffrey Hunter. Great step-by-step guide on building RAC on Linux with Firewire

  • Wim Coekaerts. Wrote the firewire shared locking modules for Linux

  • Werner Puschitz. Many great step-by-step documents for different Oracle configurations on different versions of Linux

More information l.jpg
More Information









Wrap it up already l.jpg
Wrap It Up Already!

  • Final Q&A

  • Understanding the IO Under Your Database

    • Complete IO Tuning

    • More detail on transfer size and Linux IO

  • Contact Me

    • 804.901.3992

    • [email protected]