slide1 l.
Skip this Video
Loading SlideShow in 5 Seconds..
Implementing Oracle Real Application Cluster By Charles Pfeiffer (888) 235-8916 CJPfeiffer@RemoteControlDBA PowerPoint Presentation
Download Presentation
Implementing Oracle Real Application Cluster By Charles Pfeiffer (888) 235-8916 CJPfeiffer@RemoteControlDBA

Loading in 2 Seconds...

play fullscreen
1 / 62

Implementing Oracle Real Application Cluster By Charles Pfeiffer (888) 235-8916 CJPfeiffer@RemoteControlDBA - PowerPoint PPT Presentation

  • Uploaded on

Implementing Oracle Real Application Cluster By Charles Pfeiffer (888) 235-8916 Who Are You?. DBA Experience? RAC Experience? ASM Experience? Linux Server Experience? Firewire Experience? Disk Subsystem IO Tuning Experience? Production v. Dev/Test.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

Implementing Oracle Real Application Cluster By Charles Pfeiffer (888) 235-8916 CJPfeiffer@RemoteControlDBA

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Implementing Oracle

Real Application Cluster

By Charles Pfeiffer

(888) 235-8916

who are you
Who Are You?
  • DBA Experience?
  • RAC Experience?
  • ASM Experience?
  • Linux Server Experience?
  • Firewire Experience?
  • Disk Subsystem IO Tuning Experience?
    • Production v. Dev/Test
what is this guy talking about
What Is This Guy Talking About?
  • RAC
    • Done with Firewire on Linux
    • Something you can do without a huge budget
  • Why Linux?
    • Firewire Share Locking only works on Linux
  • Why Not Linux?
    • Disk IO. More to come…
what is rac
What is RAC?
  • Multiple Instances (memory) accessing one Database (disk)
  • Fault-tolerance and Transparency
  • Performance and Scalability
  • Cheaper to grow out instead of up
    • Google Success Story
rac done cheap
RAC Done Cheap
  • Low-end RAC deployments cost around $20,000 for the hardware alone
  • You can build your own test system for 1/10th the cost, using Linux and Firewire
  • This is not a production-ready solution, but it is good for proof-of-concept and testing
rac shared storage
RAC Shared Storage
  • Fiber is the best
    • Costs more, but you get what you pay for
    • Fiber speed is up to 4 Gbps now
    • Supports traditional SCSI protocol
    • Supports up to 127 nodes
rac shared storage continued
RAC Shared Storage (continued)
  • SCSI is about ½ the cost of Fiber
    • Less nodes, but likely enough
    • Less performance, may not be good enough
    • Danger, Danger! Who comes up with this?
    • Better distance, poor performance and reliability
  • Stay away from NAS/NFS
    • Reliability and Performance Standards
    • Who says this is OK?
rac shared storage continued8
RAC Shared Storage (continued)
  • Firewire
    • 1/30th the cost of SCSI
    • Not supported, not ready for production
    • Works well for testing and proof-of-concept
    • Only works with Linux
      • Special share locking programmed by Wim Coekaerts “Oracle’s Mr. Linux” and his team.
rac shared storage continued9
RAC Shared Storage (continued)
  • Transfer Speeds (Max spec in MBps)
    • You’ll never get Max. You’re lucky to get half. Expect 1/3rd. More information in another presentation.
    • Serial = 0.014
    • IDE = 16.7, Ultra IDE = 33
    • USB1 = 1.5, USB2 = 60
    • Firewire = 50
    • ATA/100 = 12.5, SATA = 150, Ultra SATA = 187.5
    • FW SCSI = 20, Ultra SCSI = 40, Ultra3 SCSI = 80, Ultra 160 SCSI = 160, Ultra 320 SCSI = 320
    • Gb Fiber = 120, 2Gb Fiber = 240, 4 Gb Fiber = 480
hardware requirements
Hardware Requirements
  • Two Intel-based PCs
    • AMD is OK if you have it, Intel is cheaper
    • Cheap, old, used, whatever.
    • Pre-Pentium is fine, but will be slower
    • VMware is OK. Performance is not as good
      • Ask about running Linux and Oracle on VMware
hardware requirements continued
Hardware Requirements (continued)
  • 512 MB RAM each. Try hard for 1 GB.
  • 10 GB internal Hard Drive each
  • Two network cards each
  • CD-ROM, monitor, keyboard
hardware requirements continued12
Hardware Requirements (continued)
  • Firewire card each
  • Dual-port Firewire external HDD
    • Do not connect the firewire drive until the firewire modules are installed and configured correctly!
  • Cabling for all
install linux
Install Linux
  • Oracle is tightly intertwined with RedHat
    • Jeffrey Hunter did this with WBEL
    • SuSE and some others work well
  • Typical/default setup for the most part
    • Manual Partitioning
      • /boot = 100 MB
      • Swap = 150% RAM or 200% if < 1 GB RAM
      • / gets the rest
      • Flexibility here, but /usr should not be a mount point
install linux continued
Install Linux (continued)
  • Typical/default setup for the most part (continued)
    • Separate networks for the two NICs
      • Interconnect cannot be the same network as the public
      • Don’t use DHCP
    • Time sync
install linux continued15
Install Linux (continued)
  • Typical/default setup for the most part (continued)
    • Go ahead, install all packages!
      • If you don’t use/need it you can always disable it
    • up2date, up2date, up2date!!!
      • Oracle has specific package requirements
      • Don’t do this the hard way
verify packages
Verify Packages
  • Advanced Server doesn’t install everything
  • You may have chosen not to install everything
  • Find your requirements
    • There are minimum (and in some cases exact) package version requirements for RAC. Be sure to follow carefully
    • See the links to RAC documentation on the slide titled “More Information”
firewire modules
Firewire Modules
    • Choose the correct module for your system
    • Download and install it (rpm –ivh)
  • Add to /etc/modprobe.conf

options sbp2 exclusive_login=0

alias scsi_hostadapter sbp2

  • Once this is done on both machines you can power them down, connect the firewire device and restart them
firewire modules continued
Firewire Modules (continued)
  • lspci |grep –i firewire should return one line
  • lsmod |egrep “ohci1394”|sbp2|ieee1394|sd_mod|scsi_mod” should return one line each
  • fdisk
    • /CRS = 300 MB
    • /oradata (usually like to have multiples)
    • /orabackup
hangcheck timer module
Hangcheck-timer Module
  • Hangcheck tick setting
    • Time between system health checks
    • Default is 60 (seconds)
    • Oracle recommends 30
    • Maybe 10 or 15 (overhead v. discovery)
hangcheck timer module continued
Hangcheck-timer Module (continued)
  • Hangcheck margin setting
    • Time of tolerated delay before node reset
    • Default and Oracle recommendation is 180 (seconds)
    • Maybe 60 (quick reset v. delayed reset)
  • Add to /etc/modprobe.conf

options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180

more kernel settings
More Kernel Settings
  • Set via sysctl (sysctl –w net.core.)
    • rmem_default, rmem_max, wmem_default and wmem_max = 262144
  • sysctl –w kernel.
    • shmmax=2147483648 (2 GB)
    • sem=“250 32000 100 128”
    • sysrq = 0
    • core_uses_pid = 1
more kernel settings continued
More Kernel Settings (continued)
  • sysctl –w fs.
    • file-max = 65536
  • Sysctl –w net.ipv4
    • ip_forward = 0
  • Sysctl –w net.ipv4.conf.default
    • rp_filter = 1
oracle users and groups
Oracle Users and Groups
  • Create the oracle user and the dba and oinstall groups
  • UIDs and GIDs must be the same on both machines
  • Create a software install directory on both machines’ internal drives (e.g. /opt/oracle)
oracle users and groups continued
Oracle Users and Groups (continued)
  • Profile
  • ORACLE_SID (each node is unique)
  • PATH=$ORACLE_HOME/bin:$ORACLE_BASE/common/oracle/bin:$PATH:$/usr/bin:/bin:/usr/X11/bin:/usr/local/bin
  • LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/oracm/lib:/lib:/usr/lib:/usr/local/lib
  • THREAD_FLAG=native (use OS Java version)
introducing cache fusion
Introducing Cache Fusion
  • Oracle Parallel Server Disk Writes
    • RAC High Speed Interconnect
  • Performance Gains
    • Outward Scalability
  • Performance Losses
    • Overhead for Cache Fusion
the network
The Network
  • Hard-set, do not use auto-negotiate, auto-detect, auto-sense, auto-*, auto-bad
  • Gigabit Hub for the interconnect
    • We use crossover in the test.
    • Don’t mess with Cache Fusion!
    • Not supported
the network continued
The Network (continued)
  • Set up a virtual IP address in hosts and the name server
    • Only need name/IP
    • Must be the same network as the public interface
  • Remove the linux name alias on the loopback line and put it on the right line
firewall configuration
Firewall Configuration
  • Allow incoming SSH
    • Key-based access only?
    • Don’t allow root/oracle remotely?
  • Allow incoming TNS
  • Allow all outgoing
  • Allow traffic related to established sessions
firewall configuration continued
Firewall Configuration (continued)
  • Log
    • All dropped?
    • All allowed?
    • Port specific?
  • SE Linux
    • Start with warn mode to see what you want to do
    • More specific than the firewall, but the firewall is usually good enough
configuring ssh
Configuring SSH
  • Used for
    • Remote administration
    • RAC software deployment
  • Create a key pair for root and oracle on each node
    • ssh-keygen –t dsa –N “”
    • Accept the defaults
    • cd ~/.ssh
    • scp <other_node>:./.ssh/<this_node>.pubkey
      • NOTE: Password is required for this scp because the key is not in place yet
configure ssh continued
Configure SSH (continued)
  • On the other node
    • cd ~/.ssh
    • cat <other_node>.pubkey >> authorized_keys
  • Make the initial connection to accept the key familiarity question
    • Software installation will error if it is presented with unexpected text
introducing ocfs
Introducing OCFS
  • Oracle Cluster File System
  • Used for RAC on Linux and Windows
  • Eliminates the need for third-party clusters
  • Can store all RAC-related and Database files
install ocfs
Install OCFS
  • OCFS v. ASM
    • ASM can store datafiles, OCFS can store everything
    • OCFS uses DBA-friendly, filesystem-style datafiles
  • Just download and install the RPMs
  • ocfs2console (this is the ocfs2 version)
    • OCFS2/RHEL4 not supported, but it works!
  • Select cluster > configure nodes
      • Add the name/IP of the public interfaces
configuring ocfs for autostart
Configuring OCFS for Autostart
  • Fix the chkconfig bug (OCFS2)
    • Remove the lines between (and including) ‘### Begin Init Info’ and ‘### End Init Info’ in /etc/init.d/o2cb
  • Reregister
    • chkconfig --del o2cb
    • chkconfig --add o2cb
  • Reload
    • /etc/init.d/o2cb offline ocfs2
    • /etc/init.d/o2cb unload
    • /etc/init.d/o2cb configure
configure ocfs file systems
Configure OCFS File Systems
  • Back to ocfs2console
    • Tasks > format
    • 4K Block Size. Get it right now, it can’t be changed.
    • Cluster Size Setting
      • 32K or 64K for OCR.
      • 128K or 256K for OLTP Data Files.
      • 512K or 1M for Data Warehouse Data Files.
      • 1M for backups
  • Reboot both nodes
    • OCFS FS should be mounted on both
get the rac software
Get the RAC Software
  • Oracle Database Server
  • Oracle Companion Software
  • Oracle Cluster (version dependent)
    • Oracle Clusterware
    • Oracle Cluster Ready Services
install oracle cluster software
Install Oracle Cluster Software
  • Unset environment variables
  • runInstaller
    • Use the ORA_CRS_HOME
    • Use a cluster name that doesn’t match either SID
    • Mirror the OCR and voting Disks! (10gR2)
install oracle cluster software continued
Install Oracle Cluster Software (continued)
  • Update CSS miscount
    • Timeout before evicting a node
    • Can cause installs to hang
    • Set it higher for slower disks
    • Recommend 60 for most disks, 480 for firewire
    • Edit before running it on each node
install oracle cluster software continued41
Install Oracle Cluster Software (continued)
  • Run
    • Run on each node, one at a time
    • Takes a while to complete
    • Get several warnings. Ignore them.
    • May get a critical alert regarding virtual IP addresses at the end on the last node
      • Run the vipca GUI and set the interface information
  • Verify
    • $ORA_CRS_HOME/bin/olsnodes –n
      • Should see all nodes in the cluster
install the db server
Install the DB Server
  • Unset environment variables
  • runInstaller
    • After specifying the ORACLE_HOME the next screen should be a clustered install screen
      • Do not proceed if this is not the case. Something is wrong with the cluster. Fix it first!!!
install the db server continued
Install the DB Server(continued)
  • runInstaller (continued)
    • Select all nodes
    • Choose install Software only. Use dbca later, not as part of the initial installation
  • Run on all nodes, one at a time
create the listener service
Create the Listener Service
  • Run this from only one node in the cluster
  • Netca
    • Choose cluster configuration
    • Select all nodes
    • Use different names for each node’s service
install companion software
Install Companion Software
  • NCOMP is required or you will hit a bug ( –
  • runInstaller
    • Specify Hardware Cluster Installation
transparent application failover taf
Transparent Application Failover (TAF)
  • Available in 10g. Use it!!!
  • Automatically migrate sessions from a failed instance to a working instace
    • Session continues, uninterrupted to the user
  • Requires a simple modification to your configuration
  • Should be selected during DB creation
create the db
Create the DB
  • Run dbca on ONE node
    • Should be presented with a RAC welcome screen
    • Stop! If you don’t get the RAC welcome your cluster is not running or configured properly. Fix the cluster before you create the DB
    • Install on all nodes
    • Choose a “custom” installation unless you have already created a RAC template that uses TAF
    • Add a TAF Service
configure taf
Configure TAF
  • Just update the tnsnames.ora files



(ADDRESS = (PROTOCOL = TCP)(HOST = vip-rhcl1)(PORT = 1521))

(ADDRESS = (PROTOCOL = TCP)(HOST = vip-rhcl2)(PORT = 1521))






) ) )

connecting to the db
Connecting to the DB
  • You may connect to
    • An instance on either node
      • Transaction aborted when that instance dies
    • The DB service
      • Oracle chooses an instance to connect you to
      • Transaction aborted when that instance dies
    • The TAF DB service
      • Oracle chooses an instance to connect you to
      • Instance failure is transparent
test taf
Test TAF
  • Connect to the TAF Service with SQL
  • Select instance_name from v$instance;
  • Shut that instance down
  • Your session is still connected
  • Select instance_name from v$instance;
verify your configuration
Verify Your Configuration
  • Status of all instances

srvctl status database –d <db name>

  • Status of a global service

srvctl status service –d <db name> -s <TAF service name>

  • Status of all cluster applications

srvctl status nodeapps –n <server name>

verify your configuration continued
Verify Your Configuration (continued)
  • List of configured DBs

srvctl config database

  • List of services for a DB

srvctl config service –d <db name>

  • List of applications for a server

srvctl config nodeapps –n <server name> -a -g -s -l

verify your configuration continued53
Verify Your Configuration (continued)
  • DB’s perspective of all running instances

Select status, database_status dbstat,

active_state state, host_name host from



----------- ------------ ------------- --------



shutdown the cluster
Shutdown the Cluster
  • Stop an instance

srvctl stop instance –d <db name> -i <instance name>

  • Stop all instances

srvctl stop database –d <db name>

  • Stop the server applications on a server

srvctl stop nodeapps –n <server name>

start the cluster
Start the Cluster
  • Start all instances

srvctl start database –d <db name>

  • Start an instance

srvctl start instance –d <db name> -i <instance name>

  • Start the server applications on a server

srvctl start nodeapps –n <server name>

linux disk io why do i keep bringing it up
Linux Disk IO: Why Do I Keep Bringing It Up?
  • The transfer size penalty
    • Ext3 maxes out at 512 KB, Ext2 at 256 KB
    • UFS/VxFS/NTFS all do 1 MB
    • Latest version of NTFS is actually prepared to do 2 MB
  • But what does it all mean?
    • The IOps threshold is creeping up fast
    • Be careful of what you put in production
  • People Say Linux Performs well
    • What are you using it for?
    • Corvettes v. Dump Trucks
  • You can do RAC cheap
    • Use Linux and Firewire
    • Not meant for production
  • You can do RAC easy
    • More effort is spent in prepping the host server than in actually configuring RAC
    • Don’t relax your efforts there. It is crucial that the host servers are configured correctly
    • OCFS Makes It Simple
    • Don’t skip steps and “get back to them later”. Do everything in sequence and make sure it works.
this presentation
This Presentation
  • This document is not for commercial re-use or distribution without the consent of the author
  • The author does not guarantee this document to be error free
  • Submit questions/corrections/comments to the author:

Charles Pfeiffer,

take a bow
Take a Bow
  • Jeffrey Hunter. Great step-by-step guide on building RAC on Linux with Firewire
  • Wim Coekaerts. Wrote the firewire shared locking modules for Linux
  • Werner Puschitz. Many great step-by-step documents for different Oracle configurations on different versions of Linux
more information
More Information
wrap it up already
Wrap It Up Already!
  • Final Q&A
  • Understanding the IO Under Your Database
    • Complete IO Tuning
    • More detail on transfer size and Linux IO
  • Contact Me
    • 804.901.3992