slide1 l.
Skip this Video
Loading SlideShow in 5 Seconds..
Implementing Oracle Real Application Cluster By Charles Pfeiffer (888) 235-8916 CJPfeiffer@RemoteControlDBA PowerPoint Presentation
Download Presentation
Implementing Oracle Real Application Cluster By Charles Pfeiffer (888) 235-8916 CJPfeiffer@RemoteControlDBA

Loading in 2 Seconds...

play fullscreen
1 / 62

Implementing Oracle Real Application Cluster By Charles Pfeiffer (888) 235-8916 CJPfeiffer@RemoteControlDBA - PowerPoint PPT Presentation

  • Uploaded on

Implementing Oracle Real Application Cluster By Charles Pfeiffer (888) 235-8916 Who Are You?. DBA Experience? RAC Experience? ASM Experience? Linux Server Experience? Firewire Experience? Disk Subsystem IO Tuning Experience? Production v. Dev/Test.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Implementing Oracle Real Application Cluster By Charles Pfeiffer (888) 235-8916 CJPfeiffer@RemoteControlDBA' - Leo

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Implementing Oracle

Real Application Cluster

By Charles Pfeiffer

(888) 235-8916

who are you
Who Are You?
  • DBA Experience?
  • RAC Experience?
  • ASM Experience?
  • Linux Server Experience?
  • Firewire Experience?
  • Disk Subsystem IO Tuning Experience?
    • Production v. Dev/Test
what is this guy talking about
What Is This Guy Talking About?
  • RAC
    • Done with Firewire on Linux
    • Something you can do without a huge budget
  • Why Linux?
    • Firewire Share Locking only works on Linux
  • Why Not Linux?
    • Disk IO. More to come…
what is rac
What is RAC?
  • Multiple Instances (memory) accessing one Database (disk)
  • Fault-tolerance and Transparency
  • Performance and Scalability
  • Cheaper to grow out instead of up
    • Google Success Story
rac done cheap
RAC Done Cheap
  • Low-end RAC deployments cost around $20,000 for the hardware alone
  • You can build your own test system for 1/10th the cost, using Linux and Firewire
  • This is not a production-ready solution, but it is good for proof-of-concept and testing
rac shared storage
RAC Shared Storage
  • Fiber is the best
    • Costs more, but you get what you pay for
    • Fiber speed is up to 4 Gbps now
    • Supports traditional SCSI protocol
    • Supports up to 127 nodes
rac shared storage continued
RAC Shared Storage (continued)
  • SCSI is about ½ the cost of Fiber
    • Less nodes, but likely enough
    • Less performance, may not be good enough
    • Danger, Danger! Who comes up with this?
    • Better distance, poor performance and reliability
  • Stay away from NAS/NFS
    • Reliability and Performance Standards
    • Who says this is OK?
rac shared storage continued8
RAC Shared Storage (continued)
  • Firewire
    • 1/30th the cost of SCSI
    • Not supported, not ready for production
    • Works well for testing and proof-of-concept
    • Only works with Linux
      • Special share locking programmed by Wim Coekaerts “Oracle’s Mr. Linux” and his team.
rac shared storage continued9
RAC Shared Storage (continued)
  • Transfer Speeds (Max spec in MBps)
    • You’ll never get Max. You’re lucky to get half. Expect 1/3rd. More information in another presentation.
    • Serial = 0.014
    • IDE = 16.7, Ultra IDE = 33
    • USB1 = 1.5, USB2 = 60
    • Firewire = 50
    • ATA/100 = 12.5, SATA = 150, Ultra SATA = 187.5
    • FW SCSI = 20, Ultra SCSI = 40, Ultra3 SCSI = 80, Ultra 160 SCSI = 160, Ultra 320 SCSI = 320
    • Gb Fiber = 120, 2Gb Fiber = 240, 4 Gb Fiber = 480
hardware requirements
Hardware Requirements
  • Two Intel-based PCs
    • AMD is OK if you have it, Intel is cheaper
    • Cheap, old, used, whatever.
    • Pre-Pentium is fine, but will be slower
    • VMware is OK. Performance is not as good
      • Ask about running Linux and Oracle on VMware
hardware requirements continued
Hardware Requirements (continued)
  • 512 MB RAM each. Try hard for 1 GB.
  • 10 GB internal Hard Drive each
  • Two network cards each
  • CD-ROM, monitor, keyboard
hardware requirements continued12
Hardware Requirements (continued)
  • Firewire card each
  • Dual-port Firewire external HDD
    • Do not connect the firewire drive until the firewire modules are installed and configured correctly!
  • Cabling for all
install linux
Install Linux
  • Oracle is tightly intertwined with RedHat
    • Jeffrey Hunter did this with WBEL
    • SuSE and some others work well
  • Typical/default setup for the most part
    • Manual Partitioning
      • /boot = 100 MB
      • Swap = 150% RAM or 200% if < 1 GB RAM
      • / gets the rest
      • Flexibility here, but /usr should not be a mount point
install linux continued
Install Linux (continued)
  • Typical/default setup for the most part (continued)
    • Separate networks for the two NICs
      • Interconnect cannot be the same network as the public
      • Don’t use DHCP
    • Time sync
install linux continued15
Install Linux (continued)
  • Typical/default setup for the most part (continued)
    • Go ahead, install all packages!
      • If you don’t use/need it you can always disable it
    • up2date, up2date, up2date!!!
      • Oracle has specific package requirements
      • Don’t do this the hard way
verify packages
Verify Packages
  • Advanced Server doesn’t install everything
  • You may have chosen not to install everything
  • Find your requirements
    • There are minimum (and in some cases exact) package version requirements for RAC. Be sure to follow carefully
    • See the links to RAC documentation on the slide titled “More Information”
firewire modules
Firewire Modules
    • Choose the correct module for your system
    • Download and install it (rpm –ivh)
  • Add to /etc/modprobe.conf

options sbp2 exclusive_login=0

alias scsi_hostadapter sbp2

  • Once this is done on both machines you can power them down, connect the firewire device and restart them
firewire modules continued
Firewire Modules (continued)
  • lspci |grep –i firewire should return one line
  • lsmod |egrep “ohci1394”|sbp2|ieee1394|sd_mod|scsi_mod” should return one line each
  • fdisk
    • /CRS = 300 MB
    • /oradata (usually like to have multiples)
    • /orabackup
hangcheck timer module
Hangcheck-timer Module
  • Hangcheck tick setting
    • Time between system health checks
    • Default is 60 (seconds)
    • Oracle recommends 30
    • Maybe 10 or 15 (overhead v. discovery)
hangcheck timer module continued
Hangcheck-timer Module (continued)
  • Hangcheck margin setting
    • Time of tolerated delay before node reset
    • Default and Oracle recommendation is 180 (seconds)
    • Maybe 60 (quick reset v. delayed reset)
  • Add to /etc/modprobe.conf

options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180

more kernel settings
More Kernel Settings
  • Set via sysctl (sysctl –w net.core.)
    • rmem_default, rmem_max, wmem_default and wmem_max = 262144
  • sysctl –w kernel.
    • shmmax=2147483648 (2 GB)
    • sem=“250 32000 100 128”
    • sysrq = 0
    • core_uses_pid = 1
more kernel settings continued
More Kernel Settings (continued)
  • sysctl –w fs.
    • file-max = 65536
  • Sysctl –w net.ipv4
    • ip_forward = 0
  • Sysctl –w net.ipv4.conf.default
    • rp_filter = 1
oracle users and groups
Oracle Users and Groups
  • Create the oracle user and the dba and oinstall groups
  • UIDs and GIDs must be the same on both machines
  • Create a software install directory on both machines’ internal drives (e.g. /opt/oracle)
oracle users and groups continued
Oracle Users and Groups (continued)
  • Profile
  • ORACLE_SID (each node is unique)
  • PATH=$ORACLE_HOME/bin:$ORACLE_BASE/common/oracle/bin:$PATH:$/usr/bin:/bin:/usr/X11/bin:/usr/local/bin
  • LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/oracm/lib:/lib:/usr/lib:/usr/local/lib
  • THREAD_FLAG=native (use OS Java version)
introducing cache fusion
Introducing Cache Fusion
  • Oracle Parallel Server Disk Writes
    • RAC High Speed Interconnect
  • Performance Gains
    • Outward Scalability
  • Performance Losses
    • Overhead for Cache Fusion
the network
The Network
  • Hard-set, do not use auto-negotiate, auto-detect, auto-sense, auto-*, auto-bad
  • Gigabit Hub for the interconnect
    • We use crossover in the test.
    • Don’t mess with Cache Fusion!
    • Not supported
the network continued
The Network (continued)
  • Set up a virtual IP address in hosts and the name server
    • Only need name/IP
    • Must be the same network as the public interface
  • Remove the linux name alias on the loopback line and put it on the right line
firewall configuration
Firewall Configuration
  • Allow incoming SSH
    • Key-based access only?
    • Don’t allow root/oracle remotely?
  • Allow incoming TNS
  • Allow all outgoing
  • Allow traffic related to established sessions
firewall configuration continued
Firewall Configuration (continued)
  • Log
    • All dropped?
    • All allowed?
    • Port specific?
  • SE Linux
    • Start with warn mode to see what you want to do
    • More specific than the firewall, but the firewall is usually good enough
configuring ssh
Configuring SSH
  • Used for
    • Remote administration
    • RAC software deployment
  • Create a key pair for root and oracle on each node
    • ssh-keygen –t dsa –N “”
    • Accept the defaults
    • cd ~/.ssh
    • scp <other_node>:./.ssh/<this_node>.pubkey
      • NOTE: Password is required for this scp because the key is not in place yet
configure ssh continued
Configure SSH (continued)
  • On the other node
    • cd ~/.ssh
    • cat <other_node>.pubkey >> authorized_keys
  • Make the initial connection to accept the key familiarity question
    • Software installation will error if it is presented with unexpected text
introducing ocfs
Introducing OCFS
  • Oracle Cluster File System
  • Used for RAC on Linux and Windows
  • Eliminates the need for third-party clusters
  • Can store all RAC-related and Database files
install ocfs
Install OCFS
  • OCFS v. ASM
    • ASM can store datafiles, OCFS can store everything
    • OCFS uses DBA-friendly, filesystem-style datafiles
  • Just download and install the RPMs
  • ocfs2console (this is the ocfs2 version)
    • OCFS2/RHEL4 not supported, but it works!
  • Select cluster > configure nodes
      • Add the name/IP of the public interfaces
configuring ocfs for autostart
Configuring OCFS for Autostart
  • Fix the chkconfig bug (OCFS2)
    • Remove the lines between (and including) ‘### Begin Init Info’ and ‘### End Init Info’ in /etc/init.d/o2cb
  • Reregister
    • chkconfig --del o2cb
    • chkconfig --add o2cb
  • Reload
    • /etc/init.d/o2cb offline ocfs2
    • /etc/init.d/o2cb unload
    • /etc/init.d/o2cb configure
configure ocfs file systems
Configure OCFS File Systems
  • Back to ocfs2console
    • Tasks > format
    • 4K Block Size. Get it right now, it can’t be changed.
    • Cluster Size Setting
      • 32K or 64K for OCR.
      • 128K or 256K for OLTP Data Files.
      • 512K or 1M for Data Warehouse Data Files.
      • 1M for backups
  • Reboot both nodes
    • OCFS FS should be mounted on both
get the rac software
Get the RAC Software
  • Oracle Database Server
  • Oracle Companion Software
  • Oracle Cluster (version dependent)
    • Oracle Clusterware
    • Oracle Cluster Ready Services
install oracle cluster software
Install Oracle Cluster Software
  • Unset environment variables
  • runInstaller
    • Use the ORA_CRS_HOME
    • Use a cluster name that doesn’t match either SID
    • Mirror the OCR and voting Disks! (10gR2)
install oracle cluster software continued
Install Oracle Cluster Software (continued)
  • Update CSS miscount
    • Timeout before evicting a node
    • Can cause installs to hang
    • Set it higher for slower disks
    • Recommend 60 for most disks, 480 for firewire
    • Edit before running it on each node
install oracle cluster software continued41
Install Oracle Cluster Software (continued)
  • Run
    • Run on each node, one at a time
    • Takes a while to complete
    • Get several warnings. Ignore them.
    • May get a critical alert regarding virtual IP addresses at the end on the last node
      • Run the vipca GUI and set the interface information
  • Verify
    • $ORA_CRS_HOME/bin/olsnodes –n
      • Should see all nodes in the cluster
install the db server
Install the DB Server
  • Unset environment variables
  • runInstaller
    • After specifying the ORACLE_HOME the next screen should be a clustered install screen
      • Do not proceed if this is not the case. Something is wrong with the cluster. Fix it first!!!
install the db server continued
Install the DB Server(continued)
  • runInstaller (continued)
    • Select all nodes
    • Choose install Software only. Use dbca later, not as part of the initial installation
  • Run on all nodes, one at a time
create the listener service
Create the Listener Service
  • Run this from only one node in the cluster
  • Netca
    • Choose cluster configuration
    • Select all nodes
    • Use different names for each node’s service
install companion software
Install Companion Software
  • NCOMP is required or you will hit a bug ( –
  • runInstaller
    • Specify Hardware Cluster Installation
transparent application failover taf
Transparent Application Failover (TAF)
  • Available in 10g. Use it!!!
  • Automatically migrate sessions from a failed instance to a working instace
    • Session continues, uninterrupted to the user
  • Requires a simple modification to your configuration
  • Should be selected during DB creation
create the db
Create the DB
  • Run dbca on ONE node
    • Should be presented with a RAC welcome screen
    • Stop! If you don’t get the RAC welcome your cluster is not running or configured properly. Fix the cluster before you create the DB
    • Install on all nodes
    • Choose a “custom” installation unless you have already created a RAC template that uses TAF
    • Add a TAF Service
configure taf
Configure TAF
  • Just update the tnsnames.ora files



(ADDRESS = (PROTOCOL = TCP)(HOST = vip-rhcl1)(PORT = 1521))

(ADDRESS = (PROTOCOL = TCP)(HOST = vip-rhcl2)(PORT = 1521))






) ) )

connecting to the db
Connecting to the DB
  • You may connect to
    • An instance on either node
      • Transaction aborted when that instance dies
    • The DB service
      • Oracle chooses an instance to connect you to
      • Transaction aborted when that instance dies
    • The TAF DB service
      • Oracle chooses an instance to connect you to
      • Instance failure is transparent
test taf
Test TAF
  • Connect to the TAF Service with SQL
  • Select instance_name from v$instance;
  • Shut that instance down
  • Your session is still connected
  • Select instance_name from v$instance;
verify your configuration
Verify Your Configuration
  • Status of all instances

srvctl status database –d <db name>

  • Status of a global service

srvctl status service –d <db name> -s <TAF service name>

  • Status of all cluster applications

srvctl status nodeapps –n <server name>

verify your configuration continued
Verify Your Configuration (continued)
  • List of configured DBs

srvctl config database

  • List of services for a DB

srvctl config service –d <db name>

  • List of applications for a server

srvctl config nodeapps –n <server name> -a -g -s -l

verify your configuration continued53
Verify Your Configuration (continued)
  • DB’s perspective of all running instances

Select status, database_status dbstat,

active_state state, host_name host from



----------- ------------ ------------- --------



shutdown the cluster
Shutdown the Cluster
  • Stop an instance

srvctl stop instance –d <db name> -i <instance name>

  • Stop all instances

srvctl stop database –d <db name>

  • Stop the server applications on a server

srvctl stop nodeapps –n <server name>

start the cluster
Start the Cluster
  • Start all instances

srvctl start database –d <db name>

  • Start an instance

srvctl start instance –d <db name> -i <instance name>

  • Start the server applications on a server

srvctl start nodeapps –n <server name>

linux disk io why do i keep bringing it up
Linux Disk IO: Why Do I Keep Bringing It Up?
  • The transfer size penalty
    • Ext3 maxes out at 512 KB, Ext2 at 256 KB
    • UFS/VxFS/NTFS all do 1 MB
    • Latest version of NTFS is actually prepared to do 2 MB
  • But what does it all mean?
    • The IOps threshold is creeping up fast
    • Be careful of what you put in production
  • People Say Linux Performs well
    • What are you using it for?
    • Corvettes v. Dump Trucks
  • You can do RAC cheap
    • Use Linux and Firewire
    • Not meant for production
  • You can do RAC easy
    • More effort is spent in prepping the host server than in actually configuring RAC
    • Don’t relax your efforts there. It is crucial that the host servers are configured correctly
    • OCFS Makes It Simple
    • Don’t skip steps and “get back to them later”. Do everything in sequence and make sure it works.
this presentation
This Presentation
  • This document is not for commercial re-use or distribution without the consent of the author
  • The author does not guarantee this document to be error free
  • Submit questions/corrections/comments to the author:

Charles Pfeiffer,

take a bow
Take a Bow
  • Jeffrey Hunter. Great step-by-step guide on building RAC on Linux with Firewire
  • Wim Coekaerts. Wrote the firewire shared locking modules for Linux
  • Werner Puschitz. Many great step-by-step documents for different Oracle configurations on different versions of Linux
more information
More Information
wrap it up already
Wrap It Up Already!
  • Final Q&A
  • Understanding the IO Under Your Database
    • Complete IO Tuning
    • More detail on transfer size and Linux IO
  • Contact Me
    • 804.901.3992