Implementing Oracle
Download
1 / 62

Implementing Oracle Real Application ClusterBy Charles Pfeiffer888 235-8916CJPfeifferRemoteControlDBA.com - PowerPoint PPT Presentation


  • 459 Views
  • Uploaded on

Implementing Oracle Real Application Cluster By Charles Pfeiffer (888) 235-8916 [email protected] Who Are You?. DBA Experience? RAC Experience? ASM Experience? Linux Server Experience? Firewire Experience? Disk Subsystem IO Tuning Experience? Production v. Dev/Test.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Implementing Oracle Real Application ClusterBy Charles Pfeiffer888 235-8916CJPfeifferRemoteControlDBA.com' - Leo


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Slide1 l.jpg

Implementing Oracle

Real Application Cluster

By Charles Pfeiffer

(888) 235-8916

[email protected]


Who are you l.jpg
Who Are You?

  • DBA Experience?

  • RAC Experience?

  • ASM Experience?

  • Linux Server Experience?

  • Firewire Experience?

  • Disk Subsystem IO Tuning Experience?

    • Production v. Dev/Test


What is this guy talking about l.jpg
What Is This Guy Talking About?

  • RAC

    • Done with Firewire on Linux

    • Something you can do without a huge budget

  • Why Linux?

    • Firewire Share Locking only works on Linux

  • Why Not Linux?

    • Disk IO. More to come…


What is rac l.jpg
What is RAC?

  • Multiple Instances (memory) accessing one Database (disk)

  • Fault-tolerance and Transparency

  • Performance and Scalability

  • Cheaper to grow out instead of up

    • Google Success Story


Rac done cheap l.jpg
RAC Done Cheap

  • Low-end RAC deployments cost around $20,000 for the hardware alone

  • You can build your own test system for 1/10th the cost, using Linux and Firewire

  • This is not a production-ready solution, but it is good for proof-of-concept and testing


Rac shared storage l.jpg
RAC Shared Storage

  • Fiber is the best

    • Costs more, but you get what you pay for

    • Fiber speed is up to 4 Gbps now

    • Supports traditional SCSI protocol

    • Supports up to 127 nodes


Rac shared storage continued l.jpg
RAC Shared Storage (continued)

  • SCSI is about ½ the cost of Fiber

    • Less nodes, but likely enough

    • Less performance, may not be good enough

  • IPSCSI

    • Danger, Danger! Who comes up with this?

    • Better distance, poor performance and reliability

  • Stay away from NAS/NFS

    • Reliability and Performance Standards

    • Who says this is OK?


Rac shared storage continued8 l.jpg
RAC Shared Storage (continued)

  • Firewire

    • 1/30th the cost of SCSI

    • Not supported, not ready for production

    • Works well for testing and proof-of-concept

    • Only works with Linux

      • Special share locking programmed by Wim Coekaerts “Oracle’s Mr. Linux” and his team.


Rac shared storage continued9 l.jpg
RAC Shared Storage (continued)

  • Transfer Speeds (Max spec in MBps)

    • You’ll never get Max. You’re lucky to get half. Expect 1/3rd. More information in another presentation.

    • Serial = 0.014

    • IDE = 16.7, Ultra IDE = 33

    • USB1 = 1.5, USB2 = 60

    • Firewire = 50

    • ATA/100 = 12.5, SATA = 150, Ultra SATA = 187.5

    • FW SCSI = 20, Ultra SCSI = 40, Ultra3 SCSI = 80, Ultra 160 SCSI = 160, Ultra 320 SCSI = 320

    • Gb Fiber = 120, 2Gb Fiber = 240, 4 Gb Fiber = 480


Hardware requirements l.jpg
Hardware Requirements

  • Two Intel-based PCs

    • AMD is OK if you have it, Intel is cheaper

    • Cheap, old, used, whatever.

    • Pre-Pentium is fine, but will be slower

    • VMware is OK. Performance is not as good

      • Ask about running Linux and Oracle on VMware


Hardware requirements continued l.jpg
Hardware Requirements (continued)

  • 512 MB RAM each. Try hard for 1 GB.

  • 10 GB internal Hard Drive each

  • Two network cards each

  • CD-ROM, monitor, keyboard


Hardware requirements continued12 l.jpg
Hardware Requirements (continued)

  • Firewire card each

  • Dual-port Firewire external HDD

    • Do not connect the firewire drive until the firewire modules are installed and configured correctly!

  • Cabling for all


Install linux l.jpg
Install Linux

  • Oracle is tightly intertwined with RedHat

    • Jeffrey Hunter did this with WBEL

    • SuSE and some others work well

  • Typical/default setup for the most part

    • Manual Partitioning

      • /boot = 100 MB

      • Swap = 150% RAM or 200% if < 1 GB RAM

      • / gets the rest

      • Flexibility here, but /usr should not be a mount point


Install linux continued l.jpg
Install Linux (continued)

  • Typical/default setup for the most part (continued)

    • Separate networks for the two NICs

      • Interconnect cannot be the same network as the public

      • Don’t use DHCP

    • Time sync


Install linux continued15 l.jpg
Install Linux (continued)

  • Typical/default setup for the most part (continued)

    • Go ahead, install all packages!

      • If you don’t use/need it you can always disable it

    • up2date, up2date, up2date!!!

      • Oracle has specific package requirements

      • Don’t do this the hard way


Verify packages l.jpg
Verify Packages

  • Advanced Server doesn’t install everything

  • You may have chosen not to install everything

  • Find your requirements

    • There are minimum (and in some cases exact) package version requirements for RAC. Be sure to follow carefully

    • See the links to RAC documentation on the slide titled “More Information”


Firewire modules l.jpg
Firewire Modules

  • http://oss.oracle.com/projects/firewire

    • Choose the correct module for your system

    • Download and install it (rpm –ivh)

  • Add to /etc/modprobe.conf

    options sbp2 exclusive_login=0

    alias scsi_hostadapter sbp2

  • Once this is done on both machines you can power them down, connect the firewire device and restart them


Firewire modules continued l.jpg
Firewire Modules (continued)

  • lspci |grep –i firewire should return one line

  • lsmod |egrep “ohci1394”|sbp2|ieee1394|sd_mod|scsi_mod” should return one line each

  • fdisk

    • /CRS = 300 MB

    • /oradata (usually like to have multiples)

    • /orabackup


Hangcheck timer module l.jpg
Hangcheck-timer Module

  • http://oss.oracle.com/projects/hangcheck-timer/

  • Hangcheck tick setting

    • Time between system health checks

    • Default is 60 (seconds)

    • Oracle recommends 30

    • Maybe 10 or 15 (overhead v. discovery)


Hangcheck timer module continued l.jpg
Hangcheck-timer Module (continued)

  • Hangcheck margin setting

    • Time of tolerated delay before node reset

    • Default and Oracle recommendation is 180 (seconds)

    • Maybe 60 (quick reset v. delayed reset)

  • Add to /etc/modprobe.conf

    options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180


More kernel settings l.jpg
More Kernel Settings

  • Set via sysctl (sysctl –w net.core.)

    • rmem_default, rmem_max, wmem_default and wmem_max = 262144

  • sysctl –w kernel.

    • shmmax=2147483648 (2 GB)

    • sem=“250 32000 100 128”

    • sysrq = 0

    • core_uses_pid = 1


More kernel settings continued l.jpg
More Kernel Settings (continued)

  • sysctl –w fs.

    • file-max = 65536

  • Sysctl –w net.ipv4

    • ip_forward = 0

  • Sysctl –w net.ipv4.conf.default

    • rp_filter = 1


Oracle users and groups l.jpg
Oracle Users and Groups

  • Create the oracle user and the dba and oinstall groups

  • UIDs and GIDs must be the same on both machines

  • Create a software install directory on both machines’ internal drives (e.g. /opt/oracle)


Oracle users and groups continued l.jpg
Oracle Users and Groups (continued)

  • Profile

  • ORACLE_BASE and ORACLE_HOME

  • ORACLE_CRS_HOME

  • ORACLE_SID (each node is unique)

  • PATH=$ORACLE_HOME/bin:$ORACLE_BASE/common/oracle/bin:$PATH:$/usr/bin:/bin:/usr/X11/bin:/usr/local/bin

  • LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/oracm/lib:/lib:/usr/lib:/usr/local/lib

  • CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$ORACLE_HOME/network/jlib

  • THREAD_FLAG=native (use OS Java version)


Introducing cache fusion l.jpg
Introducing Cache Fusion

  • Oracle Parallel Server Disk Writes

    • RAC High Speed Interconnect

  • Performance Gains

    • Outward Scalability

  • Performance Losses

    • Overhead for Cache Fusion


The network l.jpg
The Network

  • Hard-set, do not use auto-negotiate, auto-detect, auto-sense, auto-*, auto-bad

  • Gigabit Hub for the interconnect

    • We use crossover in the test.

    • Don’t mess with Cache Fusion!

    • Not supported


The network continued l.jpg
The Network (continued)

  • Set up a virtual IP address in hosts and the name server

    • Only need name/IP

    • Must be the same network as the public interface

  • Remove the linux name alias on the loopback line and put it on the right line


Firewall configuration l.jpg
Firewall Configuration

  • Allow incoming SSH

    • Key-based access only?

    • Don’t allow root/oracle remotely?

  • Allow incoming TNS

  • Allow all outgoing

  • Allow traffic related to established sessions


Firewall configuration continued l.jpg
Firewall Configuration (continued)

  • Log

    • All dropped?

    • All allowed?

    • Port specific?

  • SE Linux

    • Start with warn mode to see what you want to do

    • More specific than the firewall, but the firewall is usually good enough


Configuring ssh l.jpg
Configuring SSH

  • Used for

    • Remote administration

    • RAC software deployment

  • Create a key pair for root and oracle on each node

    • ssh-keygen –t dsa –N “”

    • Accept the defaults

    • cd ~/.ssh

    • scp id_dsa.pub <other_node>:./.ssh/<this_node>.pubkey

      • NOTE: Password is required for this scp because the key is not in place yet


Configure ssh continued l.jpg
Configure SSH (continued)

  • On the other node

    • cd ~/.ssh

    • cat <other_node>.pubkey >> authorized_keys

  • Make the initial connection to accept the key familiarity question

    • Software installation will error if it is presented with unexpected text


Introducing ocfs l.jpg
Introducing OCFS

  • Oracle Cluster File System

  • Used for RAC on Linux and Windows

  • Eliminates the need for third-party clusters

  • Can store all RAC-related and Database files


Install ocfs l.jpg
Install OCFS

  • http://oss.oracle.com/projects/ocfs-tools

  • http://oss.oracle.com/projects/ocfs

  • OCFS v. ASM

    • ASM can store datafiles, OCFS can store everything

    • OCFS uses DBA-friendly, filesystem-style datafiles

  • Just download and install the RPMs

  • ocfs2console (this is the ocfs2 version)

    • OCFS2/RHEL4 not supported, but it works!

  • Select cluster > configure nodes

    • Add the name/IP of the public interfaces



Configuring ocfs for autostart l.jpg
Configuring OCFS for Autostart

  • Fix the chkconfig bug (OCFS2)

    • Remove the lines between (and including) ‘### Begin Init Info’ and ‘### End Init Info’ in /etc/init.d/o2cb

  • Reregister

    • chkconfig --del o2cb

    • chkconfig --add o2cb

  • Reload

    • /etc/init.d/o2cb offline ocfs2

    • /etc/init.d/o2cb unload

    • /etc/init.d/o2cb configure


Configure ocfs file systems l.jpg
Configure OCFS File Systems

  • Back to ocfs2console

    • Tasks > format

    • 4K Block Size. Get it right now, it can’t be changed.

    • Cluster Size Setting

      • 32K or 64K for OCR.

      • 128K or 256K for OLTP Data Files.

      • 512K or 1M for Data Warehouse Data Files.

      • 1M for backups

  • Reboot both nodes

    • OCFS FS should be mounted on both



Get the rac software l.jpg
Get the RAC Software

  • Oracle Database Server

  • Oracle Companion Software

  • Oracle Cluster (version dependent)

    • Oracle Clusterware

    • Oracle Cluster Ready Services


Install oracle cluster software l.jpg
Install Oracle Cluster Software

  • Unset environment variables

    • ORA_CRS_HOME

    • ORACLE_HOME

  • runInstaller

    • Use the ORA_CRS_HOME

    • Use a cluster name that doesn’t match either SID

    • Mirror the OCR and voting Disks! (10gR2)


Install oracle cluster software continued l.jpg
Install Oracle Cluster Software (continued)

  • Update CSS miscount

    • Timeout before evicting a node

    • Can cause installs to hang

    • Set it higher for slower disks

    • Recommend 60 for most disks, 480 for firewire

    • Edit root.sh before running it on each node


Install oracle cluster software continued41 l.jpg
Install Oracle Cluster Software (continued)

  • Run root.sh

    • Run on each node, one at a time

    • Takes a while to complete

    • Get several warnings. Ignore them.

    • May get a critical alert regarding virtual IP addresses at the end on the last node

      • Run the vipca GUI and set the interface information

  • Verify

    • $ORA_CRS_HOME/bin/olsnodes –n

      • Should see all nodes in the cluster


Install the db server l.jpg
Install the DB Server

  • Unset environment variables

    • ORA_CRS_HOME

    • ORACLE_HOME

  • runInstaller

    • Use ORACLE_HOME

    • After specifying the ORACLE_HOME the next screen should be a clustered install screen

      • Do not proceed if this is not the case. Something is wrong with the cluster. Fix it first!!!


Install the db server continued l.jpg
Install the DB Server(continued)

  • runInstaller (continued)

    • Select all nodes

    • Choose install Software only. Use dbca later, not as part of the initial installation

  • Run root.sh on all nodes, one at a time


Create the listener service l.jpg
Create the Listener Service

  • Run this from only one node in the cluster

  • Netca

    • Choose cluster configuration

    • Select all nodes

    • Use different names for each node’s service


Install companion software l.jpg
Install Companion Software

  • NCOMP is required or you will hit a bug (10.1.0.3 – 10.1.0.4)

  • runInstaller

    • Use ORACLE_HOME

    • Specify Hardware Cluster Installation


Transparent application failover taf l.jpg
Transparent Application Failover (TAF)

  • Available in 10g. Use it!!!

  • Automatically migrate sessions from a failed instance to a working instace

    • Session continues, uninterrupted to the user

  • Requires a simple modification to your configuration

  • Should be selected during DB creation


Create the db l.jpg
Create the DB

  • Run dbca on ONE node

    • Should be presented with a RAC welcome screen

    • Stop! If you don’t get the RAC welcome your cluster is not running or configured properly. Fix the cluster before you create the DB

    • Install on all nodes

    • Choose a “custom” installation unless you have already created a RAC template that uses TAF

    • Add a TAF Service


Configure taf l.jpg
Configure TAF

  • Just update the tnsnames.ora files

    ORCL1 =

    (DESCRIPTION =

    (ADDRESS = (PROTOCOL = TCP)(HOST = vip-rhcl1)(PORT = 1521))

    (ADDRESS = (PROTOCOL = TCP)(HOST = vip-rhcl2)(PORT = 1521))

    (LOAD_BALANCE = yes)

    (CONNECT_DATA = (SERVER = DEDICATED)

    (SERVICE_NAME = orcltest.idevelopment.info)

    (FAILOVER_MODE = (TYPE = SELECT)

    (METHOD = BASIC) (RETRIES = 180) (DELAY = 5)

    ) ) )


Connecting to the db l.jpg
Connecting to the DB

  • You may connect to

    • An instance on either node

      • Transaction aborted when that instance dies

    • The DB service

      • Oracle chooses an instance to connect you to

      • Transaction aborted when that instance dies

    • The TAF DB service

      • Oracle chooses an instance to connect you to

      • Instance failure is transparent


Test taf l.jpg
Test TAF

  • Connect to the TAF Service with SQL

  • Select instance_name from v$instance;

  • Shut that instance down

  • Your session is still connected

  • Select instance_name from v$instance;


Verify your configuration l.jpg
Verify Your Configuration

  • Status of all instances

    srvctl status database –d <db name>

  • Status of a global service

    srvctl status service –d <db name> -s <TAF service name>

  • Status of all cluster applications

    srvctl status nodeapps –n <server name>


Verify your configuration continued l.jpg
Verify Your Configuration (continued)

  • List of configured DBs

    srvctl config database

  • List of services for a DB

    srvctl config service –d <db name>

  • List of applications for a server

    srvctl config nodeapps –n <server name> -a -g -s -l


Verify your configuration continued53 l.jpg
Verify Your Configuration (continued)

  • DB’s perspective of all running instances

    Select status, database_status dbstat,

    active_state state, host_name host from

    gv$instance;

    STATUS DBSTAT STATE HOST

    ----------- ------------ ------------- --------

    OPEN ACTIVE NORMAL rhcl1

    OPEN ACTIVE NORMAL rhcl2


Shutdown the cluster l.jpg
Shutdown the Cluster

  • Stop an instance

    srvctl stop instance –d <db name> -i <instance name>

  • Stop all instances

    srvctl stop database –d <db name>

  • Stop the server applications on a server

    srvctl stop nodeapps –n <server name>


Start the cluster l.jpg
Start the Cluster

  • Start all instances

    srvctl start database –d <db name>

  • Start an instance

    srvctl start instance –d <db name> -i <instance name>

  • Start the server applications on a server

    srvctl start nodeapps –n <server name>


Linux disk io why do i keep bringing it up l.jpg
Linux Disk IO: Why Do I Keep Bringing It Up?

  • The transfer size penalty

    • Ext3 maxes out at 512 KB, Ext2 at 256 KB

    • UFS/VxFS/NTFS all do 1 MB

    • Latest version of NTFS is actually prepared to do 2 MB

  • But what does it all mean?

    • The IOps threshold is creeping up fast

    • Be careful of what you put in production

  • People Say Linux Performs well

    • What are you using it for?

    • Corvettes v. Dump Trucks



Summary l.jpg
Summary

  • You can do RAC cheap

    • Use Linux and Firewire

    • Not meant for production

  • You can do RAC easy

    • More effort is spent in prepping the host server than in actually configuring RAC

    • Don’t relax your efforts there. It is crucial that the host servers are configured correctly

    • OCFS Makes It Simple

    • Don’t skip steps and “get back to them later”. Do everything in sequence and make sure it works.


This presentation l.jpg
This Presentation

  • This document is not for commercial re-use or distribution without the consent of the author

  • The author does not guarantee this document to be error free

  • Submit questions/corrections/comments to the author:

    Charles Pfeiffer, [email protected]


Take a bow l.jpg
Take a Bow

  • Jeffrey Hunter. Great step-by-step guide on building RAC on Linux with Firewire

  • Wim Coekaerts. Wrote the firewire shared locking modules for Linux

  • Werner Puschitz. Many great step-by-step documents for different Oracle configurations on different versions of Linux


More information l.jpg
More Information

  • http://download-west.oracle.com/docs/pdf/B10766_08.pdf

  • http://download-west.oracle.com/docs/cd/B14117_01/rac.101/b10765.pdf

  • http://download-west.oracle.com/docs/cd/B14117_01/rac.101/b10768.pdf

  • http://oss.oracle.com/projects/

  • http://www.oracle.com/technology/oramag/webcolumns/2002/opinion/coekaerts_linux01.html

  • http://www.oracle.com/technology/pub/articles/hunter_rac10gr2.html

  • http://www.puschitz.com/

  • http://www.cisecurity.org/


Wrap it up already l.jpg
Wrap It Up Already!

  • Final Q&A

  • Understanding the IO Under Your Database

    • Complete IO Tuning

    • More detail on transfer size and Linux IO

  • Contact Me

    • 804.901.3992

    • [email protected]


ad