1 / 106

Inside RAC

Inside RAC. Julian Dyke Independent Consultant. Web Version. juliandyke.com. Agenda. Introduction to RAC Memory Structures The Buffer Cache Global Cache Services The Library Cache Global Enqueue Services. Introduction to RAC. Node 1. Node 2. Instance 1. Instance 2. Interconnect.

ina
Download Presentation

Inside RAC

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. InsideRAC Julian Dyke Independent Consultant Web Version juliandyke.com

  2. Agenda • Introduction to RAC • Memory Structures • The Buffer Cache • Global Cache Services • The Library Cache • Global Enqueue Services

  3. IntroductiontoRAC

  4. Node 1 Node 2 Instance 1 Instance 2 Interconnect LocalDisk Shared Storage LocalDisk What is RAC? • Multiple instances running on separate servers (nodes) • Single database on shared storage accessible to all nodes • Instances exchange information over an interconnect network

  5. Architecture Public Network Private Network(Interconnect) Instance 1 Instance 2 Node 1 Node 2 Storage Network SharedStorage

  6. Some Definitions • Resource • Object to which access must be controlled at instance level • Enqueue • Memory structure that serializes access to a resource • Global Resources • Object to which access must be controlled at cluster level • Global Enqueue • Locks and enqueues which need to be consistent between all instances

  7. Some Definitions... • Global Resource Directory (GRD) • Records current state and owner of each resource • Contains convert and write queues • Distributed across all instances in cluster • Maintained by GCS and GES • Global Cache Services (GCS) • Implements cache coherency for database • Coordinates access to database blocks for instances • Global Enqueue Services (GES) • Controls access to other resources (locks) including library cache and dictionary cache • Performs deadlock detection

  8. Instance 1 Instance 2 SGA UGA UGA SGA ARCH CKPT PMON SMON LMD LMS DIAG LCK LCK LGWR LMON LMD DIAG CKPT ARCH LGWR PMON DBWR SMON DBWR LMON LMS Background Processes DatafilesControlfiles Redo Logs Redo Logs

  9. Background Processes • LMSn • Global Cache Service Process • Manage requests for data access across cluster • Up to 20 in Oracle 10.1 • LMS0-LMS9 LMSa-LMSj • Up to 36 in Oracle 10.2 • LMS0-LMS9 LMSa-LMSz • In Oracle 10.1 and above, number of GCS server processes can be configured using gcs_server_processes parameter • Default value is 1 (single CPU system)

  10. Background Processes • LCK0 • Instance Enqueue Process • Manages • instance resource requests • cross-instance call operations • Assists LMS processes • Formerly known as lock process • In 9.0.1 and below, number of lock processes may be configurable using _gc_lck_procs parameter

  11. Background Processes • LMD0 • Global Enqueue Service Daemon • Manages requests for global enqueues • Updates status of enqueues when granted to / revoked from an instance • One LMD0 process per instance • In 8.1.7 and below number of lock daemons may be configurable using _lm_dlmd_processes parameter

  12. Background Processes • LMON • Global Enqueue Service Monitor • One LMON process per instance • Monitors cluster to maintain global enqueues and resources • Manages • instance and process expirations • recovery processing for cluster enqueues

  13. Background Processes • DIAG - Diagnosibility Process • Collects diagnostic data in the event of a failure • Creates subdirectories in BACKGROUND_DUMP_DEST directory • In Oracle 9.0.1 and above can be disabled using _diag_daemon parameter • Do not try this on a production system

  14. Fixed Tables • Memory structures externalized in X$ tables • Instance specific • Underlying structures for dynamic performance views • Can contain • Structures accessed directly from executable • X$KSLLD => V$LATCHNAME • X$KSUSD => V$STATNAME • Structures accessed directly from SGA • X$KSUSE => V$SESSION • X$KSUPR => V$PROCESS • Executable and/or SGA structures joined in PGA • X$KQLFXPL => V$SQL_PLAN • X$KGLOB => V$SQL, V$SQL_AREA

  15. Dynamic Performance Views • In a RAC environment each V$ view has an equivalent GV$ view • GV$ view includesINST_ID column. For example • V$SGA • GV$SGA • In Oracle 9.2 and below PARALLEL_MIN_SERVERSmust be >= number of hosts to use GV$ views • In Oracle 10.1 and above PZnn background processes are used to return data on remote hosts e.g. PZ99

  16. CATCLUST.SQL • Some additional views/synonyms are created for RAC databases using $ORACLE_HOME/rdbms/admin/catclust.sql

  17. ORADEBUG • ORADEBUG includes LKDEBUG • Must be run by user with SYSDBA privilege SQL> ORADEBUG LKDEBUG HELP Usage:lkdebug [options] -l [r|p] <enqueue pointer> Enqueue Object -r <resource pointer> Resource Object -b <gcs shadow pointer> GCS shadow Object -p <process id> client pid -P <process pointer> Process Object -O <i1> <i2> <types> Oracle Format resname -a <res/lock/proc> all <res/lock/proc> pointer -A <res/lock/proc> all <res/lock/proc> contexts -a <res> [<type>] all <res> pointers by an optional type -a convlock all converting enqueue (pointers) -A convlock all converting enqueue contexts -a convres all res ptr with converting enqueues -A convres all res contexts with converting enqueues

  18. ORADEBUG • Continued... -a name list all resource names -a hashcount list all resource hash bucket counts -t Traffic controller info -s summary of all enqueue types -k GES SGA summary info -m pkey <objectno> request for remastering this object at current instance -m dpkey <objectno> request for dissolving remastering of this object at current instance

  19. MemoryStructures

  20. STOP Memory Areas 00000000 • An Oracle process includesthe following memory areas • Executable • SGA • Shared Libraries • PGA/Session Heap • Stack Executable 20000000 SGA SharedLibraries 40000000 PGASession Heap Stack FFFFFFFF

  21. Shared Memory • Shared memory areas can be dumped to trace file using $ sqlplus /nolog SQL> CONNECT SYS/<password> AS SYSDBAConnected SQL> ORADEBUG SETMYPIDStatement processed SQL> ORADEBUG IPCInformation written to trace file ORADEBUG SETMYPIDORADEBUG IPC

  22. STOP ORADEBUG IPC - Example Area #0 `Fixed Size' containing Subareas 0-0 Total size 0000000000129968 Minimum Subarea size 00000000 Area Subarea Shmid Stable Addr Actual Addr 0 0 65537 0x00000020000000 0x00000020000000 Subarea size Segment size000000000012a000 0000000010800000 Area #1 `Variable Size' containing Subareas 2-2 Total size 0000000010000000 Minimum Subarea size 00400000 Area Subarea Shmid Stable Addr Actual Addr 1 2 65537 0x00000020400000 0x00000020400000 Subarea size Segment size0000000010000000 0000000010800000 Area #2 `Redo Buffers' containing Subareas 1-1 Total size 00000000002d6000 Minimum Subarea size 00000000 Area Subarea Shmid Stable Addr Actual Add 2 1 65537 0x0000002012a000 0x0000002012a000 Subarea size Segment size00000000002d6000 0000000010800000 Area #3 `skgm overhead' containing Subareas 3-3 Total size 0000000000001000 Minimum Subarea size 00000000 Area Subarea Shmid Stable Addr Actual Addr 3 3 65537 0x00000030400000 0x00000030400000 Subarea size Segment size0000000000001000 0000000010800000 Area #0 `Fixed Size' containing Subareas 0-0 Total size 0000000000129968 Minimum Subarea size 00000000 Area Subarea Shmid Stable Addr Actual Addr 0 0 65537 0x00000020000000 0x00000020000000 Subarea size Segment size 000000000012a000 0000000010800000 Area #1 `Variable Size' containing Subareas 2-2 Total size 0000000010000000 Minimum Subarea size 00400000 Area Subarea Shmid Stable Addr Actual Addr 1 2 65537 0x00000020400000 0x00000020400000 Subarea size Segment size 0000000010000000 0000000010800000 Area #2 `Redo Buffers' containing Subareas 1-1 Total size 00000000002d6000 Minimum Subarea size 00000000 Area Subarea Shmid Stable Addr Actual Add 2 1 65537 0x0000002012a000 0x0000002012a000 Subarea size Segment size 00000000002d6000 0000000010800000 Area #3 `skgm overhead' containing Subareas 3-3 Total size 0000000000001000 Minimum Subarea size 00000000 Area Subarea Shmid Stable Addr Actual Addr 3 3 65537 0x00000030400000 0x00000030400000 Subarea size Segment size 0000000000001000 0000000010800000 Fixed Area Variable Area Redo Buffers OS Specific

  23. Shared Global Area • Contains • Fixed SGA • Buffer Pool • Shared Pool Oracle 7.0 and above • Large Pool Oracle 8.0 and above • Java Pool Oracle 8.1.5 and above • Streams Pool Oracle 10.1 and above • Redo buffers • Buffer Pool includes • Default cache • Keep and Recycle cache Oracle 8.0 and above • 2K, 4K, 8K, 16K and 32K cache Oracle 9.0.1 and above

  24. V$SGAINFO • Summarizes SGA pools SELECT * FROM v$sgainfo; NAME BYTES RESIZEABLE-------------------------------- ---------- ----------Fixed SGA Size 1218920 NoRedo Buffers 2973696 NoBuffer Cache Size 176160768 YesShared Pool Size 83886080 YesLarge Pool Size 4194304 YesJava Pool Size 4194304 YesStreams Pool Size 0 YesGranule Size 4194304 NoMaximum SGA Size 272629760 NoStartup overhead in Shared Pool 46137344 NoFree SGA Memory Available 0

  25. Granules • Introduced in Oracle 9.0.1 • SGA divided into granules • In Oracle 9.2 Unix granule size dependent on SGA_MAX_SIZE • 4 mb SGA_MAX_SIZE <= 128 mb • 16 mb SGA_MAX_SIZE > 128 mb • If SGA_MAX_SIZE not set explicitly then defaults to sum of individual pool parameters • SGA_MAX_SIZE cannot be dynamically modified

  26. Granules • In Oracle 10.1 and above • SGA_MAX_SIZE dependent on SGA_TARGET • Granule size dependent on SGA_MAX_SIZE • 4 mb SGA_MAX_SIZE <= 256 mb?? • 16 mb SGA_MAX_SIZE > 256 mb ?? • Granule size can be controlled using _ksmg_granule_size unsupported parameter

  27. X$KSMGE • Introduced in Oracle 9.2 • Describes individual granules Granule ID 0 Free1 Shared Pool2 Large Pool3 Java Pool6 Buffer Pool INVALID (Free)ALLOCATED Granule Address Granule Size Next granule ID Previous granule ID

  28. STOP Granules 4M Fixed SGA + Redo Buffers 4M SGA_TARGET = 260M SGA_MAX_SIZE = 260M GRANULE SIZE = 4M Buffer Pool 172M 256M Large Pool 4M Java Pool 4M This is an example of an SGA mapped using X$KSMGE Shared Pool 76M

  29. V$SGASTAT • Enhanced in Oracle 10.1 and above • In Oracle 10.2 describes around 700 memory areas

  30. V$SGASTAT • RAC-specific areas include

  31. V$SGASTAT • Significant RAC areas in Oracle 10.2 • In Oracle 9.2 all five structures were stored in segmented arrays

  32. 2AC00000 2B800000 2BC00000 2C000000 2C400000 2C800000 2CC00000 2D000000 2D400000 2D800000 2DC00000 2E000000 2E400000 2E800000 2EF00000 2F000000 2F400000 2F800000 2FC00000 30000000 STOP Permanent Areas • Allocated at instance startup • Contain structures such as • processes • sessions • segmented arrays Heap SELECT ksmchptr, ksmchsizFROM x$ksmspWHERE ksmchcls = 'perm'; PermanentArea

  33. X$KSMSP • Externalises chunks in the shared pool Description of chunk type Address of chunk Size including header Class • Perm • Recreate • Freeable • Free • R-Free • R-Freeable Type (recreate only) • 0x18 - KGL Handle • 0x24 - Fixed Allocation • 0x74 - KQR PO • 0x80 - KQR SO • 0xFFF - Extended header Address of parent

  34. X$KSMSP • Some RAC components are stored in the shared pool heap SELECT ksmchcom, SUM(ksmchsiz), COUNT(*)FROM x$ksmspGROUP BY ksmchcom;

  35. Segmented Arrays • Used to store arrays of objects • Array too large to fit in granule • Array may grow dynamically • Objects using segmented arrays include • enqueues (locks) • enqueue resources (resources) • transactions • transaction branches • Segmented arrays managed recursively by segmented array • Segmented arrays externalised in X$KSMDD

  36. SegmentedArrays Call Enqueues EnqueueResources STOP Segmented Arrays SegmentedArray SegmentedArray Header SegmentedArray SegmentedArray Header

  37. X$KSMDD • Externalises segmented array headers Name of array Number of chunks Address of parent heap

  38. Segmented Arrays • In Oracle 10.2 there are three RAC-specific segmented arrays: • GCS Resources • GCS Shadows • GES Enqueues • In previous releases there were five segmented arrays: • GCS Resources • GCS Shadows • GES Resources • GES Shadows • GES Messages

  39. Chunks STOP Heaps • A heap consists of one or more extents • Each heap extent occupies a single granule • Each extent contains one or more chunks • Each heap has a header containing • list of used chunks • list of free chunks Extent 1 Extent 2 Extent 0 Heap Header Free List

  40. X$KSMHP • Externalises chunks in the heap • Can only be accessed using KSMCHDS e.g. SELECT * FROM x$ksmhpWHERE ksmchds = HEXTORAW ('2CA54040');

  41. The BufferCache

  42. STOP Buffer Headers • Each buffer has a buffer header • Buffer headers are stored in same granule as buffers • Buffer headers include • Replacement list • Hash list • Pointer to buffer • In RAC only pointer to Lock Element Buffer Headers Buffers Granule

  43. X$BH

  44. 45 33 42 11 52 72 71 66 34 66 42 49 92 87 45 45 52 71 71 42 72 33 33 87 42 33 11 42 52 71 11 11 45 42 71 42 52 92 34 34 72 45 92 11 72 34 2 1 1 1 1 2 1 1 0 1 2 0 0 3 4 2 1 1 2 1 0 2 0 4 1 1 1 2 2 4 4 0 1 4 1 2 0 1 0 1 2 4 0 3 1 2 Read Block 87 Read Block 34 Read Block 33 Read Block 42 Read Block 11 Update touch count for block 42 Update buffer contents Get first available buffer from cold end Update buffer contents Insert buffer at head of cold end Get first available buffer from cold end Insert buffer at head of cold end Insert buffer at head of coldend Update buffer contents Move block 71 to head of hot end Set touch count on block 71 to zero Get first available bufferfrom cold end Update buffer contents Set touch counton block 42 to zero Insert buffer at head of coldend Move block 42 to headof hot end Update touch countfor block 34 Get first available bufferfrom cold end STOP Single-Block Reads Block Number Head of Hot End Head of Cold End Touch Count Read Block 42

  45. 1 3 6 2 8 2 3 4 7 1 2 1 2 1 4 1 2 1 6 1 2 3 1 5 5 5 3 7 8 2 1 3 4 7 6 5 6 5 4 2 3 1 5 5 5 6 7 6 7 6 5 8 Read Block 7 Read Block 5 Read Block 2 Read Block 4 Read Block 8 Read Block 6 Read Block 1 Read Block 3 Move block 7 to cold end Move block 5 to cold end Move block 8 to cold end Get first four available buffers from cold end Read next four blocks into buffers Insert buffers at head of cold end Move block 1 to cold end Move block 3 to cold end Read next four blocks into buffers Insert buffers at head of cold end Move block 4 to cold end Get next four available buffers from cold end Move block 6 to cold end Move block 2 to cold end STOP Multi-Block Reads DB_FILE_MULTIBLOCK_READ_COUNT = 4 Head of Hot End Head of Cold End

  46. GlobalCacheServices

  47. 3 2 1 4 STOP Read with No Transfer Request shared resource N S ResourceMaster Instance 2 Instance 3 Request granted Read request Block returned Instance 4 Instance 1 Instance 2 requests current read on block 1318 1318

  48. 4 3 2 1 STOP Read to Write Transfer Transfer block to Instance 1 for exclusiveaccess ResourceMaster N S N Block and resource status 1318 Instance 2 Instance 3 Request exclusiveresource N X 1320 Resource status Instance 1 Instance 4 Instance 1 requests exclusive read on block 1318

  49. 4 3 2 1 STOP Write to Write Transfer Request block in exclusive mode Transfer block to Instance 4 in exclusive mode ResourceMaster N S N 1318 Instance 2 Instance 3 Resource status N X N N X 1320 1320 1323 Block and resource status Instance 1 Instance 4 Instance 4 requests exclusive read on block 1318 Note that Instance 1 will create a past image (PI) of the dirty block

  50. Past Images • When an instance passes a dirty block to another instance it • Flushes redo buffer to redo log • Retains past image (PI) of block in buffer cache • PI is retained until another instance writes block to disk • Used to reduce recovery times • Recorded in V$BH.STATUS as PI • Based on X$BH.STATE (value 8 in Oracle 10.2)

More Related