2006 HP Dutch World OpenVMS - RMS

1. 2006 HP Dutch WorldOpenVMS - RMS Hein van den Heuvel

2. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 2 Topics First presented at 2006 OpenVMS bootcamp Nashua NH session A915 - Advanced RMS Tuning Credentials 1: Indexed File Internals 2: Buffers and Locks 3: Common Performance Problems 4: Large Files HP Technical Symposium, Houston 2006. 5: FDL Magic Questions? Obligatory introObligatory intro

3. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 3 My RMS Credentials 25+ years of working with RMS 70�s Programmer (Netherlands) 80�s European Support (France) 90�s VMS Engineer (Nashua, NH) 2000: Hobbyist, fire-drills HP ITRC Forums COMP.OS.VMS Decus, CETS, HP World,� 2005: Independent consultant Was anyone in the audience NOT aware of the existence of the OpenVMS ITRC Forum> Hein knows RMSWas anyone in the audience NOT aware of the existence of the OpenVMS ITRC Forum> Hein knows RMS

4. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 4 What are your RMS Credentials? If this was an Oracle application� (and if you don�t pay attention it will be ? ) Training & Certification Developers, DBAs, Consultants Tools (Enterprise Manager, Toad, Statspack,�) How much RMS training did YOU have? Experiment! Read The Fine Manuals Cobol Users Guide Guide to OpenVMS File Applications OpenVMS Record Management Utilities Reference Manual OpenVMS Utility Routines Manual

5. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 5 Indexed File Internals

6. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 6 before Bucket Split, adding �4�

7. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 7 Anatomy of an Indexed Fileafter Bucket Split, adding �4�

8. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 8 Buffers and Locks What about Buffers Local Buffers Global Buffers What about locks Bucket Locks Record Locks File Lock Obligatory introObligatory intro

9. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 9 What about buffers Chunks of RMS maintained memory Used to read a bucket into (or to write from) RMS was very frugal: Pre VMS 5.4 (vax): just 2 local buffers! RMS is still conservative: Default: deepest index at open time + 2 Typical: 5 Gotcha: freshly created file DCL: 2 (P1 space concerns)

10. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 10 How many buffers needed? Only 1 buffer for Indexed Sequential GETs Read-Ahead only implemented for Sequential files Keyed GETs need N : Depth of index, 1 for each level 1 ? for SIDR data bucket if alternate key used 1 : DATA bucket 1 ? Real data bucket if alternate key use & RRV found Default allocation : Depth + 2

11. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 11 How many buffers needed? (2) Many buffers for PUTs and UPDATEs Sum of: a data buckets for each key buffers to perform splits. index trees depths for each key Typical 3 key file needs 20? 30? Specially important with repeated operations Keep Root and Top index in memory for each Key Buffers with data less likely to be re-used, but need them to avoid flushing out Index just before re-use.

12. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 12 Local Buffer Count? Set on $CONNECT time (HLL OPEN call) Through RAB max = 127 RAB$B_MBF Cobol: FILE CONTROL - RESERVE nnn AREAS Through DCL max = 255 Defaults at process and system level $SET RMS [/SYSTEM] [/INDEXED] /BUF=� applies to every file opened! Specify up to 32K buffers through XABitm XAB$_MULTIBUFFER_COUNT Gotcha� Linear search

13. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 13 Local Buffer Addresses Space Can be significant Virtual Memory usage Open_files * Avg_buffers * Avg_bucketsize 150 * 32 * 16 = 80,000 pagelettes = 40 MB Exec mode, Protected against User Write Not just �your� stuff. Buffers hold potentially shared bucket data May be User Read for locate mode = RAB$V_LOC, rarely applicable C, Cobol & Sort RTLs do not use RMS Record IO for unshared file access to avoid CMEXEC

14. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 14 Local Buffer Addresses Space (2) Initially from P1 (with RTLs and Stack) Default SYSGEN param IMGIOCNT Consider raising from Hundreds to Thousands Per IMAGE selectable through LINK OPTION IOSEGMENT=100000,[NO]P0 RMS uses $EXPREG to grow in P0 as needed If NOP0 and P1 space under allocated ? %RMS-F-DME, dynamic memory exhausted

15. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 15 Local Buffer Addresses Space (2) $ANAL/SYSTEM SDA> SHOW PROC test/RMS=BDBSUM Process index: 0003 Name: TEST Extended PID: 000F4903 ------------------------------------------- BDB/GBPB Summary Address USERS SIZE VBN BLB_PTR ADDR ------- ----- ---- ---- --- ---- --- 7B085290 0 00002800 00000000 00000000 7B07CC00 7B085380 0 00002800 00000000 00000000 0008A000 7B085600 0 00002800 00000000 00000000 0008C800 7B085670 0 00002800 00000000 00000000 0008F000 7B085720 0 00002800 00000000 00000000 00091800 5. BDBs were processed

16. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 16 Local Buffer Linear Search Test 1000 Duplicate key $PUTs adding to 1400+ bucket chain. DS20, 7.2-1, 500Mhz, SWXCR, VIOC

17. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 17 What about GLOBAL Buffers? Share buffers by all accessors on the node Efficient in clusters no change broadcasts) Can be used to save Memory. Sure, they use memory. Many users, each using fewer local buffers 200 + 100 * 2 is less than 100 * 5

18. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 18 What about GLOBAL Buffers (2) Great to cache (all) index buckets Primary and Alternate indexes alike Great for read and write accessed files PUTs first need to read, to know where to write. Can write from global cache, avoiding next read �Deferred Write� option forces a local buffer copy

19. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 19 Global Buffer Address Space Created with $CRMPSC + Expreg in P0 Can take up serious Virtual Address space 1 file with 5000 buffer @ 16 blocks = 40 MB Uses GBLPAGES and GBLSECTIONS VMS V8.3 : Optional P2 Space usage New FDL GLBUFF_CNT_V83 0 GLBUFF_FLAGS_V83 none Old FDL GLOBAL_BUFFER_COUNT 0

20. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 20 Global Buffer Address Space (2) Deleted with $DELTVA Gotcha: OpenVMS really only returns address space at the very end of P0. Address �laddering� when mixing (de-)allocations Can result in VASFUL errors Keep files open, or Close files in LIFO order. Pre-allocate (and release) all VM needed (Malloc) V8.2 : Virtual Address (VA) space management Attempts to re-use prior used VA reduces the possibility of a VASFUL

21. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 21 Global Buffer are Not used for� Deferred write buffers Write intend buffers, if not already global File operations: key & area descriptors Garbage collection (Fast Delete, RU Delete) Process Permanent (DCL) files IOSEGMENT=NOP0 images Clearing FAB$W_GBC before $CONNECT Good idea for �bad citizens�, reading sequentially

22. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 22 Global Buffer Gotchas (2) Needs LOCKs to work : shared access only Up to V7.2 sorted Sequential Scan was used Protected by exclusive, local, lock. Limits max useful count to 500 � 1000 Desirable buckets were often at the end HASHED lookups in VMS 7.2 (patch kit for 7.1) SHOW PROC/RMS=GBHSH

23. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 23 How many GLOBAL Buffers �None� is the only wrong answer. Do NOT postpone to get it perfect, Try some! RMS Limit is (was) 32K, Practical 200 - 2000 ? 42? One for each user? Cache (top of) index and then some? Goal is 80 - 99% hit rate. But really the goal is an acceptable IO rate. 99% hit rate @ 5000 accesses/second ? 50 IO/second 90% hit rate @ 50 accesses/second ? 5 IO/second

24. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 24 What about locks Increasingly Locks are the critical resource for shared files Disk IO used to be the bottleneck Faster disks : 15Krpm Smarter controllers: Striping solves load balancing (S.A.M.E.) Write back Caching speeds up writes Faster connections : Ultra-3, 4gb Fibre XFC (or VIOC) caches resolves reads

25. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 25 RMS File Lock Masters other RMS locks for the file Name: RMS$<binary-file-id><DEVLOCKNAM> SDA> SHOW PROC/LOCK Flags: CONVERT SYNCSTS SYSTEM QUECVT Lock id: 700001C4 SUBLCKs: 3 BLKAST: 82C75F10 Granted at EX Resource: 000C818D 24534D52 RMS$.... Length 26 33524553 55020100 ...USER3 Exec. mode 00202020 20202020 . System 00000000 00000000 ........

26. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 26 Group Grant Mode problem Locks under the RMS File Resource were not properly ordered, requiring a full scan. Very expensive when file is opened hundreds of times All other lock activity blocked: High MPSYNC for traditional SMP systems High KERNEL mode with dedicated lock manager Fixed in 8.3 (Thanks Greg! ?) Patch kit on request

27. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 27 Group Grant Mode Similar load T4 picture before & after

28. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 28 Bucket Locks Name: 32-bit, 4 byte, VBN of Bucket Transparent to application. Not held (without BLKAST) during user mode Lock data: Flags: VALBLK CONVERT SYSTEM NODLCKW NODLCKB Par. id: 700001C4 SUBLCKs: 0 BLKAST: 00000000 Granted at NL Resource: 00000000 0000001F ........ Length 04 00000000 00000000 ........ Exec. mode 00000000 00000000 ........ System 00000000 00000000 ........

29. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 29 Bucket Locks Modes EX while bucket in active use Scan comparing key value, Inserting new data PW for Deferred write buckets �ping� process with dirty data to write it out NL for inactive buckets, caching the lock Basically one per buffer CR New for 7.3, Global Buffers Only VERY important new feature

30. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 30 Record Locks Name: 8 byte Record RFA (example 10,2) Under application control Held during user mode SDA> SHOW PROC/LOCK Flags: NOQUEUE SYNCSTS SYSTEM Par. id: 700001C4 SUBLCKs: 0 Granted at EX Resource: 0000000A 00000002 ........ Length 08 00000000 00000000 ........ Exec. mode 00000000 00000000 ........ System 00000000 00000000 ........

31. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 31 Record Lock Control Default: Implicit Locking/Unlocking If FAB$B_FAC=UPD|DEL lock on $GET/$FIND Unlock on $UPDATE/$DELETE Unlock current, lock next on $GET/$FIND Explicit Unlocking RAB$L_ROP=ULK : Do not auto Unlock Wait for user $FREE (all records) or $UNLOCK call (specific record)

32. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 32 Record Lock Control RAB$L_ROP Options NLK do not keep record lock in user mode Customers often expect this to mean No Lock DOES lock and unlock record while in RMS and sets status accordingly (SUC/RLK) RRL Read Regardless of Lock DOES get a lock and keeps it if possible Set status to SUC/OK_RRL

33. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 33 RRL + NLK Record Lock Options Customers expect no lock activity at all However, RMS still gets and releases! New V7.3 : No Query Lock (NQL) Truly NO record lock activity back ported to V72-1H1, V7.2-2 Access through RAB$W_ROP2 (and FAB$B_SHR) Also has new Deadlock control bits DCL Option to convert RRL+NLK ? NQL $ SET RMS_DEFAULT/QUERY_LOCK=DISABLE[/SYSTEM] Great for COBOL applications

34. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 34 Bucket Locks are critical 2 - 20 times more busy than record locks 10 - 1000 times more busy than file lock Often Exclusive Hard to tune Bigger bucket do not help (much) with SEQ reads N+1 bucket locks for N records per bucket Lower Index Depth helps with KEYed reads Best solution to date: Global buffer ? CR lock Alternative: Consider READ ONLY data copy

35. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 35 Locking Tools SET FILE/STAT [/SHARE] �. MONITOR RMS/ITEM=LOCK/FILE=� SDA> SHOW PROC/RMS=(FSB,BLBSUM,RLB) RMS_STATS OpenVMS Freeware SDA RMS EXTENSION SDA> RMS LOAD SDA> RMS START TRACE/BUF=5000 SDA> RMS SHOW TRACE/SUM

36. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 36 (edited, NQL) RMS_Stats output 0.41 IO/Ops. Rec Ops: 103,935,333 Reads 42,404,167 Cache trials 180,053,203 hit 137,649,166 Locks: Enqueue Dequeue Convert ----------------+----------+-----------+---------- Shared file: 2,148,780 2,148,730 2,148,758 Local buffer: 42,396,199 44,543,519 317,708,333 Global buffer: 0 0 0 Append locks: 0 0 0 Global section: 0 0 0 Data record: 5,494,201 5,494,201 0 Open: 2,148,780 Close:2,148,730 Xqpqio: 2148780 Find seq: 0 key: 2,652,061 rfa: 28 Getseq:90,669,610 key: 10,593,160 rfa: 115

37. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 37 (edited) SDA> RMS SHOW SUM TotEntries Bucket-VBN TotCount Filename * 1181 DRUG.IDX;1 9619 356 1 36 9709 32 11078 26 9757 24 9763 24 9649 22 1839 22 4623 20 $MCR FREEWARE:RMS-SHOW-KEYS DRUG.IDX # LVL DB IB DTP TKS POS NUL Root-VBN Data-VBN Name -- -- -- -- --- --- --- --- --------- --------- DRUG.IDX 0 2 6 6 STG 49 0 9619 3 DRUG KEY 1

38. QUESTIONS ? Hein van den Heuvel

39. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 39 Common Performance Problems Duplicate Key Chains Empty Bucket Chains Index Depth Deferred Write Obligatory introObligatory intro

40. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 40 Duplicate Chains,Overview Duplicate alternate keys stored in: Secondary Index Data Records (SIDR) Single Key value plus Array of: 7 byte RRV pointers which are: 4 Byte VBN 2 Byte ID 1 Byte Flag If SIDR larger than Bucket Size then start fresh SIDR, with same key, in a new Bucket. These form a �chain�.

41. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 41 Duplicate Chains,Overview (2) RMS maintains arrival order Deliberate documented behavior Applications count on. Last record added with key value �Request� should become last entry in last SIDR with that key �Request� New inserts go at end of SIDR chain 500 � 5000 RRVs / Bucket (70 /block) potentially requires many read IOs Millions of dups ? Thousands of IOs Write intend: No Global Buffers used

42. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 42 Duplicate Chains,Picture.

43. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 43 Duplicate Chains,Detection methods. Often seen on SHORT Alternate Keys Equally possible on primary keys, but those are more often then not made unique or almost unique. High READ IO during $PUTs. High XFC / IO-controller Cache Hit Rate. ANALYZE/RMS/FDL <your-file.idx> RMS_TUNE_CHECK tool on VMS Freeware Used to be called �sidr�. Suggestion: check .txt help file, and try - arguments

44. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 44 Duplicate Chains,Detection example 1 of 2. ANALYZE/RMS/FDL/OUT=SYS$OUTPUT X.IDX Bucket size was 16 Therefore� 1968/16 = 123 SIDR buckets. Just 9 index entries: 112 buckets with duplicates! And� maximum SIDR size = 1168 entries (16*512 � 15 � compressed-key-size ) / 7

45. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 45 Duplicate Chains,Detection example 2 of 2. RMS_TUNE_CHECK Y.IDX New records inserted with key value 0000000000 cause 4000+ read IOs and 1 Write. Most new records have that key value� how else did those 1.7 millions duplicates get there?

46. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 46 Duplicate Chains,Solution 1: Drop Key Does the application really use that key? Unfortunately no RMS stats to help Source code analyzer? How often is it needed? Consider SORTING just before run Consider Filtering data read by primary key. Alternate keys reads: 1+ IO per Record Primary key read: 10+ Records per Bucket

47. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 47 Duplicate Chains,Solution 2: Apply Null Key Value Sample C code Set �the million dollar bit� : xab$v_nul = 1 Zero as �null value� byte : xab$b_nul = 48 Sample FDL sets �space� as null key value: NULL_KEY�����yes NULL_VALUE���32 NO index entry IF each key byte = �null value� byte Gotcha: Can no longer find those by alternate key What is the business value of a query: �find the first of the 2 million records without an appointment�. NO application code change needed. Just re-convert! Single byte Restricts use. May need to re-define application data ( �NA�,�unknown�)

48. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 48 Duplicate Chains,Solution 3: Increase Bucket Size Use AREAS to specify SIDR bucket size IO Size not so important @50MB/sec The number IOs matter more @5ms/IO RMS Maximum bucket size is 63 If current bucket size is 12 (or less) then read IO count can be divided by 5 (or more). Makes 100,000+ duplicates manageable Requires CONVERT after FDL edit.

49. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 49 Duplicate Chains,Solution 4: De-duplicate some Add (ordered) key segment Goal is NOT to make each key unique, Avoid buckets full of duplicates. 100+ duplicates is just fine (even efficient!) No new data in record. Change key definition to add segment (XAB$W_POSx, XAB$B_SIZx) Mew segment will change the sorting order. Ok? Sample: change STATE to STATE + ZIP(9) Each byte added divides count by factor 10 � 256 Ascii numbers: 10, text: 50+ binary data: 200+

50. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 50 Deleted key ranges Deleted records are purged from data buckets. Space re-used for records with similar keys. Deleted �ID� remains gone allows alternate key, and RFA, access to conclude record was deleted. BTW: With �Fast Delete� a future alternate key access will remove that entry. Exceptions: The last key is never deleted, but left All Keys stay behind is Duplicates allowed

51. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 51 Deleted key ranges (2) Primary key index is never removed. Buckets remain �reserved� for key range. CONVERT/RECLAIM designed to clean up Requires Exclusive access PROBLEM: �GET First Record� krf=0, key=0, rop=kge� sys$get/sys$find Or: sys$rewind Or Cobol: initialize key� start greater than First bucket� no data. Next bucket� no data

52. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 52 Deleted key ranges, example �work queue� file with date & time key. Work needs to be done? Insert record. Server process takes action and deletes record. Btw: may be advisable to delete, then process. Even if processing fails, record should be removed. At end of day, no records should be left. If server(s) falls behind New records over-fill bucket �spill� into next bucket (after bucket split). Bucket(s) never re-used because keys always increase

53. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 53 Deleted key ranges, Solutions More frequent CREATE/FDL Do NOT tune FDL with ANAL/RMS of empty file Use worst case to size new files Increase Bucket Size Try to hold 95 percentile sized queue Maximum bucket size may be needed (63) Should only adds buckets when queue very deep. Watch out for RMS AIJ with large buckets. Shorter records Pointer records instead of full data Use Alternate Key as queue, not primary Ease pain with Global Buffers

54. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 54 Deleted key ranges, Solutions Better Starting points Remember last record processed, use KGE. Master record? (yuck!) Have client communicate start key Lock Value Block? Mailbox data? Does NOT have to be exact Update crossing buckets (peek into RFA?) Update every 100 records or 100 seconds?

55. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 55 Deleted key ranges (picture)

56. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 56 Index Depth Each level requires a bucket LOCK Each level to traverse may need an IO Bucket locking is cluster wide �Flat� files are often best May cause too much contentions. Compensate extra levels somewhat through more buffers

57. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 57 Index Depth (2) Global buffers Amplify locking pre V7.2 Need to serialize on Global Section Lock as well Decrease locking V7.3+ Concurrent Read mode lock Index Compression Compression disables Binary Search. Binary Search minimizes lock duration 1000 keys need only 10 compares Compression puts more indexes in buckets Only large keys (20+) with good effect (40+%)

58. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 58 Index Depth Cheat Sheet Sample

59. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 59 Index Depth Excel spread sheet

60. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 60 Deferred Write Quick win Somewhat similar to Write Behind on un-shared files. Fab$l_fop.fab$v_dfw = 1 Cobol: IO CONTROL � APPLY DEFERRED WRITE ON � Only useful when returning to same buckets Inserting records with similar keys Updating/Deleting record in a sequence. Defers the IO until Other process in cluster needs bucket (BLKAST) Process itself needs the buffer Explicit SYS$FLUSH call

61. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 61 Deferred Write Gotchas May want to add more local buffers No more carefully ordered writes RMS normally first writes objects, then pointer. External events may trigger DFW writes No Timer based flush Large time windows may open to data loss. Larger quantities of data prone to loss Double IO whammy if all buffers are dirtied First have to write, before even starting to read

62. QUESTIONS ? Hein van den Heuvel

63. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 63 Large Files (First presented @ HP Tech Forum Houston 2006) OpenVMS Tools: Convert, Sort, FDL Bucket splits Commercial Tools: select (EGH) Freeware Tools: tune_check, my_backup, DIX (Fekko Stubbe) compare RMS details, pratices.

64. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 64 What is the problem? What is big? More than a minute to scan at full speed? 60 seconds @50mb/sec ? 3GB = 6,000,000 blocks Converts taking 20+ minutes 80 GB file taking 2 hours ? Time and resources. Disk IO, cpu time, Elapsed time, LOCKS! Too easy wait and waste hours Too easy to decide �it is almost done, best let it go�

65. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 65 Convert at least Once a Year �Whether it needs it or not� Convert !

66. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 66 Tune : perfection not needed,Just get it roughly right. Tuned FDL

67. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 67 Minimize bucket splits ADD at logical END Add with primary key value higher than any existing key RMS will not split, but just add a fresh bucket and pointer to it in the index. No RRVs to update. Gotcha: YYMMDDHHMM with 1999 record left behind FILL factor not so great Pay everywhere, Gain in few places

68. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 68 FDLFile Description Language Describes main file DESIGN attributes Sequential / Relative / Indexed Record Format / Size Key(s) Type / Position(s) / Length(s) Describes TUNING attributes Mostly in AREA definitions Allocation, Extend BUCKETSIZE(s) Container for STATISTICS data

69. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 69 FDLUse more than 1 per file DESIGN attributes ? .fdl_design TUNING attributes ? .fdl_tuned_<ident> Ident could be NODE name, WHAREHOUSE,� STATISTICS data ? .fdl_stats_<ident>_<date> Optimize: $EDIT /FDL /NOINTERACTIVE file.fdl_design /OUT = file.fdl_tuned_rome � /ANALYZE =file.fdl_stats_rome_20060901

70. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 70 COUNTING recordsSimple problem no?

71. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 71 COUNT toolPart 1 struct FAB fab; struct RAB rab; char buf[32700]; int i=0, s, sys$open(), sys$connect(), sys$find(); fab = cc$rms_fab; rab = cc$rms_rab; fab.fab$b_shr = FAB$M_SHRPUT; fab.fab$b_fac = FAB$M_GET; rab.rab$b_mbf = 2; rab.rab$l_fab = &fab; rab.rab$l_ubf = (char *) &buf; rab.rab$w_usz = sizeof (buf); #ifdef RAB$M_NQL rab.rab$w_rop_2 = RAB$M_NQL; #else rab.rab$v_rrl = 1; rab.rab$v_nlk = 1; #endif

72. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 72 COUNT toolpart � 2 s = sys$open ( &fab ); if (!(s&1)) return s; fab.fab$w_gbc = 0; /* Avoid being a bad citizen */ s = sys$connect ( &rab ); while (s & 1) { s = sys$find( &rab ); i++; } if (s==RMS$_EOF) { s=1; printf ("%s has %d record%s.\n", argv[1], --i, (i==1)?"":"s");

73. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 73 CONVERTWhy ? Change a permanent attribute Bucket-size, keys, TUNE Clean External file fragmentation Clean Internal file fragmentation bucket splits Sanity check Silent corruptions Record count Clean output file Record consistent Backup

74. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 74 CONVERTWhen ? At least once a year �Whether it needs it or not� Regular interval based on usage patterns (Daily? Weekly?) Index levels above �target� Allocation above �target� Hot file list Your preferred tool (ECP, DecPS, PawZ,�) $ SHOW MEMORY/CACHE=(VOL=DKA100,TOPQIO=50) After major record purges RMS_TUNE_CHECK Provides all data listed above Online

75. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 75 File of files Consider a (CSV) file with file tune data. Filled by analysis tools Used by rules to trigger convert request Name, fdl file name Last convert date, record count, [cpu,dirio,elapsed] Last : size, index depths, record count Target : size, index depths, record count

76. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 76 CONVERTTools to help to know when ANALYZE/RMS/FDL Always use /FDL default output less useable Exclusive file access Use last-nights backup Use temporary BACKUP/IGNORE=INTERLOCK copy RMS_TUNE_CHECK Provides all data listed above, in compace Online http://h71000.www7.hp.com/freeware/freeware60/rms_tools/ DFU SEARCH /ORG=INDEXED /FRAGMENT=MINI=100

77. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 77 RMS_TUNE_CHECKexample � command line $ rms_tune_check_v3 -v -s=-1 -a=mail.ana MAIL.MAI * 21-SEP-2006 11:00:40.20 ALQ=2454 BKS=5 GBC=0 dev:[dir]name - ROOT: Primary key 0 (IBS=5,DBS=5) index root level is high: 2 (goal=1). - SKIP: 153 empty data buckets before Key: "B3210000 00A5AFE5". Convert. - RECS: Only 895 records in 434 bucket sampled. 2.1 rec/bkt. Inefficient? - SIDR: Key 1, 3148 Dups in 8 Buckets for value "NEWMAIL............"

78. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 78 RMS_TUNE_CHECKexample � analysis file part 1 ! Tune_check 21-SEP-2006 11:00:40.20 dev:[dir]file ! Created 14-APR-2006 13:17:36.37 ALQ=2454 BKS=5 GBC=0 ! LVL DB IB DTP TKS POS NUL Root-VBN Data-VBN Name !-- -- -- -- --- --- --- --- --------- --------- ! 0 2 5 5 BN8 8 0 1321 4 (null) ! 1 1 3 3 STG 39 9 0 793 790 (null) ! Records=895,rrv=4,del=246,lrl=1578,user_size=464,data_size=416 ! fill=35% in 10 ranges 0-100 :56:0:0:0:3:10:1:2:1:24 ! hit=151,miss=283,blocks_read=2220,series>5=2,max=124,avg=76 ! LVL/BKS/DAT/IDX: 2/4/896/12, 1/5/895/5 ! Compressed: 2/4/896/12, 1/5/895/5

79. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 79 RMS_TUNE_CHECKexample � analysis file part 2 FILE CLUSTER_SIZE 16 ANALYSIS_OF_KEY 0 DATA_RECORD_COMPRESSION 11 MEAN_DATA_LENGTH 464 ! average stored = 416 DATA_RECORD_COUNT 895 ANALYSIS_OF_KEY 1 !Dups: Key 1, 3148 Dups in 8 Buckets for value "NEWMAIL.........." DATA_KEY_COMPRESSION 71 INDEX_COMPRESSION 71 ! Faked MEAN_DATA_LENGTH 523 DATA_RECORD_COUNT 18

80. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 80 RMS_TUNE_CHECKexample � other file !Tune_check 30-AUG-2006 16:47:02.53 `file_name� ! � Created 13-OCT-2004 19:08:42.81 ALQ=28587392 BKS=2 GBC=0 ! �LVL DB IB DTP TKS POS NUL �Root-VBN �Data-VBN Name !-- -- -- -- --- --- --- --- --------- --------- ! 0 �5 �2 �2 STG �37 � 0 � � � 4213415 � � � � 3 (null) ! !Records=54031167,rrv=15434946,del=0,lrl=221,u_size=221,a_size=165 ! fill=69% in 10 ranges 0-100 :0:0:0:0:0:42:5:17:28:5 ! hit=3481422,miss=10319725,blocs_read=46251936, series>5=198164,max=46,avg=14 ! LVL/BKS/DAT/IDX: 5/3/20261688/547626, 4/4/19647700/400980,3/10/18631440/147880, 2/49/18644598/29841 ! Compressed: 5/2/21612468/514588, 4/3/20261688/311724,3/7/18910913/122808, 2/34/18745526/24854

81. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 81 RMS_TUNE_CHECKexample � DCL symbols $ show symb rms* !different VMSmail file from other example RMS_FILL = "fill=71% in 10 ranges 0-100 :1:2:2:4:7:11:10:13:16:30" RMS_IO_STATS = "hit=1312,miss=182,read=15430,series>5=59,max=253,avg=16� RMS_KEY_NUL_DUP_TOT_LIST = "" RMS_RECORDS = "Records=10084,rrv=81,del=321,lrl=1571,u_size=309,d_size=259" RMS_ROOT_LEVELS = "LVL/BKS/DAT/IDX: 2/4/5764/36, 1/11/5555/11.! Compressed: 2/4/5764/36, 1/11/5555/11"

82. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 82 CONVERTHow? Dedicated temp drive Freshly tuned FDL file CONVERT /FAST default. /NOSORT default if primary key the same /STATISTICS free information. Take it /SECONDARY=99 New! All alternates in 1 input scan /FDL=tuned /SHARE ? Severe read performance impact

83. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 83 CONVERTMinimizing downtime - classic Don�t do anything in the downtime window that could be done outside. Notably: ANAL/RMS/FDL Fine to do AFTER the convert, to get ready for the next time and to verify the file. EDIT/FDL/NOINTERACTIVE Prepare based on data from last convert, apply growth by changing the record count. When converting, no need to backup. SET FILE/NOBACKUP old.dat

84. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 84 CONVERTMinimizing downtime - special Pick a low usage window Extract all records, while application still up to .seq Convert .seq to .new Quick sanity check on new indexed file? Shutdown application Rename .dat to .old Rename .new to .dat Redefine logical for data as needed Startup application Binary compare .seq and .old, carrying forward changes.

85. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 85 CONVERTMinimizing downtime, extract Extract to sequential CONVERT/SHARE Vselect (EGH tool) DIX ? Or extract to Indexed file Primary key only Maximum bucket size (63) Key and data compression enabled OpenVMS Freeware �Indexed_file_backup� does this. http://h71000.www7.hp.com/freeware/freeware60/rms_tools/

86. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 86 CONVERTMinimizing downtime - compare Often one �knows� the application somewhat. Records added with ever increasing (primary) key New records have special flag Records have a date/time embedded in some key Could write dedicated (cobol/basic/c) program Could use Datatrieve Could use my new �Compare� tool!

87. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 87 COMPARE tool - 1 Driven by primary key (no dups allowed for now) Reads from two (sorted) files Write up to 4 output / change files with records .NEW holding all records which were not in extracted file, but were in the main file when application was stopped .DEL records which probably should be deleted .MOD_NEW after image for modified record .MOD_OLD before image for modified record To be submitted to OpenVMS freeware �real soon now�. Email for early copy.

88. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 88 COMPARE tool - 2 TEST new file using all keys seeded by selection of records from .seq file APLY change files $PUT any record from .NEW $DELETE, and $UPDATE as needed Safety features: additions from .ADD can be undone by renaming to .DEL Generates ERROR and EXCEPTION files to capture the operation, record number, error message (RNF? RLK?) and the data itself Performance features: Deferred write (effective as data likely in order) Large number of local buffer

89. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 89 CONVERT � new 8.3 Large output buffer Already double buffered for each level Now 127 block buffers hold several buckets 7.2 Control-T for approximate status (similare to backup) Uses record interface for primary key Combine Alternate keys values in CONVWORK records. Uses more disk space, Default is 1, which is the old behavior $HELP CONVERT file /SECONDARY=N �Increases the Convert utility's performance by reducing the number of required passes through the input data. This is accomplished by placing alternate key information into the CONVWORK file�

90. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 90 SORTING SORT Hypersort Syncsort EGH Select and Sort Evans Griffiths & Hart

91. 19-Feb-12 RMS Tuning, Hein van den Heuvel 2006 TUD RMS - 91 Adequate Extends High price if done wrong Not acceptable to run out of disk space while in production Frequent file system (XQP) requests each requiring several IOs (worse with High Water Marking) Fragmentation One bucket requiring multiple clusters = SPLIT IO Unwarranted disk head movements Very easy to do right Rely primarily on adequate allocation If file needs to grow, allow significant growth Gotcha: Maximum extend is 65536

92. QUESTIONS ? Hein van den Heuvel

93. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 93 FDL �magic� Topics FDL Overview Short �n Sweet Single stream command file Sequential file converts Windows/Unix/VMS Edit/FDL/Optimize Analysis files Spreadsheet alternative / complement Indexed file tuning Obligatory introObligatory intro

94. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 94 FDL Overview (no magic ?, just work) Structure Utilities Callable Dynamic vs Static attributes $edit/fdl (sys$system:fdl.exe) $edit your.fdl Read The Fine Manuals Guide to OpenVMS File Applications OpenVMS Utility Routines Manual OpenVMS Record Management Utilities Reference Manual Need to establish a common ground, so let�s begin with the basic, but not too long. Various sub-topics are listedNeed to establish a common ground, so let�s begin with the basic, but not too long. Various sub-topics are listed

95. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 95 FDL OverviewStructure RTFM: OpenVMS Record Management Utilities Reference Manual Chapter 4: File Definition Language Facility TITLE Sequential, variable records up to 320 bytes IDENT 25-SEP-1993 13:08:17 OpenVMS FDL Editor FILE ALLOCATION 5050 EXTENSION 505 ORGANIZATION sequential RECORD BLOCK_SPAN yes CARRIAGE_CONTROL carriage_return FORMAT variable SIZE 320 Just a few examples. FILE and RECORD are SECTIONS. Other sections are AREA 0, KEY 0, KEY 1,� A semicolon can replace a new line, and optionally follows a line. Spacing not important Tokens can be abbrevriated as short as possible to remain unique, but no shorter.Just a few examples. FILE and RECORD are SECTIONS. Other sections are AREA 0, KEY 0, KEY 1,� A semicolon can replace a new line, and optionally follows a line. Spacing not important Tokens can be abbrevriated as short as possible to remain unique, but no shorter.

96. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 96 FDL OverviewUtilities Record Management Utilities Reference Manual CONVERT/FDL CREATE/FDL ANALYZE/RMS/FDL EDIT/FDL Will not go into details here, except EDIT/FDLWill not go into details here, except EDIT/FDL

97. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 97 FDL OverviewCallable OpenVMS Utility Routines Manual Chapter 11 File Definition Language (FDL) Routines FDL$GENERATE, FDL$CREATE Freeware tool using fdl$generate to obtain fdl for open file. FDL$PARSE, FDL$RELEASE. Creates RAB�s and FABs and releases those. API allows to pass FDL as a string or file (V8.3 DCL enhancement) Need to establish a common groundNeed to establish a common ground

98. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 98 FDL Overview Runtime vs File-attributes Sometimes called: Dynamic vs Static Main FDL outputs are permanent file attributes. Some can be changed: $SET FILE /ATTR Bucket size, key length, compression ? static Extend, RAT, Global Buffers ? semi-static (set) FDL syntax exists for �deferred write�. However, this is a runtime bit, only useful for FDL$PARSE, not useful for $CREATE/FDL Need to establish a common groundNeed to establish a common ground

99. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 99 FDL Overview$edit/fdl (sys$system:fdl.exe)

100. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 100 FDL Overview $edit your.fdl FDL files are plain text files. There is no magic about them, you can just edit them, which is magic ? Need to establish a common groundNeed to establish a common ground

101. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 101 EDIT/FDLUndocumented features $DEFINE/USER EDF$$JOURNAL_INPUT <filespecification> An UNDOCUMENTED, UNSUPPORTED, logical to help support folks make a log. It is used to capture the user input during an EDIT/FDL session. Whilest such log is being made, one is encouraged to provide spelled-out, explicit, answers rather than accepting defaults or using shorthands as this will aid subsequent analysis. $DEFINE/USER EDF$$PLAYBACK_INPUT �YES� An undocumented, unsupported logical which is really not expected to be used and in fact slightly more clumbsy to use. If this logical name exists (any value) then the input for the EDF session which is expected to come from a terminal type device is tolerated to come from a disk file. This could theoretically be used to provide fixed input from a data lines in a command file or is from a journal file as per above! (after define/user sys$input)

102. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 102 Short �n Sweet (sequential) /FDL=NL: $ CONVERT/FDL=NL: funny.log normal.log �normal� VMS text file: Variable Length Records �funny� = DCL & Cobol VFC Print Format output �funny� = Stream format files. $ DIR/FULL � File organization: Sequential Record format: Variable length, maximum 0 bytes, longest X bytes Record attributes: Carriage return carriage control $CREATE/FDL=NL: test.dat $OPEN/APPE test test.dat Print format = Variable Length Fixed Control, 2 byte record header with carriage control CC = pre-linefeeds, post linefeeds, formfeed. PRN is superset, usefull for process permanent files. Print format = Variable Length Fixed Control, 2 byte record header with carriage control CC = pre-linefeeds, post linefeeds, formfeed. PRN is superset, usefull for process permanent files.

103. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 103 Short �n Sweet (indexed) $ CREATE/FDL=SYS$INPUT: test.idx FILE; ORG IND; KEY 0; SEG0_L X Very simple, single key indexed file Key is �character�, offset=0, length=X $ DIR/FULL � File organization: Indexed, Prolog: 3, Using 1 key Shelved state: Online File attributes: Allocation: 9, Extend: 0, Maximum Bucket size: 2 Global buffer count: 0, No version limit Record format: Variable length, maximum 0 bytes, longest 0 bytes Record attributes: Carriage return carriage control

104. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 104 Single stream command file $ convert/fdl=sys$input sys$input tmp.tmp $DECK file; org ind; key 0; seg0_l 5 $EOD aap aap aap noot noot noot mies mies mies $type tmp.tmp aap aap aap mies mies mies noot noot noot This avoid �carrying around� a separate FDL file, or creating one on the fly. Now if only DCL could do substitutions in data lines�.This avoid �carrying around� a separate FDL file, or creating one on the fly. Now if only DCL could do substitutions in data lines�.

105. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 105 Edit/FDL [/Optimize interactive] $EDIT /FDL VMS$PASSWORD_DICTIONARY.FDL /ANALYS=VMS$P� � Invoke script � Optimize Tuning � Line graph � Index Depth 4| 3| 3 2| 2 2 2 2 2 2 1| 1 1 1 1 1 1 1 1 1 1 1 1 1 +- + - - - + - - - - + - - - - + - - - - + 1 5 10 15 20 Bucket Size (number of blocks) PV-Prolog Version 3 KT-Key 0 Type String EM-Emphasis Flatter ( 9) DK-Dup Key 0 Values No KL-Key 0 Length 32 KP-Key 0 Position 0 RC-Data Record Comp 0% KC-Data Key Comp 81% IC-Index Record Comp 63% BF-Bucket Fill 100% RF-Record Format Fixed RS-Record Size 32 LM-Load Method Fast_Conv IL-Initial Load 42979 AR-Added Records 0 Shows the �break up� points for the bucket size vs index root level, single param change The assumption being that deep indexes are bad (and they are!) Al listed settings can be changes, and graph redrawn.Shows the �break up� points for the bucket size vs index root level, single param change The assumption being that deep indexes are bad (and they are!) Al listed settings can be changes, and graph redrawn.

106. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 106 Edit/FDL [/Optimize interactive] $EDIT /FDL VMS$PASSWORD_DICTIONARY.FDL /ANALYSIS=VMS$P� � Invoke script � Optimize Tuning � Additional Records Additional Record Count 66672| 3 3 2 2 2 2 2 2 2 2 2 2 2/1 1 1 1 1 1 1 | 3 3 2 2 2 2 2 2 2 2 2 2/1 1 1 1 1 1 1 1 50004| 3 2 2 2 2 2 2 2 2 2 2/1 1 1 1 1 1 1 1 1 | 3 2 2 2 2 2 2 2 2 2/1 1 1 1 1 1 1 1 1/1 33336| 3 2 2 2 2 2 2 2 2/1 1 1 1 1 1 1 1 1/1 1 | 3 2 2 2 2 2 2 2/1 1 1 1 1 1 1 1 1/1 1 1 16668| 3 2 2 2 2 2 2/1 1 1 1 1 1 1 1 1/1 1 1 1 | 3 2 2 2 2 2 2/1 1 1 1 1 1 1 1 1/1 1 1 1 0| 3 2 2 2 2 2 2/1 1 1 1 1 1 1 1 1/1 1 1 1 +- + - - - + - - - - + - - - - + - - - - + 1 5 10 15 20 Bucket Size (number of blocks) PV-Prolog Version 3 KT-Key 0 Type String EM-Emphasis Flatter DK-Dup Key 0 Values No KL-Key 0 Length 32 KP-Key 0 Position 0 RC-Data Record Comp 0% KC-Data Key Comp 81% IC-Index Record Comp 63% BF-Bucket Fill 100% RF-Record Format Fixed RS-Record Size 32 LM-Load Method Fast_Conv IL-Initial Load 42979 Double parametersDouble parameters

107. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 107 Analysis files $EDIT/FDL/NOINTER � /ANALYSIS=my_file.fdl_stats my_fdl_file Where the .fdl_stats file would look like: IDENT "Hein's minimal template for EDIT/FDL/NOINTERACTIVE analyze input" FILE; CLUSTER_SIZE 12 ANALYSIS_OF_KEY 0; MEAN_DATA_LENGTH 123 DATA_RECORD_COUNT 1234 EDIT/FDL/NOINERT Really only uses a few params: cluster for roundups, record count, record size. Average Record size stays constant? Record count known from convert + businss skills Freeware tool indexed_files_backup to guestimate analysis file data from record samples.EDIT/FDL/NOINERT Really only uses a few params: cluster for roundups, record count, record size. Average Record size stays constant? Record count known from convert + businss skills Freeware tool indexed_files_backup to guestimate analysis file data from record samples.

108. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 108 Spreadsheet alternative / complement

109. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 109 Indexed file tuning,Pre 7.2 global buffers Up to OpenVMS V7.1 the globals buffers where maintained as double linked list, through the GBDs, ordered by vbn. For a 'normal', freshly converted file the ROOT bucket ends up with the highest VBN. During lookup the global section lock was held. Solution? Use areas to resequence root and index vbn�s for buckets Put DATA in AREA 1 Put INDEX in AREA 0

110. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 110 Indexed file tuning,Pre 7.2 global buffers, Example Here are some numbers for a somewhat artificial file some 30,000 buckets, bucket size = 1 3000 global buffers, a root level of = 2 randomly accessed by 20 processes for some 2000 keyed gets. The file was loaded in two ways:

111. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 111 Indexed file tuning,Pre 7.2 global buffers, Results

112. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 112 Indexed file tuning,Pre 7.2 global buffers, VAX Results

113. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 113 Indexed file tuning.Aligning DATA buckets. WHY? XFC Cache lines �like� 16 block buffers aligned by 16 Not just 16, powers of 2 will do. EVA�s like IO start VBNs in multiples of 4 Knowledge needed to get there DISK CLUSTERSIZE a multiple of 16 RMS internal usage of file EXTENTs aligns on cluster size RMS disturbs alignment with Prologue/Descriptors FDL can be used to just mis-align top index for some key Top index likely in rms cache anyway.

114. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 114 Compression DATA and KEY compression is goodness INDEX compression good for large keys Save SPACE and save CPU TIME Fewer data buckets => fewer IO Fewer index keys => fewer index levels More effective caches Less memory to walk

115. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 115 Compression (continued) Remember to verify effectiveness FDL has no smarts to detect negative compression Index compression often discouraged as it prohibits binary searches in index buckets EDIT/FDL/NOINT default is to switch off index compression with no consideration for the key size

116. QUESTIONS ? Hein van den Heuvel

117. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 117 BACKUP Material

118. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 118 7.2 Enhancements Global buffers: Implemented hashing algorithm for cache searches (replaced sequential search) Improved concurrency to global section (replaced global section distributive lock with inline atomic instructions) Directory cache limits removed. Prior to V7.2, the maximum size of a directory file that RMS would cache was 127 blocks; wildcard lookups to larger directories would go directly to the file system. Beginning with V7.2, RMS attempts to cache directories of any size. CONVERT /SECONDARY=n : Avoid multiple passes over input Control-T for approximate status (similar to backup) Uses record interface for primary key

119. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 119 7.2 Enhancements (cont) Extended file naming (ODS-5) enhancements added to RMS (see Guide to Extended File Specifications): Support for a wider range of characters in a file name, extension, and directory Access to file specifications with extended characters Support for directory structures deeper than eight levels Access to file specifications longer than 255 bytes through the NAM block with some restrictions in functionality Access and complete specification of file specifications longer than 255 bytes by callers who are aware of the new naming characteristics through a new interface (NAML block)

120. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 120 7.3 Enhancements Global buffers: Read-mode global buffer bucket locks RAB$V_NQL: No query record locking option introduced RAB$V_NLK + RAB$V_RRL + SET RMS new record locking options to control deadlock detection) Enhancements also backported to V72-1H1, V7.2-2 V7.3-1: Global buffers Global section changed from temporary to permanent section Sequential file out-of-box improvements: Increase in default I/O transfer size (from 16 to 32 blocks) New system RMS writebehind performance option (sysgen parameter RMS_SEQFILE_WBH)

121. OpenVMS Bootcamp, May 2006 HvdH Performance Consulting 2006 TUD RMS - 121 V8 Enhancements V8.2: Global buffers: Enhancement to virtual address (VA) space management, which greatly reduces the possibility of a VASFUL error due to repeated opens and closes of files with RMS global buffers by an application. V8.3: P2 space global buffers (more than 1 GB global buffers) Convert large output buffer (124 blocks)