Divide and Conquer: Dealing with 140GB of SMF Data Daily

1. Divide and Conquer: Dealing with 140GB of SMF Data Daily Chuck Hopf Merrill Consultants chuck@mxg.com

2. Gathering Data - Steps • Determining what data is available • Determining what data is needed • Determining data retention requirements • Gathering the data • Dealing with increasing volumes

3. Data Availability • Different OS’s have different sources • For MVS • SMF • RMF • Vendor SMF records • Other software logs/sources

4. SMF/RMF • Job accounting • System utilization • CICS Transactions • CICS Statistics • DB2 Accounting • DB2 Statistics • Dataset activity

5. Vendor SMF Records • NETSPY • Stop X37 • MXG Tape Mount Monitor • IND$FILE • TPX • HSM • many many more

6. Other Data Sources • DCOLLECT - DASD space utilization • CA1 - Tape utilization • TMON - alternative CICS/DB2 transaction data source • SYSLOG - console commands • many more

7. Data Requirements • Not all data is always useful • Largely driven by reporting requirements • But… some can be discarded outright and others cannot • type 4 5 34 35 are obsolete • other types are not generally useful • for our purposes, SMF types 4 5 16 19 34 35 40 and 69 are suppressed

8. Data Rentention • How long do you keep the raw data? • Ask your internal auditors • At least partially driven by volume • at 120GB/day, keeping CICS detail for very long is impractical and DB2 is not far behind • Third largest volume is type 74 RMF data • Fourth is type 42

9. Gathering the Data • SMF records to a SYS1.xxxxxx dataset • When the dataset is full, it is automatically switched to a non-full dataset and (if the IEFU29 exit is coded) a dump/clear process is started

10. Gathering the Data • Best case - a single MAN dataset lasts all day • Dump and clear after a switch at ‘midnight’ • Worst case - a MAN dataset fills every 15-20 minutes or less • Dump and clear frequently and consolidate after ‘midnight’

11. Gathering the Data • ‘Midnight’ may not be midnight. It may depend on a batch cycle and the availability of storage devices (tape drives) • The process is driven by the volume of data

12. Flies in the Ointment • Lost data can occur • When buffer expansions occurs • When MANx dataset fills during an interval ‘pop’ • When something slows down the IO to the MANx volume

13. Flies in the Ointment • CICS and DB2 may complain about SMF not being available • Can result in CICS slowdowns and response time problems

14. Flies in the Ointment • At 10MB/second, it takes about 270 seconds to fill a 3390-3 • At 10MB/second, it takes about 600 seconds to dump and clear a volume • We are running at about 600-900 seconds at peak loads to fill a volume • What happens if volumes double? • Can the SMF writer handle the load?

15. Fly Swatters • Must have at least two SYS1.Manx datasets and three is better • As volumes increase more may be needed • Make each SYS1.MANx dataset a full volume • Eliminates IOSQ delays • Eliminates device pending delays • Prevents problems

16. Fly Swatters • Add an auto operations rule: • 5 minutes before the intervals are due to ‘pop’ issue a D SMF command and if the active MAN volume is more than 85% full, issue an I SMF command. • Eliminates filling the MAN volume during the interval processing in all but some extreme (and bizarre) cases

17. Fly Swatters • Use Maximum Values for Buffer sizes • Tune CI Size on MAN datasets to match your workload • APARs are in progress to change the buffer acquisition algorithms

18. Fly Swatters • Turn off CACHE in RMF on all but a single system. It will not only reduce the volume of data but will reduce the CPU time consumed by RMF. Do the same with RMF III (it is a BIG CPU reduction with RMF III.)

19. The Flies Will Win • Unless changes are made in the way the SMF writer works, we are going to break it in the near term (1-2 years?) It will simply not be able to keep up with the arrival rate of data.

20. Gathering the Data - Simple Case MAN dataset IFASMFDP Dump and Clear Daily SMF

21. Dealing with Volume • Treat it as you would any other application • Parallelism is the answer as volumes increase

22. Dealing with Volume • Split the data using IFASMFDP into chunks of a manageable size • No arcane exits just simple control statements • OUTDD(CICS,TYPE(110)) • OUTDD(DB2,TYPE(100:102)) • OUTDD(SMFDUMP,NOTYPE(100:102,110))

23. Dealing with Volume Less Simple Case Daily CMF Data MAN Dataset IFASMFDP Dump and Clear Daily DB2 Data Daily SMF Data

24. Dealing with Volume • As volumes grows, the structure can grow with it - up to a point

25. Dealing with Volume Complex Case CICS Data Daily CICS Data IFASMFDP MAN Dataset IFASMFDP DB2 Data IFASMFDP Daily DB2 Data Daily SMF Data Other SMF IFASMFDP

26. Dealing with Volume • At some point, it gets too huge to process in a single piece. • Time to use the IFASMFDP exits • Break CICS/DB2 up by APPLID/SUBSYSTEM

27. Dealing with Volume CICS1 CICS2 MAN dataset IFASMFDP CICS3 DB2 Other SMF

28. Dealing with Volume • SMF data can be piped with batch pipes • Must be VB NOT VBS and it must be set to VB at the initial dump of the MAN volume • Best not to pipe when dumping MAN volumes • The dump runs at the speed of the downstream process which may cause problems (dump/clear process running slower than next MAN volume fills.)

29. Piping SMF • If you intend to pipe SMF data, you would be wise to put in an IFASMFDP exit to catch any records larger than 32756 and either route them to a separate VBS file or discard them. There are not supposed to be any but there are (type 8 and type90.32 - APARs pending.)

30. Building the PDB • Before you start • Customizing • Retention • What cycles to run

31. Customizing • Define workloads • Define shifts • Define accounting fields • Define user records to be added • Define variables to be kept • To compress or to not compress - that is a question

32. Defining Workloads • Used to be limited to 15 workloads new limit is 114 (but that does not mean you should have 114 - more than 20 becomes cumbersome.) • Was restricted to control performance groups may now use report performance groups (be careful not to double dip.)

33. Defining Workloads • Two members affected • IMACWORK - old method • RMFINTRV - new method

34. Defining Workloads - IMACWORK • Simple IF THEN ELSE logic • IF PEFGRP=2 THEN WORK=‘TSO’; • IF SRVCLASS=‘TSO’ then WORK=‘TSO’;

35. Defining Workloads - RMFINTRV • Workloads defined using a parameter passed to a MACRO • Each workload (WORK1-WORK99) parameter has five components separated with a / • Name - 3-4 character description • 9 characters to be used in label describing workload • list of performance groups to include in the workload • list of service classes to include in the workload • number of periods in perfgrp/service class

36. Defining Workloads - RMFINTRV • Performance groups and service classes can be mixed • Report and control groups can be mixed • But… it is not a good idea. If you are going to use report groups use NOTHING but report groups

37. Defining Workloads - RMFINTRV %VMXGRMFI( …. WORK1=BATT/Test Jobs/1 3/BATCHLO, WORK2=TSPD/Dev TSO/2/DEVTSO/2, WORK3=DB2A/DB2A/ /DB2A, WORK4=DB2B/DB2B/5 … );

38. Defining Shifts • How are shifts important in your world? • Batch cycle vs online • Operations shifts • Weekends • Holidays

39. Defining Shifts - Holidays

40. Defining Shifts - Holidays • Holidays usually look like a weekend day • Can distort weekly data and plans • Should you or should you not exclude them • Let the best technicians in your organization (the managers) make the call

41. Defining Shifts - IMACSHFT • More simple IF THEN ELSE logic IF 8 LE HOUR(TIMEPART(DATETIME)) LE 16 AND 2 LE WEEKDAY(DATEPART(DATETIME) LE 6 THEN DO: SHIFT=‘P’; DATETIME=DHMS(DATEPART(DATETIME),8,0,0); END; ELSE IF ...

42. Defining Shifts - $SHIFT • The first question when you present a report will be ‘What does this P mean under SHIFT?’ It will not matter how many other reports show the same thing or how many times you explain it. • A user format solves the problem.

43. Defining Shifts - $SHIFT PROC FORMAT LIB=LIBRARY; VALUE $SHIFT ‘P’=‘First ’ ‘S’=‘Second’ ‘T’=‘Third ’ other=‘Weekend/Holiday’;

44. Defining Accounting Fields • How many do you really really need? • More than 2 or 3 will start to be redundant • Limit the size and number

45. Defining Accounting Fields - IMACACCT • Assume 3 fields each 5 bytes long DROP ACCOUNT4-ACCOUNT9 SACCT4-SACCT9 LENACCT4-LENACCT9 ; LENGTH ACCOUNT4-ACCOUNT9 SACCT1-SACCT9 $ 5 ;

46. Define User Records • Originally required multiple exits • Now done ‘instream’ in BUILDPDB process • Still invokes the original exits • EXPDBINC - include the source code • EXPDBVAR - build the datasets/variables • EXPDBCDE - read the data • EXPDBOUT - sort the data into the PDB

47. Define User Records • Each user record must have an SMF ID defined • These should be stored in IMACKEEP MACRO _IDTPX 205 % /* TPX */ MACRO _NSPYID 132 % /* NETSPY */ MACRO _SYNCID 208 % /* SYNCSORT */ etc...

48. Define User Records • Modify SYSIN %LET EPDBINC=%QUOTE( VMACNSPY VMACTPX VMACSYNC ... ); %LET EPDBVAR=%QUOTE( _VARNSPY _VARTPX _VARSYNC ... );

49. Define User Records %LET EPDBCDE=%QUOTE( _CDESPY _CDETPX _CDESYNC ... ); %LET EPDBOUT=%QUOTE( _SNSPY _STPX _SSYNC ... );

50. Defining Kept Variables • Can be done using the ‘MACKEEP’ macro variable to redefine the _Vdddddd macro for a dataset or by using the _Kdddddd macro for the dataset.