The Dimensions of Service Exploring WLM’s Solution Space

1. The Dimensions of ServiceExploring WLM’s Solution Space NC CMG Tuesday 4 Nov 2003 Rich Olcott Schering-Plough Corp (now with IBM Global Services)

2. Agenda • Elements of control theory • Overview of WLM • A strategic tool: the Goal Map • The Attainment dimension: PI • The Cost dimension: SU absorption • A tactical tool: the Actuals Map • Another tactical tool: the CR Map

3. Control/Response Curve Describes the relationship between what you do and what you get. Generally depends on both the controlled system and its environment. “If you can’t measure the response curve, you can’t manage the response.” X

4. WLM’s Bailiwick

5. WLM meets its goals by… I/O • DP for interrupts • (Add’l paths) CPU access • Dispatch priority • (Add’l engine) Server count • Enclaves (DDF, Web, stor’d proc) • (Batch inits) Memory • Fencing • Swap in/out … robbing Peter to pay Paul

6. Workload Management Imp=2 Imp=2 Imp=2 Imp=3 Imp=3 Imp=3 Imp=3 Imp=2 Imp=3

7. Importance 0 1 to 5 6 3 Dimensions of A Goal Type Value SYSTEMS Percentile Response Time Velocity Average Response Time Discretionary

8. Goal Types • No goal (System, Discretionary) • Response Time • Average Response Time • Sum (RT) / (Transaction count) • Percentile Response Time • P% completed in less than target RT • Throughput • Velocity • Has nothing to do with speed (CPUsec / sec) • Has nothing to do with priority • Has nothing to do with Importance

9. 100 * U U + D Vel = Delayed Using What Goes into Velocity? Task Managed Other (Unk) Idle

10. 100 * U U + D Vel = I/O Operation IOSQ (UCB wait) Pending (path wait) Disconnect (latency) Disconnect (latency) OW47667 OW47667, IOPQ PAV IRD Velocity and I/O Task Managed Other (Unk) Idle Delayed Using IOSQ (UCB wait) Pending (path wait) Disconnect (latency) Connect (Xfer data)

11. Task Managed Other (Unk) Idle Delayed Using 100 * U U + D Vel = Velocity and CPU Estimated “Using samples” z/OS OW55665 Measured CPU time

12. Attained vs Target Target velocity

13. “High” Velocities SMFDUMP LOGREC NETVWLOG RMFIIIBK SYSLOG* SYS1DUMP WTRHSM (Mean Etime = 15sec) Bottom-feeders

14. Umbrella, genl purpose DDF, etc, and outliers are politically visible <10 completions per 20 minutes All else PRT vs ART vs Velocity Velocity PRT ART

15. Barred from RT Eden Long-running tasks, such as: • Conversational CICS • “Some” MQ-originated transactions • DDF with: • THREADS=ACTIVE in DB2 instance • RELEASE(DEALLOCATE) in DB2 plan • CATIA • CICS instance with no active RT goals

16. 4 Dimensions of A Period Importance Period Id 0 1 2 3 4 5 6 CICPXN TSO P1 TSO P2 SBAT LBAT P1 LBAT P2 ….

17. The Goal Map • A one-page display of goal structure • Maps the strategic dimensions: • SC/Period Id - Importance • Goal type - Value(s) • Each SC period goal is plotted according to its Importance and its Workload

18. A Sample Goal Map

19. Assigned Jobclasses

20. A Sample Goal Map

21. A Sample Goal Map

22. Calculating PI • General rules: • PI is a ratio that compares actual with target • PI is < 1.0 iff performance is better than target Ave RT goal: PI=RTactual / RTtarget Velocity goal: PI =Vtarget / Vactual

23. Percentile Response Goal A brilliant solution, but with pitfalls… • Most shops use percentile RT goals • Calculating any percentile is expensive • Solution: use buckets • Goal is “p% with response time < t ” • Define PI = (RT for p%)/(t) • Tally completions into buckets by RT • Calculate PI by interpolation between buckets

24. PRT Buckets - The RTDA 75%ile ~108% of target ==> PI = 1.08

25. Interpolation

26. Percentile Response Goal A brilliant solution, but with pitfalls… • Most shops use percentile RT goals • Calculating any percentile is expensive • Solution: use buckets • Goal is “p% with response time < t ” • Define PI = (RT for p%)/(t) • Tally completions into buckets by RT • Calculate PI by interpolation between buckets

27. Control/Response Curve WLM’s perspective on how the system behaves “Controlled variable” = WLM’s Internal variables (SRM, etc.) “Response” = PI One curve per driver per SC period with a managed goal Periodically re-estimated from current data

28. A Good Sight-picture

29. Case 1 - Sight Off-target

30. Case 1 - Lenient Goals

31. Case 2 - High-x Scope

32. Case 2 - Broad Workload

33. The Cost of Service Service Units used = C1 * (“TCB” CPU seconds) + C2 * (“SRB” CPU seconds) + C3 * (IO activity) + C4 * (CPU * memory) C1 = CPU Coefficient * SRM constant C2 = SRB coefficient * SRM constant C3 = IOC coefficient C4 = MSO Coefficient (should be 0.000) ... in WLM “Options” panel

34. V=30 V=50 The Goal Map And SU Usage

35. The Goal Map in 3D SU Load Importance Workload

36. How to Read a Goal Map • WLM satisfies goals from left to right. • Within an importance level, WLM tends to maintain all workloads near the same PI value. • Within an importance level, the periods that draw the most resources are the ones with the most tasks and/or have the most aggressive goals.

37. T72(3), SCS section Record per period per RMF recording interval Goal type Importance Target value Data for PI calc RT %ile Buckets T99(6) Record per period per WLM policy adjustment intvl Goal type Importance Target value Local and Sysplex PI’s SMF Data

38. Lessons from SMF Data • Generated only for in-use periods • Reports only the in-effect policy • In-effect classification rules are assumed, not reported

39. SDPDS Structure A standard ISPF ISPTLIB-style library

40. Service Goal Tables * - For Periods 1-8: Disc (Y/N), Importance, SRM Duration, Velocity, %ile, Hr, Min, Sec Policy name is either the base or one alternate.

41. Lessons from the SDPDS • An alternate policy is the base, plus one set of overrides. • Overrides apply to the base policy only. • Classification rules are global to an SD. • The SDPDS is the place to go for comprehensive knowledge of an SD. • Change mgmt and availability mgmt must be accomplished by PDS mgmt.

42. Chg Mgmt and the SDPDS Install &SYSID.WLM.SDPDS.TRIAL 1 – Clone with WLM SAVE AS 3 – Clone with WLM SAVE AS CDS &SYSID.WLM.SDPDS.PROD 2 – Copy with ISPF =3.3 &SYSID.WLM.SDPDS.RESERVE

43. Tactical Dimensions • Service class/period identifier • Importance • Goal attainment (PI) • Cost (SU absorption rate) • Overall system load (total SU rate) • Time

44. One SC Period in 3D Cost Time Attainment 1.4 <= PI SU’s used relative to other SC/pd’s in intrvl 0.6 < PI < 1.4 PI <= 0.6 15 min

45. The Actuals Map

46. Reading An Actuals Map - 1 Attainment dimension SYSTEMS and DISC will always be Expect high-importance lines to be Expect patches of lower down, especially in a stressed system. A lenient goal ==> persistent An unmanageable goal ==> persistent

47. Reading An Actuals Map - 2 Cost dimension Bar heights highlight the costliest SC/period in each interval. “Tot SU Usage” is key to evaluating impact of a “heavy user” at a given time. Watch for timing correlations between different lines.

48. After Some Tuning...

49. CR “Maps” • Need to locate cross-assignments, duplicates; identify missing items • WLM’s report • one report per subsystem • precedence structure within Subsystem • Alternative map structures • group work assigned to a Service Class • group work within a Report Class • line per workload, including descriptors • Spreadsheet format • slice and dice to taste

50. Classification Rule Tables