Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems

1. Customer Experiences withHiperDispatch & Soft Capping inIBM Mainframe Systems Prepared for presentation at CMG Canadaon April 18, 2012 Jonathan Gladstone, P.Eng.Senior Tech. Specialist, Mainframe & Mid-range Systems Capacity PlanningTechnology & OperationsBMO Financial Group

2. Agenda • HiperDispatch • A high-level discussion of BMO’s implementation of this feature • Soft Capping • Detailed presentation with circles and arrows and a paragraph… and apologies to Arlo Guthrie • BONUS TOPIC!Transition to z196 – Performance Implications • Just a preview; detailed analysis not yet complete Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFG

3. HiperDispatch: how is it supposed to work? • HiperDispatch aligns workloads “vertically” on physical CPs • Builds a strong affinity between logical and physical processors - details available in zJournal article viewable at www.mainframezone.com • https://www.mainframezone.com/article/hiperdispatch-a-conceptual-overview • Applies to all processors by type, when logically shared: zAAPs, zIIPs, GCPs • VH (100%), VM (50-99%), VL (<50%; discretionary) by weight, but avoiding VL where possible • Purports to improve performance by reducing latency times, e.g. for CP state re-loads • Keep data and instructions in lowest-level (fastest) cache • Performance improvement claims vary depending on configuration • Largest (8-10%) for large, multi-book CECs with many large systems sharing logical resources extensively; least (0-2%) for single-book CECs with few systems with limited sharing. • Performance improvements baked in to LSPR ratings for z/OS 1.11 and up • Turning off HiperDispatch now yields less than optimal performance. Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFG

4. HiperDispatch: how is it applied and working at BMO? • Turned on when we went to z/OS 1.11 • Default in z/OS 1.11 and up is “ON” for HiperDispatch; we left it that way • Nasty surprises! • Specialty workload flowing back to GCPs • VL engines left parked while workloads not meeting WLM target performance • Causes? • Investigated multiple changes: new z/OS, zAAP-on-zIIP, HiperDispatch • Fixes? • Set zIIP (and zAAP) weights properly – never mattered before • Changed GCP weights to minimize impacts • Reviewing WLM profiles • Results? • Things working much better now (see third discussion regarding z196 performance) • Conclusions • HiperDispatch appears to yield performance benefits claimed by IBM, but… • Weights are now important all the time (not just when box is maxed out) • WLM profiles are more important than ever • Some situations still difficult (e.g. K-LPARs for GDPS) Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFG

5. Soft Capping: how is it supposed to work? • Soft Capping is available for single LPARs or “Capacity Groups” (CGs) of LPARs • Available since 2005 or earlier • Applies only to GCPs • Uses same MSU ratings as SCRT reports for VWLC • Limits CPU utilization of LPAR or Group based on four-hour rolling average (4HRA) • Checked by PR/SM every 5 minutes • Utilization can go as high as enabled capacity until 4HRA hits cap; then PR/SM will limit utilization until 4HRA drops below cap again • A little more complicated for Capacity Groups: cap is applied to individual software products rather than for LPARs, and LPAR weights are used as needed • 4HRA can exceed cap • After Cap is reached, utilization at cap will often increase 4HRA for a few intervals until it settles back • Reports suggest 4HRA will exceed cap by about 3% in these circumstances • IBM VWLC charge is based on cap rather than on actual utilization • True for whole CPC if CG includes all LPARs Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFG

6. Soft Capping: how is it applied and working at BMO? • In place at BMO in four instances (three current) • One z10 Production CEC from Jul/09 through Jan/10 • One z196 Dev/Test/QA CEC from Sep/11 through present • Two z196 Production CECs from Jan-Feb/12 through present • SCRT reports show one instance of capping at BMO • Nov. 19, 2011 in z196D1: SCRT report shows MSU utilization hit 321 MSU on cap of 312 MSU in capacity of 408 MSU • Analysis based on data from TDS/z and from SCRT reports as submitted to IBM • All IBM tools • Interesting results, with differences to customer benefit • Data clearly show Soft Capping working, as documented • No charge for over-utilization, as documented • Unexplained difference between computed value and SCRT report is to customer advantage Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFG

7. Soft Capping: November, 2011 – Cap takes effect • Capacity 408 MSU • CG cap 312 MSU • Utilization above cap for several intervals beforehand • 4HRA computed from TDS/z crosses cap around 01:20 • Rises to 325.7 MSU • Peak hourly 4HRA 325 MSU according to TDS/z, only 321 MSU according to SCRT • 1.3% difference due to truncation instead of averaging? • PR/SM not counted • Working exactly as expected! Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFG

8. Soft Capping: December, 2011 – Cap doesn’t kick in • Capacity 408 MSU • CG cap 312 MSU • Utilization above cap for several intervals in morning hours • 4HRA computed from TDS/z never crosses cap • Rises to 302.1 MSU • Peak hourly 4HRA 301 MSU according to TDS/z, only 297 MSU according to SCRT • 1.2% difference due to truncation instead of averaging? • PR/SM not counted • Still working exactly as expected! Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFG

9. Soft Capping: January, 2012 – Effects of POR • Capacity 408 MSU • CG cap 312 MSU • POR around 00:50 drops 4HRA to 1MSU as documented • SMF70LAC catches up with results computed from TDS/z around 04:40 • Utilization above cap for several intervals • 4HRA computed from TDS/z never crosses cap • Rises to 295.6 MSU • Peak hourly 4HRA 291 MSU according to TDS/z, only 285 MSU according to SCRT • 2.1% difference due to truncation instead of averaging? • PR/SM not counted • 4HRA computed from TDS/z never crosses cap • Rises to 302.1 MSU • Still working exactly as expected! Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFG

10. Soft Capping: February, 2012 – Cap takes effect, unreported • Capacity 408 MSU • CG cap 312 MSU • Utilization above cap for several intervals beforehand • 4HRA computed from TDS/z crosses cap around 17:10 • Rises to 316.5 MSU • Peak hourly 4HRA 316 MSU according to TDS/z, only 303 MSU according to SCRT • 4.3% difference due to truncation instead of averaging? • PR/SM not counted • Capping never shows in SCRT, but capping effect still clear Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFG

11. Transition to z196 – Production Site Experience • Before upgrades: z10ECs, one book with GCPs, zAAPs and zIIPs • After upgrade: z196s, one book with GCPs and zAAP-on-zIIP • Drops in CPU demand evident for GCP and Total utilization • MIPS normalized using LSPR (1.9 for z10, 1.11 for z196) • CPU demand normally rises to a peak at end of February, RRSP season • Driven by transactions • Analysis will look at MIPS per transaction for several workload classes • CICS • DB2 • WebSphere • Batch Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFG

12. Transition to z196 – Dev/Test/QA Site Experience • Before upgrade: z10EC, two books with GCPs, zAAPs and zIIPs • After upgrade: z196, one book with GCPs and zAAP-on-zIIP • CPU demand rises for GCP and Total utilization • MIPS normalized using LSPR (1.9 for z10, 1.11 for z196) • Not explained • Harder to analyse D/T/Q environment • Analysis will look at MIPS per transaction for several workload classes • CICS • DB2 • WebSphere • Batch Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFG

13. Summary Questions? • HiperDispatch • Drives performance benefit, but… • Requires extra vigilance in setting LPAR weights (GCP, zAAP, zIIP) • Requires careful review of WLM profiles • Has difficulty where normally low-utilization systems (e.g. GDPS K-systems) need high weights • Soft Capping • Performs as expected – yay! • Minor added benefits (SCRT calculation, PR/SM left out) • Transition to z196 • Performance expectations based on LSPR for z/OS 1.11 • Includes HiperDispatch • Performing better than expected • Detailed analysis pending Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFG

14. About the Author • Jonathan Gladstone is an IT Capacity Management professional with well over 20 years experience in computer systems management and planning. He has been at BMO Financial Group for almost 15 years, and working in capacity planning for over a decade. He is BMO’s representative on Georgian College’s Computer Studies Advisory Committee, is certified in ITIL v2 & v3 fundamentals and holds a B.A.Sc. degree in Electrical Engineering from the University of Toronto and P.Eng. certification from the Province of Ontario. • Jonathan wishes to thank many colleagues who helped with this presentation, in particular Steve Pritchard (BMO Financial Group), Horace Dyke (independent consultant) and Don Mackay (IBM Canada). • Jonathan can be found on LinkedIn, Twitter (@jbglad59) and on his own (largely I/T) blog, http://alwaysgrumpy.wordpress.com. Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFG