Performance Modeling

1. Performance Modeling John Meisenbacher, MasterCard Worldwide

2. If You Have a Transaction Processing System… • I can create a simple accurate CPU performance model… ….that can help you make your system more efficient… …in one week… …with the data you already collect… …and so can you !

3. What is a Performance Model? It provides the quantitative information you need to: • Manage the performance of your system; and • Understand the impact of: • software changes; • hardware / configuration changes; and • actual / projected transaction load changes.

4. The CPU Performance Model • What is the best model? • The best model is only as complex as it needs to be to provide information that is sufficiently accurate to help you make the right decision.

5. Performance Management Goals • Support capacity planning • Identify performance changes with changing hardware, software and transactions. • Isolate performance issues within the system. • An accurate production model should: • Be easy to use / understand; and • Provide low cost and timely information.

6. About Transaction Processing… • There are basic features of most transaction systems in the real world. • The rate of transactions per second (TPS), varies: • During the day, there are peak hours • During the week, there are peak days • During the year, there are peak months / seasons • Transaction processing systems are typically sized for the peak hour of the peak day of the next peak season.

7. Daily Transaction Cycles

8. So, You’re Asking Yourself… Q. Is my hardware (CPU) properly sized to support my peak transaction period in my next peak season? A. Possible ways to answer this question… • Use a commercial capacity planning tool. • Run a performance test in the lab. • Use a CPU model validated in production.

9. Commercial Capacity Planning Tools • Commercial capacity planning tools typically focus on: • system size, • acute system issues, and • hardware projections. • Potential challenges: • These tools typically do not easily integrate with transaction rates, and therefore cannot model transaction systems. • CCP tools may be good for monitoring and alerts, but unless transaction loads are factored in, this approach cannot give an precise answer.

10. Daily CPU Usage Pattern

11. Build a Performance Test Environment This approach requires: • Building and maintaining a hardware test environment that is as close as possible to production. • Loading production software and building a database that is as close as possible to production. • Developing a way to send production-like transactions through the system. • Maxing out the system to see when it breaks. • Hoping your test setup is close enough to production to matter (or calibrating it with a production CPU model).

12. A CPU Model of the System A CPU model of the transaction processing systems integrates three key workstreams: • Business – the transactions processed; • Capacity Planning – the hardware systems used to process the transactions; and • Software Engineering – the software systems and databases that process the transactions.

13. About Transaction Rates… Most online real world transactions rise and fall during the day, week, month, and year. The CPU method takes advantage of these cycles. Multiple data points are used for correlation analysis. The approach can be used in production, possibly eliminating the need for a performance test environment.

14. Transaction Processing and CPU Usage Sample

15. Data Reported in Time Increments • Report data around the peak period. • CPU usage in some time increment, generally between 10 and 20 minutes. (15 minutes is common) • Shorter time periods provide more data and variation, but may not improve accuracy. • Longer time periods require less data and hide variation-reducing accuracy and precision. • Transactions are typically logged by time. • Multiple days can be easily merged into one dataset.

16. Create the Model • Display results graphically • The model from the previous graph is simply defined: • CPU utilization = CPU per TPS * TPS + overhead • y = a*x + b • “System capacity” is the TPS rate that drives the system to your “safe” limit, such as 80% CPU utilization. • TPS Limit = (“Safe” CPU limit – overhead ) / CPU per TPS • Most systems are linear, but also can be exponential or logarithmic.

17. Graph of Correlation Shapes

18. Correlate TPS and CPU

19. Transaction Processing Systems • The software-intensive system typically includes processes related to transactions: • The actual transaction processing • Data management for look-ups and logging • Communication processing • Format conversion • Also may include other processes, such as system monitoring, that are independent of transaction rates.

20. CPU Divided by Type of Processing

21. Use the Model to Manage the System • Factor the results into your planning • Use the data to help identify any efficiencies for the upcoming peak season • The change, in this real-world case, was made to transaction replication in the database. • The release was rolled out into product and a new model created.

22. System After Application Performance Modification

23. Manage Performance within a Release The precision of the model can be used to measure changes in the system. Data from the previous day can be collected and quickly analyzed to determine if changes to the system have improved or degraded performance.

24. Variation of System Capacity OK during peak period OK after release Other scripts optimized And removed Monitoring Scripts added

25. Work with System Experts • Key to success: support the business through cooperation between capacity planning and software experts. • Creating the model, generally fairly easy, is only the first step. • Work with developers or vendors to successfully manage and adjust system performance. • Support the model’s quantitative information with experience and insight

26. Trust Your Experience and Intuition • Use the model along with your understanding of hardware and software systems to direct your analysis and decision. • Collect more production data to evaluate and refine these decisions with the goals of: • saving money for your company, and • delivering quality results to your customers.

27. Thank you.