Evaluating Transaction System Performance Using Real Workloads Ashutosh Tiwary Co-Founder and CTO www.performant.com Confidential & Proprietary to Performant, Inc.
Performance is a real problem • IT projects fail due to multiple reasons • Performance is an important component • Application complexity is growing dramatically • Many layers of opaque 3rd party components • More and more IT developers know about less and less • Deploying scalable J2EE applications is hard • 12 SUN CPUs, 12 GB J2EE App server performance? • Packaged apps have their own challenges • $10M - $1B spent on deployments
Performance problem examples • Inability to recreate real world problems • “Our frustration is that we cannot recreate the production problems in our test environment”(Director of Performance Testing) • Inability to anticipate problems with current tools • “Why did we not find this problem with our load testing before we deployed the application?”(VP Engineering) • Inability to predict future performance • “Will our current system be able to handle the anticipated load for next 24 months?”(VP Information Systems)
Transaction from User Manipulation HTTP Requests REPOSITORY WEB SERVER Capture JSP Requests Performance Data Correlation P P Playback APPLICATION EJB Requests P P P Performance Data APPLICATION SERVER JDBC Requests P Performance Data P P P P DATABASE SERVER Real Workload Approach A new approach Real workload instead of synthetic Diagnose and fix performance problems in production systems A view inside the application while under load – detailed analysis Automated application instrumentation
Current implementation: OptiBench • HTTP Instrumentation using plug-ins • Support major web servers with minimal overhead (1-5%) • Automated byte code instrumentation of Java interfaces for J2EE applications • JSP and JDBC on major application servers • Insert byte-code into Jar files, minimal execution overheads (1-7%) • General purpose automated instrumentation technique that can instrument J2EE and potentially MS .NET interfaces • Multi-user and distributed capture, analysis and playback of workloads • Test management, data analysis • Correlation of system and application metrics with workloads
Sample Usage Scenarios • Solving production performance problems • Load testing with a view inside the application • Scaling and capacity planning • Benchmarking
Production Problem-Solving EXISTING SYSTEM DEVELOPMENT SYSTEM TEST SYSTEM PERFORMANCE SERVER Workload & Performance WORKLOAD CODE FIXES PERFORMANCE Triggers • Process: • Instrument application and deploy to production • Set workload capture thresholds using monitoring systems. Capture workload and state when problem occurs. • Playback the workload to recreate the problem in the test environment to diagnose and fix it. • Replay the workload with the code fixes to validate that the problem is solved • Deploy the fixed code to production Monitoring System OptiBench Console Monitoring Console
Conclusions • Innovative technology and shipping product • Recording, playback, and correlation of real production workload with low overheads • Addresses real performance problems for customers • Product currently being deployed at major customer sites