TDC 369 / TDC 432

1. TDC 369 / TDC 432 Spring 2003 Greg Brewster DePaul University

2. Agenda • Course Overview • What is performance analysis? • Methods for performance analysis • Measurements • Simulation • Analysis

3. Course Goals • Study analytical tools for performance analysis of web-based computer networks and systems.

4. Course Text • Required: Capacity Planning for Web Services: Metrics, Models and Methods, Menasce & Almeida, Prentice-Hall, 2002. • Excellent coverage of performance analysis issues for web services and web servers • For more math: Queueing Networks and Markov Chains: Modeling and Performance Evaluation with Computer Science Applications, Bolch, Greiner, de Meer and Trivedi, Wiley, 1998. • A good reference book covering mathematics and analysis aspects of the course in more detail

5. Office Hours and Web Site • Course Web Site: • http://dlweb.cs.depaul.edu • Class Times: MW 11:50 AM – 1:20 PM • I’ll be available in my office for 1.5 hours after each class • Office Hours: MW 1:30 – 3:00 PM

6. Course Web Site and Handouts • Class presentation handouts (PPT slides) will be available on the web site • Monday class slides on web site by 3:00 PM on Friday • Wednesday class slides on web site by 8:00 AM on Tuesday • It is your responsibility to print out copies of PPT slides for your own note-taking before coming to each class session.

7. Performance Analysis • Our goal is to determine values for performance attributes of systems, such as: • Delay (Response Time) • Elapsed time between two events • Throughput • Events per second sustained by a system • Utilization • Fraction of time a system resource is busy • Reliability • Fraction of time a system resource is available

8. Example: Web Server Performance • Response Time • What is the delay between the time a user clicks on a hyperlink and the time the web page is displayed? • Throughput • How many web page accesses per second can the server support? • Utilization • What fraction of time is the server busy? • Reliability • How often is the server down?

9. Applications of Performance Analysis Results • Capacity Planning • Predicting the resources needed to provide some Quality of Service in an existing system • Systems Design • Evaluate design tradeoffs for components of a system to be constructed • Systems Research • Quantify the value of proposed new protocols, etc.

10. Sample Applications • How many servers does Amazon.com need to provide average search times of less than 10 seconds? • How much faster would DePaul’s web access be if we doubled the bandwidth to our ISP? • Can performance of the TCP protocol be improved?

11. Capacity Planning • Capacity Planning Question: Given current system performance and demand, how much can the demand increase before • the system becomes saturated • Saturation occurs when some vital system resource nears exhaustion – example: network utilization > 95%. • or some critical system service level is exceeded • Example: user response time exceeds 5 seconds

12. Capacity Planning Example • AnyCo runs an Intranet server which currently handles about 60 requests per second and gives acceptable performance. • Management would like to hire more employees, which will increase requests/sec. • Key Service Level: Server response time must be kept under 8 seconds

13. AnyCo Reponse Time Analysis

14. Response Time Results • AnyCo finds that the web server, as currently configured, can handle up to about 150 requests per second before response times become unacceptable. • Most response time graphs have a knee value for the graph, where response time starts to increase rapidly with increasing load • For this example, the knee of the curve occurs at around 130 requests/second.

15. Server CPU Utilization Analysis

16. Utilization Results • CPU utilization defines the fraction of the time that the CPU is busy processing requests. • AnyCo finds that the web CPU utilization increases linearly with requests up to about 190 requests per second, when it reaches maximum utilization (100%). • This suggests that CPU utilization represents a bottleneckresource for this system and that increasing the server CPU speed (or installing multiple processors or multiple servers) would be necessary to increase system capacity beyond 190 requests per second.

17. AnyCo Throughput Analysis

18. Throughput Results • Results show that all requests are processed (that is, throughput = offered load) up to around 185 requests per second. • Past 185 requests per second, the system starts to drop requests (throughput < offered load), indicating that the system capacity has been exceeded.

19. Performance Analysis Approaches • Systems Measurement • Simulation • Analytical Modeling

20. Comparison From: Tools for Designing Data Networks, Al Edlund, Data Communications Review, Dec. 1999.

21. Systems Measurement • Idea: Directly measure the performance of an existing physical system • Tools: • Hardware analyzers • Network “sniffers” • Metered software

22. Systems Measurement • Advantage • Highly accurate performance results • Disadvantages • System must exist and be measureable • Cannot measure proposed systems or modifications to existing systems • Measurement may interfere with system operation • Generalizeable? Results may apply only to particular time period measured

23. Measurements Marketplace • Measurement Tools • Traffic generators, Sniffers, probes, etc. • Network General, Computer Associates, Microsoft, etc • Measurement Services • Companies will generate traffic to test your web site • Gomez, Keynote Systems, Matrix NetSystems

24. Simulation • Idea: Utilize simulation software that calculates progressive system states and “measures” resulting performance values • Tools: • Simulation packages, e.g. COMNET • Custom software

25. Simulation • Advantages • Can measure performance of systems that do not yet exist • Accurate results if specifications known • Disadvantages • Software is complex to write and debug • Simplifying assumptions are typically required • Simulation must be run for thousands or millions of iterations for accurate results, which may require hours of simulation execution time.

26. Simulation Marketplace • Simulation Packages • NS (network simulator) • Free software from Berkeley • OPNET • www.opnet.com • COMNET • www.compuware.com

27. Analytical Models • Idea: Represent system performance via mathematical model that can be solved directly • Tools: • Probability Theory • Queueing Theory • Model solution packages, e.g. GreatSPN, PMVA • Mathematical software, e.g. MatLab or Mathematica

28. Analytical Models • Advantages • Can analyze systems that do not yet exist • Results are derived much more quickly compared with simulation • Disadvantages • May be difficult to determine appropriate models to use • Simplifying assumptions are used (more than simulation) to ensure that solution is mathematically feasible

29. Analysis Marketplace • Special purpose software packages exist • MMNK software for queue delay analysis • MVA software for queueing network analysis • Petri Net software packages • Markov chain solvers • Once a model is understood, some modeling and analysis can be done with basic tools, such as • Excel • Mathematica • Companies will sell you analysis services • NetForeCast.com and others

30. Example: AnyCo Web Server • AnyCo runs a web server and would like to support up to 100 users performing transactions on this server at any one time • Key criterion: User response time must be less than 8 seconds

31. AnyCo Web Server Example

32. AnyCo Web Server • AnyCo workload characterization: • Each active user reads a web page for an average of 5 seconds before clicking a new hyperlink • HTTP Request message length averages 200 bytes • HTTP Responses (web pages) average 40,000 bytes • Each request requires • 80 ms. of Web server CPU time • 3 disk accesses – avg. 50 ms. per access

33. AnyCo Web ServerSystems Measurement Approach • AnyCo sets up test terminals that generate transactions on the web server. • They generate traffic equivalent to 1, 10, 20, 30, …, 100 users using test terminals and measure response times. • Each set of measurements is repeated many times to ensure a statistically significant result.

34. AnyCo Measurements

35. AnyCo Web Server ExampleSystems Measurement Approach • Result: Response time exceeds 8 seconds for more than 60 users • Follow-Up: • Look for causes of unacceptable performance: • Detailed server code measurements • Network utilization measurements • Problem: How to predict impact of any system modifications?

36. AnyCo Web ServerSystems Measurement Approach • Effort and Expense are High: • Test measurement software must be purchased and configured on test workstations • Server must be off-line and available for measurement • Network Sniffer or probe must be installed to capture performance information

37. AnyCo Web ServerSimulation Approach • AnyCo creates a detailed simulation of their web server system. • Key system parameters are used as inputs to the simulation: • Data packet sizes, overheads • Router delays, • CPU time required for various operations • Setting up / writing simulation may take many hours or days depending on software used and complexity of simulation.

38. AnyCo Web ServerSimulation Approach • Assumptions: • Delay components: • ISP router, T1 access line, AnyCo router, Ethernet, Web Server CPU time, Web Server disk access times, Ethernet, AnyCo router, T1 access line, ISP router. • Since we do not know details of user access to the Internet, we do not include delay for transporting data over user access lines • Router processing times negligible (only include data transmission times) • Ethernet transmission times negligible

39. AnyCo Web ServerSimulation Approach • Simulation is now run: • Each point on performance graph requires one simulation run. • Simulation run must iterate thousands or millions of times to get a statistically valid result. • Each simulation run may typically take 10 minutes to one hour to run.

40. AnyCo Web ServerSimulation Model Example: Graphical model, simulation and results analysis using COMNET software.

41. AnyCo Web ServerSimulation Approach • Model Accuracy: How accurately does this simulation model capture the behavior of the real system?? • Must be addressed via model validation study where simulation results are compared against measured system results. • Follow-Up: After examining simulation results, we can modify parameters and do what-if analysis for system changes.

42. AnyCo Web ServerAnalysis Approach • AnyCo sets up a simple analytical model of their web server system. • Key system parameters are used as inputs to the simulation: • Data packet sizes, overheads • Router delays, • CPU time required for various operations • Model can typically be set up quickly, as it only captures the few essential performance attributes of the system

43. AnyCo Web ServerAnalysis Approach • Assumptions: • Delay components: • ISP router, T1 access line, AnyCo router, Ethernet, Web Server CPU time, Web Server disk access times, Ethernet, AnyCo router, T1 access line, ISP router. • Since we do not know details of user access to the Internet, we do not include delay for transporting data over user access lines • Router processing times negligible (only include data transmission times) • Ethernet transmission times negligible • There are some restrictions on distributions of service times and request routing (details later in course)

44. AnyCo Web ServerAnalytical Approach • Analytical Solution package is run: • Each point on performance graph requires one solution of an analytical model. • Each analytical model solution is exact (just 1 iteration) and takes about 0.1 second. • All graph data generated in less than 1 second!!

45. Analytical Models • The fundamental building block for many analytical models is Queueing Theory

46. Queueing Theory • Queueing Theory deals with the behavior of jobs arriving to queues connected to servers.

47. Queue Operations • Jobs arrive according an arrival process, A(t). • Arriving jobs are placed in queue • Server provides service to first job in queue, other jobs wait their turn. • Service time required by each job determined by service process, S(t) • Each job leaves queue after receiving service.

48. AnyCo Web Server Example

49. AnyCo Queueing Network Model

50. AnyCo Web ServerQueueing Network Modelmore details later in course!! • Each time a user clicks a hyperlink: • Request message sent by ISP router across T1 (~0.001 s) and across Ethernet • Web server processes request using 3 disk accesses (0.05 s. each) interspersed with 4 CPU time slots (0.02 s. each = 0.08 s.) • Response message sent across Ethernet and by AnyCo router across T1 (~0.02 s) • User examines result for average of 5 sec before making next request.