1 / 108

The Performance Bottleneck Application, Computer, or Network

The Performance Bottleneck Application, Computer, or Network Richard Carlson Internet2 Part 1 Outline Why there is a problem What can be done to find/fix problems Tools you can use Ramblings on what’s next Basic Premise Application’s performance should meet your expectations!

Download Presentation

The Performance Bottleneck Application, Computer, or Network

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. The Performance BottleneckApplication, Computer, or Network Richard Carlson Internet2 Part 1

  2. Outline • Why there is a problem • What can be done to find/fix problems • Tools you can use • Ramblings on what’s next

  3. Basic Premise • Application’s performance should meet your expectations! • If they don’t you should complain!

  4. Questions • How many times have you said: • What’s wrong with the network? • Why is the network so slow? • Do you have any way to find out? • Tools to check local host • Tools to check local network • Tools to check end-to-end path

  5. Underlying Assumption • When problems exist, it’s the networks fault!

  6. NDT Demo First

  7. Simple Network Picture Bob’s Host Network Infrastructure Carol’s Host

  8. Network Infrastructure Switch 2 Switch 3 R5 R4 R8 R1 R3 R6 Switch 1 R9 R2 R7 Switch 4

  9. Possible Bottlenecks • Network infrastructure • Host computer • Application design

  10. Network Infrastructure Bottlenecks • Links too small • Using standard Ethernet instead of FastEthernet • Links congested • Too many hosts crossing this link • Scenic routing • End-to-end path is longer than it needs to be • Broken equipment • Bad NIC, broken wire/cable, cross-talk • Administrative restrictions • Firewalls, Filters, shapers, restrictors

  11. Host Computer Bottlenecks • CPU utilization • What else is the processor doing? • Memory limitations • Main memory and network buffers • I/O bus speed • Getting data into and out of the NIC • Disk access speed

  12. Application Behavior Bottlenecks • Chatty protocol • Lots of short messages between peers • High reliability protocol • Send packet and wait for reply before continuing • No run-time tuning options • Use only default settings • Blaster protocol • Ignore congestion control feedback

  13. TCP 101 • Transmission Control Protocol (TCP) • Provides applications with a reliable in-order delivery service • The most widely used Internet transport protocol • Web, File transfers, email, P2P, Remote login • User Datagram Protocol (UDP) • Provides applications with an unreliable delivery service • RTP, DNS

  14. Summary – Part 1 • Problems can exist at multiple levels • Network infrastructure • Host computer • Application design • Multiple problems can exist at the same time • All problems must be found and fixed before things get better

  15. Summary – Part 2 • Every problem exhibits the same symptom • The application performance doesn’t meet the users expectations!

  16. Outline • Why there is a problem • What can be done to find/fix problems • Tools you can use • Ramblings on what’s next

  17. Real Life Examples • I know what the problem is • Bulk transfer with multiple problems

  18. Example 1 - SC’04 experience • Booth having trouble getting application to run from Amsterdam to Pittsburgh • Tests between remote SGI and local PC showed throughput limited to < 20 Mbps • Assumption is: PC buffers too small • Question: How do we set WinXP send/receive window size

  19. SC’04 Determine WinXP info http://www.dslreports.com/drtcp

  20. SC’04 Confirm PC settings • DrTCP reported 16 MB buffers, but test program still slow, Q: How to confirm? • Run test to SC NDT server (PC has Fast Ethernet Connection) • Client-to-Server: 90 Mbps • Server-to-Client: 95 Mbps • PC Send/Recv window size: 16 Mbytes (wscale 8) • NDT Send/Recv window Size: 8 Mbytes (wscale 7) • Reported TCP RTT: 46.2 msec • approximately 600 Kbytes of data in TCP buffer • Min window size / RTT: 1.3 Gbps

  21. SC’04 Local PC Configured OK • No problem found • Able to run at line rate • Confirmed that PC’s TCP window values were set correctly

  22. SC’04 Remote SGI • Run test from remote SGI to SC show floor (SGI is Gigabit Ethernet connected). • Client-to-Server: 17 Mbps • Server-to-Client: 16 Mbps • SGI Send/Recv window size: 256 Kbytes (wscale 3) • NDT Send/Recv window Size: 8 Mbytes (wscale 7) • Reported RTT: 106.7 msec • Min window size / RTT: 19 Mbps

  23. SC’04 Remote SGI Results • Needed to download and compile command line client • SGI TCP window is too small to fill transatlantic pipe (19 Mbps max) • User reluctant to make changes to SGI network interface from SC show floor • NDT client tool allows application to change buffer (setsockopt() function call)

  24. SC’04 Remote SGI (tuned) • Re-run test from remote SGI to SC show floor. • Client-to-Server: 107 Mbps • Server-to-Client: 109 Mbps • SGI Send/Recv window size: 2 Mbytes (wscale 5) • NDT Send/Recv window Size: 8 Mbytes (wscale 7) • Reported RTT: 104 msec • Min window size / RTT: 153.8 Mbps

  25. SC’04 Debugging Results • Team spent over 1 hour looking at Win XP config, trying to verify window size • Single NDT test verified this in under 30 seconds • 10 minutes to download and install NDT client on SGI • 15 minutes to discuss options and run client test with set buffer option

  26. SC’04 Debugging Results • 8 Minutes to find SGI limits and determine maximum allowable window setting (2 MB) • Total time 34 minutes to verify problem was with remote SGIs’ TCP send/receive window size • Network path verified but Application still performed poorly until it was also tuned

  27. Example 2 – SCP file transfer • Bob and Carol are collaborating on a project. Bob needs to send a copy of the data (50 MB) to Carol every ½ hour. Bob and Carol are 2,000 miles apart. How long should each transfer take? • 5 minutes? • 1 minute? • 5 seconds?

  28. What should we expect? • Assumptions: • 100 Mbps Fast Ethernet is the slowest link • 50 msec round trip time • Bob & Carol calculate: • 50 MB * 8 = 400 Mbits • 400 Mb / 100 Mb/sec = 4 seconds

  29. Initial SCP Test Results

  30. Initial Test Results • This is unacceptable! • First look for network infrastructure problem • Use NDT tester to examine both hosts

  31. Initial NDT testing shows Duplex Mismatch at one end

  32. NDT Found Duplex Mismatch • Investigating this it is found that the switch port is configured for 100 Mbps Full-Duplex operation. • Network administrator corrects configuration and asks for re-test

  33. Duplex Mismatch Corrected

  34. SCP results after Duplex Mismatch Corrected

  35. Intermediate Results • Time dropped from 18 minutes to 40 seconds. • But our calculations said it should take 4 seconds! • 400 Mb / 40 sec = 10 Mbps • Why are we limited to 10 Mbps? • Are you satisfied with 1/10th of the possible performance?

  36. Default TCP window settings

  37. Calculating the Window Size • Remember Bob found the round-trip time was 50 msec • Calculate window size limit • 85.3KB * 8 b/B = 698777 b • 698777 b / .050 s = 13.98 Mbps • Calculate new window size • (100 Mb/s * .050 s) / 8 b/B = 610.3 KB • Use 1MB as a minimum

  38. Resetting Window Value

  39. With TCP windows tuned

  40. Steps so far • Found and fixed Duplex Mismatch • Network Infrastructure problem • Found and fixed TCP window values • Host configuration problem • Are we done yet?

  41. SCP results with tuned windows

  42. Intermediate Results • SCP still runs slower than expected • Hint: SCP uses internal buffers • Patch available from PSC

  43. SCP Results with tuned SCP

  44. Final Results • Fixed infrastructure problem • Fixed host configuration problem • Fixed Application configuration problem • Achieved target time of 4 seconds to transfer 50 MB file over 2000 miles

  45. Why is it hard to Find/Fix Problems? • Network infrastructure is complex • Network infrastructure is shared • Network infrastructure consists of multiple components

  46. Shared Infrastructure • Other applications accessing the network • Remote disk access • Automatic email checking • Heartbeat facilities • Other computers are attached to the closet switch • Uplink to campus infrastructure • Other users on and off site • Uplink from campus to gigapop/backbone

  47. Other Network Components • DHCP (Dynamic Host Resolution Protocol) • At least 2 packets exchanged to configure your host • DNS (Domain Name Resolution) • At least 2 packets exchanged to translate FQDN into IP address • Network Security Devices • Intrusion Detection, VPN, Firewall

  48. Network Infrastructure • Large complex system with potentially many problem areas

  49. Why is it hard to Find/Fix Problems? • Computers have multiple components • Each Operating System (OS) has a unique set of tools to tune the network stack • Application Appliances come with few knobs and limited options

  50. Computer Components • Main CPU (clock speed) • Front & Back side bus • Main Memory • I/O Bus (ATA, SCSI, SATA) • Disk (access speed and size)

More Related