the performance bottleneck application computer or network n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
The Performance Bottleneck Application, Computer, or Network PowerPoint Presentation
Download Presentation
The Performance Bottleneck Application, Computer, or Network

Loading in 2 Seconds...

play fullscreen
1 / 57

The Performance Bottleneck Application, Computer, or Network - PowerPoint PPT Presentation


  • 98 Views
  • Uploaded on

The Performance Bottleneck Application, Computer, or Network. Richard Carlson Internet2 Part 3. Outline. Why there is a problem What can be done to find/fix problems Tools you can use Ramblings on what’s next. Active Measurement Tools.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'The Performance Bottleneck Application, Computer, or Network' - lauren


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
outline
Outline
  • Why there is a problem
  • What can be done to find/fix problems
  • Tools you can use
  • Ramblings on what’s next
active measurement tools
Active Measurement Tools
  • Tools that inject packets into the network to measure some value
    • Available Bandwidth
    • Delay/Jitter
    • Loss
  • Requires bi-directional traffic or synchronized hosts
passive measurement tools
Passive Measurement Tools
  • Tools that monitor existing traffic on the network and extract some information
    • Bandwidth used
    • Jitter
    • Loss rate
  • May generate some privacy and/or security concerns
tools tools tools
Tools, Tools, Tools
  • Ping
  • Traceroute
  • Iperf
  • Tcpdump
  • Tcptrace
  • BWCTL
  • NDT
  • OWAMP
  • AMP
  • Advisor
  • Thrulay
  • Web100
  • MonaLisa
  • pathchar
  • NPAD
  • Pathdiag
  • Surveyor
  • Ethereal
  • CoralReef
  • MRTG
  • Skitter
  • Cflowd
  • Cricket
  • Net100
network diagnostic tool ndt
Network Diagnostic Tool (NDT)
  • Measure performance to users desktop
  • Identify real problems for real users
    • Network infrastructure is the problem
    • Host tuning issues are the problem
  • Make tool simple to use and understand
  • Make tool useful for users and network administrators
slide7

Different Host, same switch port

  • 10 Mbps NIC
    • Throughput 6.8/6.7 mbps send/receive
    • RTT 20 ms
    • Retransmission/Timeouts 25/3
  • 100 Mbps NIC
    • Throughput 84/86 mbps send/receive
    • RTT 10 ms
    • Retransmission/Timeouts 0/0
web100 project
Web100 Project
  • Joint PSC/NCAR project funded by NSF
  • ‘First step’ to gather TCP data
    • Kernel Instrument Set (KIS)
  • Developed patches Linux kernel
  • Geared toward wide area network performance
  • Future steps will automate tuning to improve application performance
ndt s web100 based approach
NDT’s Web100 Based Approach
  • Simple bi-directional test to gather E2E data
  • Gather multiple data variables (a breadth of measurements)
  • Compare measured performance to analytical values
  • Translate network values into plain text messages
  • Geared toward campus area network
sc 04 real life example
SC’04 Real Life Example
  • Booth having trouble getting application to run from Amsterdam to Pittsburgh
  • Tests between remote SGI and local PC showed throughput limited to < 20 Mbps
  • Assumption is: PC buffers too small
  • Question: How do we set WinXP send/receive buffer
sc 04 determine winxp info
SC’04 Determine WinXP info

http://www.dslreports.com/drtcp

sc 04 confirm pc settings
SC’04 Confirm PC settings
  • DrTCP reported 16 MB buffers, but test program still slow, Q: How to confirm?
  • Run test to SC NDT server (PC has Fast Ethernet Connection)
    • Client-to-Server: 90 Mbps
    • Server-to-Client: 95 Mbps
    • PC Send/Recv Buffer size: 16 Mbytes (wscale 8)
    • NDT Send/Recv Buffer Size: 8 Mbytes (wscale 7)
    • Reported TCP RTT: 46.2 msec
      • approximately 600 Kbytes of data in TCP buffer
    • Min buffer size / RTT: 1.3 Gbps
sc 04 local pc configured ok
SC’04 Local PC Configured OK
  • No problem found
  • Able to run at line rate
  • Confirmed that PC’s TCP buffers were set correctly
sc 04 remote sgi
SC’04 Remote SGI
  • Run test from remote SGI to SC show floor (SGI is Gigabit Ethernet connected).
    • Client-to-Server: 17 Mbps
    • Server-to-Client: 16 Mbps
    • SGI Send/Recv Buffer size: 256 Kbytes (wscale 3)
    • NDT Send/Recv Buffer Size: 8 Mbytes (wscale 7)
    • Reported RTT: 106.7 msec
    • Min Buffer size / RTT: 19 Mbps
sc 04 remote sgi results
SC’04 Remote SGI Results
  • Needed to download and compile command line client
  • SGI TCP buffer is too small to fill transatlantic pipe (19 Mbps max)
  • User reluctant to make changes to SGI network interface from SC show floor
  • NDT client tool allows application to change buffer (setsockopt() function call)
sc 04 remote sgi tuned
SC’04 Remote SGI (tuned)
  • Re-run test from remote SGI to SC show floor.
    • Client-to-Server: 107 Mbps
    • Server-to-Client: 109 Mbps
    • SGI Send/Recv Buffer size: 2 Mbytes (wscale 5)
    • NDT Send/Recv Buffer Size: 8 Mbytes (wscale 7)
    • Reported RTT: 104 msec
    • Min Buffer size / RTT: 153.8 Mbps
sc 04 debugging results
SC’04 Debugging Results
  • Team spent over 1 hour looking at Win XP config, trying to verify Buffer size
  • Single NDT test verified this in under 30 seconds
  • 10 minutes to download and install NDT client on SGI
  • 15 minutes to discuss options and run client test with set buffer option
sc 04 debugging results1
SC’04 Debugging Results
  • 8 Minutes to find SGI limits and determine maximum allowable buffer setting (2 MB)
  • Total time 34 minutes to verify problem was with remote SGIs’ TCP send/receive buffer size
  • Network path verified but Application still performed poorly until it was also tuned
ndt benefits
NDT Benefits
  • End-user based view of network
  • Can be used to identify performance bottlenecks (could be host problem)
  • Provides some ‘hard evidence’ to users and network administrators to reduce finger pointing
  • Doesn’t rely on historical data
web based performance tool
Web Based Performance tool
  • Operates on Any client with a Java enabled Web browser
  • What it can do
    • Positively state if Sender, Receiver, or Network is operating properly
    • Provide accurate application tuning info
    • Suggest changes to improve performance
web base performance tool
Web base Performance tool
  • What it can’t do
    • Tell you where in the network the problem is
    • Tell you how other servers perform
    • Tell you how other clients will perform
ndt methodology
NDT methodology
  • Identify specific problem(s) that affect end users
  • Analyze problem to determine ‘Network Signature’ for this problem
  • Provide testing tool to automate detection process
duplex mismatch detection
Duplex Mismatch Detection
  • Developing analytical model to describe how network operates (no prior art?)
  • Expanding model to describe UDP and TCP flows
  • Test models in LAN, MAN, and WAN environments

NIH/NLM grant funding

test environment
Test environment

NDT

Srv

NDT

Clt

100 Mbps

Full Duplex

100 Mbps

Mismatch

Switch

Receiver

Source

  • Receiver is put is various states
    • Switch = full & Host = full or half
    • Switch = half & Host = full or half
ethernet transmission strategy
Ethernet transmission strategy
  • Half Duplex
    • Use carrier sense signal to determine if link in use
    • If not, send frame at head of queue
    • Else, wait for frame to end and send frame
    • Use collision detection signal to determine if other station also sends
  • Full Duplex
    • Send packet at head of queue
    • Disable carrier sense
    • Disable collision detection
tcp operation on lan
TCP Operation on LAN
  • Observed behavior depends on direction of TCP flow and direction of mismatch
    • Data and ACK packets delivered
    • Data packets lost and ACKs delayed
    • ACKs packets lost and Data delayed
  • Losing ACKs has bigger effect than losing Data packets
  • Web100 details are only available when NDT server is source and client is sink
four cases of duplex setting
Four Cases of Duplex Setting

FD-FD

FD-HD

HD-FD

HD-HD

tentative mismatch detection
Tentative Mismatch Detection
  • Full to Half mismatch detection
    • Large percentage of duplicate ACKs
    • Connection spends majority of the time in CwndLimited state
    • Asymmetric throughput
      • opposite direction is less
tentative mismatch detection1
Tentative Mismatch Detection
  • Half to Full mismatch detection
    • Large number of timeouts causes long idle time (RTO x timeout value)
    • Connection spends majority of the time in CwndLimited state
    • Asymmetric throughput
      • opposite direction is greater
bottleneck link detection
Bottleneck Link Detection
  • What is the slowest link in the end-2-end path?
    • Monitors packet arrival times using libpacp routine
    • Use TCP dynamics to create packet pairs
    • Quantize results into link type bins (no fractional or bonded links)

Cisco URP grant work

normal congestion detection
Normal congestion detection
  • Shared network infrastructures will cause periodic congestion episodes
    • Detect/report when TCP throughput is limited by cross traffic
    • Detect/report when TCP throughput is limited by own traffic
faulty hardware link detection
Faulty Hardware/Link Detection
  • Detect non-congestive loss due to
    • Faulty NIC/switch interface
    • Bad Cat-5 cable
    • Dirty optical connector
  • Preliminary works shows that it is possible to distinguish between congestive and non-congestive loss
full half link duplex setting
Full/Half Link Duplex setting
  • Detect half-duplex link in E2E path
    • Identify when throughput is limited by half-duplex operations
  • Preliminary work shows detection possible when link transitions between blocking states
additional functions and features
Additional Functions and Features
  • Provide basic tuning information
  • Basic Features
    • Basic configuration file
    • FIFO scheduling of tests
    • Simple server discovery protocol
    • Federation mode support
    • Command line client support
  • Created sourceforge.net project page
internet2 pipes project
Internet2 piPEs Project
  • Develop E2E measurement infrastructure capable of finding network problems
  • Tools include
    • BWCTL: Bandwidth Control wrapper for NLANR Iperf
    • OWAMP: One-Way Active Measurement
    • NDT: Network Diagnostic Tool
bwctl design goals
BWCTL Design Goals
  • Bandwidth Control Server
  • Wrapper for Dast Iperf tool
  • Performs scheduled tests between 11 peers
  • Supports on-demand tests between peer nodes
specific difficulties
Specific difficulties

UDP

  • Iperf doesn’t always send at requested rate
  • Iperf sender hangs (likely Linux/iperf interaction – could be due to signal handling of the bwctl level)
  • End of session is difficult to detect, which is problematic for a “scheduled” timeslot
  • Iperf sometimes takes large amounts of time to finish
specific difficulties1
Specific difficulties

TCP

  • Large pipe to small pipe
    • Launch a large window
    • Test waits until completion
    • Terminate test to remain within schedule
    • Þ Sets of incomplete tests to interpret
  • Full mesh presents difficulties for window size selection (and other path specific characteristics)
    • bwctl uses the peer to peer server connection to deduce a “reasonable” window
    • If at all possible path specific parameters need to be dynamically configured
future possibilities
Future Possibilities
  • Server-less client side for end hosts
  • Closer integration with test engine (iperf API?)
    • Better error detection
    • Better timing control (begin and end of test is currently a problem)
  • 3-party tests (client not on one of the endpoints)
  • Open source development
availability
Availability
  • Beta version currently available

www.internet2.edu/bwctl/

Mail lists:

  • bwctl-users
  • bwctl-announce

https://mail.internet2.edu/wws/lists/engineering

owamp design goals
OWAMP Design Goals
  • One-Way-Active-Measurement-Protocol
    • Possible due to growing availability of good time sources
    • Wide deployment of “open” servers would allow measurement of one-way delay to become as commonplace as measurement of RTT using ICMP tools such as ping.
    • Current Draft: draft-ietf-ippm-owdp-07.txt
      • Shalunov,Teitelbaum,Karp,Boote,Zekauskas
      • RFC just released
    • Sample implementation under development

Alpha code currently available

abilene owamp deployment
Abilene OWAMP deployment
  • 2 overlapping full meshes (IPv4 & IPv6)
    • 11 measurement nodes = 220 ongoing tests
    • UDP singletons
    • Rate: 10 packets/second*
    • Packetsize: (32 byte payload)*
    • Results are continuously streamed back to “Measurement Portal” for long-term archive and data dissemination (Near real-time)
owamp errors
OWAMP Errors
  • Preliminary Findings:
    • Min error estimates look to be in the 55-60 usec range.
    • Serialization Delay: ~5usec x 2
    • Get Timestamp: ~15usec x 2
    • Additional error is:
      • Time from userland “send” to 1st byte hits the wire
      • Time from kernel has packet to userland “recv” returns
      • Potentially recv process data processing before calling “recv”
owamp implementation status
OWAMP implementation status
  • Sample implementation
  • http://e2epi.internet2.edu/owamp/
    • Alpha Release ver 1.6c:
      • No “policy”
      • No authentication/encryption
      • Tested on FreeBSD & Linux
sample traceroute tree map
Sample Traceroute Tree map

S1

R1

R2

S2

S3

R7

R6

R4

R3

R5

S7

S8

S4

S5

S6

sample traceroute from s1 to client
Sample Traceroute from S1 to Client

S1

Client

R1

Rc

R4

Ra

R5

Rb

Traceroute to Client

sample traceroute tree map1
Sample Traceroute Tree map

S1

Client

R1

R1

R2

Rc

R4

R7

R6

R4

R3

Ra

R5

R5

Rb

S1

S2

S3

S7

S8

S4

S5

S6