Sizing Router Buffers - PowerPoint PPT Presentation

Sizing router buffers
1 / 53

  • Uploaded on
  • Presentation posted in: General

Sizing Router Buffers. Guido Appenzeller Thesis Defense May 24 th , 2004. Routers need Packet Buffers. It’s well known that routers need packet buffers It’s less clear why and how much Goal of this work is to answer the question: How much buffering do routers need?.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Sizing Router Buffers

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Sizing router buffers

Sizing Router Buffers

Guido Appenzeller

Thesis DefenseMay 24th, 2004

Routers need packet buffers

Routers need Packet Buffers

  • It’s well known that routers need packet buffers

    • It’s less clear why and how much

  • Goal of this work is to answer the question:

    How much buffering do routers need?

How much buffer does a router need

How much Buffer does a Router need?





  • Universally applied rule-of-thumb:

    • A router needs a buffer size:

      • 2T is the two-way propagation delay (or just 250ms)

      • C is capacity of bottleneck link

  • Context

    • Mandated in backbone and edge routers.

    • Appears in RFPs and IETF architectural guidelines..

    • Usually referenced to Villamizar and Song: “High Performance TCP in ANSNET”, CCR, 1994.

    • Already known by inventors of TCP [Van Jacobson, 1988]

    • Has major consequences for router design




  • 10Gb/s linecard

    • Requires 300Mbytes of buffering.

    • Read and write 40 byte packet every 32ns.

  • Memory technologies

    • DRAM: require 4 devices, but too slow.

    • SRAM: require 80 devices, 1kW, $2000.

  • Problem gets harder at 40Gb/s

    • Hence RLDRAM, FCRAM, etc.

Outline of this work

Outline of this Work

  • Main Results

    • The rule of thumb is wrong for a core routers today

    • Required buffer is instead of

  • Outline of this talk

    • Where the rule of thumb comes from

    • Why it is incorrect for a core router in the internet today

    • Correct buffer requirements for a congested router

    • Buffer requirements for short flows (slow-start)

    • Experimental Verification

    • Conclusion



  • The Rule of Thumb

    • Where does the rule of thumb comes from? (Answer: TCP)

    • Interaction of TCP flows and a router buffers

  • The buffer requirements for a congested router

  • Buffer requirements for short flows (slow-start)

  • Experimental Verification

  • Conclusion

Sizing router buffers



Only W=2packets may be outstanding



C’ > C


  • TCP Congestion Window controls the sending rate

    • Sender sends packets, receiver sends ACKs

    • Sending rate is controlled by Window W,

    • At any time, only W unacknowledged packets may be outstanding

    • The sending rate of TCP is

Single tcp flow router without buffers

Link not fully utilized

Single TCP FlowRouter without buffers

Only Wpackets may be outstanding




C’ > C




  • Rule for adjusting W

    • If an ACK is received: W ← W+1/W

    • If a packet is lost:W ← W/2





Single tcp flow router with large enough buffers for full link utilization

For every W ACKs received,

send W+1 packets


Window size


Single TCP FlowRouter with large enough buffers for full link utilization




C’ > C


Required buffer is height of sawtooth

Required buffer is height of sawtooth




Buffer rule of thumb

Buffer = rule of thumb

Microscopic tcp behavior when sender pauses buffer drains

Microscopic TCP BehaviorWhen sender pauses, buffer drains

one RTT


Over buffered link

Over-buffered Link

Under buffered link

Under-buffered Link

Origin of rule of thumb

Origin of rule-of-thumb

  • Before and after reducing window size, the sending rate of theTCP sender is the same

  • Inserting the rate equation we get

  • The RTT is part transmission delay T and part queuing delay B/C . We know that after reducing the window, the queuing delay is zero.

Rule of thumb


  • Rule-of-thumb makes sense for one flow

  • Typical backbone link has > 20,000 flows

  • Does the rule-of-thumb still hold?

  • Answer:

    • If flows are perfectly synchronized, then Yes.

    • If flows are desynchronized then No.



  • The Rule of Thumb

  • The buffer requirements for a congested router

    • Synchronized flows

    • Desynchronized flows

    • The 2T×C/sqrt(n) rule

  • Buffer requirements for short flows (slow-start)

  • Experimental Verification

  • Conclusion

If flows are synchronized

If flows are synchronized

  • Aggregate window has same dynamics

  • Therefore buffer occupancy has same dynamics

  • Rule-of-thumb still holds.


When are flows synchronized

When are Flows Synchronized?

  • Small numbers of flows tend to synchronize

    • In ns2 simulation they are synchronized

    • In at least some cases holds for real networks as well

  • Large aggregates of flows are not synchronized

    • For > 500 flows, synchronization disappears in ns2

    • On a Cisco GSR, 100 flows were not synchronized

    • Measurements in the core give no indication of synchronization

      • C. Fraleigh, “Provisioning Internet Backbone Networks to support Latency Sensisitve Applications”, Ph.D. Thesis, Stanford

      • Hohn, Veitch, Papagiannaki and Diot – “Bridging Router Performance and Queuing Theory”

If flows are not synchronized



Buffer Size

If flows are not synchronized



Quantitative model

Quantitative Model

model as


  • For many de-synchronized flows

    • We assume congestion windows are independent

    • All congestion windows have the same probability distribution

  • Model congestion window of a flow as random variable

  • Now central limit theorem gives us the distribution of the sum of the window sizes

Buffer vs number of flows for a given bandwidth

Buffer vs. Number of Flowsfor a given Bandwidth

  • For a given C, the window W scales with 1/n and thus

  • Standard deviation of sum of windows decreases with n

  • If for a single flow we have

  • Thus as n increases, buffer size should decrease

Required buffer size

Required buffer size




  • Flows in the core are desynchronized

    • Substantial experimental evidence

    • Supported by ns2 simulations

  • For desynchronized flows, routers need only buffers of



  • The Rule of Thumb

  • The buffer requirements for a congested router

  • Buffer requirements for short flows (slow-start)

    • M/G/1 Model

  • Experimental Verification

  • Conclusion

Short flows

Short Flows

  • So far we were assuming a congested router with long flows in congestion avoidance mode.

    • What about flows in slow start?

    • Do buffer requirements differ?

  • Answer: Yes, however:

    • Required buffer in such cases is independent of line speed and RTT (same for 1Mbit/s or 40 Gbit/s)

    • In mixes of flows, long flows drive buffer requirements

      • Short flow result relevant for uncongested routers

A single short lived tcp flow flow length 62 packets rtt 140 ms

A single, short-lived TCP flowFlow length 62 packets, RTT ~140 ms


Flow Completion Time (FCT)




fin ackreceived




Modelling short flows

Modelling Short Flows

Idea: Find buffer size by modelling queue behaviour

  • Problem: Arrival process is hard to model

    • Simplify by modelling bursts as independent

    • Buffer empties several times during one RTT

Poisson arrivals offlows

Service time is Lflow, the flow length in packets

Poisson arrivals of bursts

Service time is the lengthof the burst: 2,4,8,16…

M g 1 model for short flows

M/G/1 Model for short flows

  • TCP flows generate independent bursts

    • Service times is burst length :

    • Poisson arrivals of rate

  • To verify if this approach works, let’s compare the average queue length in the model and in simulation

Average queue length

Average Queue length

Buffer requirements for short flows

Buffer B


Packet Loss

P(Q = x)

Buffer Requirements for Short Flows

  • Buffers absorb fluctuations in queue, reduce packet loss

    • Reduce retransmits, Timeouts and thereby flow completion time

    • Utilization not a good measure of QoS as load << 1

  • We can find a good upper bound for loss, if we have the queue length distribution for an infinite buffer

    • If a packet arrives and queue length is shorter than buffer, packet will not be dropped

  • Problem: For M/G/1 there is no closed form expression for the queue distribution

Queue distribution

Queue Distribution

We derived closed-form estimates of the queue

distribution using Effective Bandwidth

  • Gives very good closed form approximation

Short flow summary

Short Flow Summary

  • Buffer requirements for short flows

    • Can be modeled by M/G/1 model

    • Only depends on load and burst size distribution

    • Example - for bursts of up to size 16 at load 0.8

      • For 1% loss probability B = 115 Packets

      • For 0.01% loss probability B = 230 packets etc.

      • Bursts of size 12 is maximum for Windows XP

    • Independent of line speed and RTT

  • In mixes of flows, long flows dominate buffer requirements



  • The Rule of Thumb

  • The buffer requirements for a congested router

  • Buffer requirements for short flows (slow-start)

  • Experimental Verification

  • Conclusion

Experimental evaluation overview

Experimental Evaluation Overview

  • Simulation with ns2

    • Over 10,000 simulations that cover range of settings

      • Simulation time 30s to 5 minutes

      • Bandwidth 10 Mb/s - 1 Gb/s

      • Latency 20ms -250 ms,

  • Physical router

    • Cisco GSR with OC3 line card

    • In collaboration with University of Wisconsin

  • Experimental results presented here

    • Long Flows - Utilization

    • Mixes of flows - Flow Completion Time (FCT)

    • Mixes of flows - Heavy Tailed Flow Distribution

    • Short Flows – Queue Distribution

Long flows utilization i small buffers are sufficient oc3 line 100ms rtt

Long Flows - Utilization (I)Small Buffers are sufficient - OC3 Line, ~100ms RTT




Long flows utilization ii model vs ns2 vs physical router gsr 12000 oc3 line card

Long Flows – Utilization (II) Model vs. ns2 vs. Physical RouterGSR 12000, OC3 Line Card

Mixes of flows flow completion time fct of 14 packet flows that share a link with long lived flows

Mixes of FlowsFlow Completion TimeFCT of 14 packet flows that share a link with long-lived flows.

Heavy tailed flow length distribution

Heavy-tailed flow length distribution

  • Experiment

    • Flow arrivals are a Poisson process

    • Flow lengths are Pareto distributed

  • Results

    • Buffers in the order of are still sufficient

    • Number of “long-lived” flows n is now defined as number of flows in congestion avoidance mode

Pareto flow distribution finding the number of flows

Pareto Flow DistributionFinding the number of flows

No. of flows in CA mode

For buffer sizing, pick n = 100

time [seconds]

Pareto flow distribution

Pareto Flow Distribution

Flow arrivalson link 1





Short flows queue distribution m g 1 model vs gsr 12000 oc3 line card

Short Flows – Queue DistributionM/G/1 Model vs. GSR 12000, OC3 Line Card



  • The Rule of Thumb

  • The buffer requirements for a congested router

  • Buffer requirements for short flows (slow-start)

  • Experimental Verification

  • Conclusion

Related work

Related Work

  • Related Publications

    • Buffer sizing

      • “High Speed TCP in ANS Net” - Villamizar and Song, ACCR 1994

      • “TCP behaviour with many flows” – R. Morris, IEEE ICNP 1997

      • “Scalable TCP congestion control” – R. Morris, INFOCOM 2000

    • Queue Modelling

      • “Modelling, Simulation and Measurement of Queuing Delay” – Garetto and Towsley, SIGMETRICS 2003

Original contributions

Original Contributions

  • Main original contributions of this work

    • Routers only require buffers of instead of

    • Models on TCP buffer interaction for

      • Congestion avoidance mode

      • Slow start

    • Experimental Verification

  • Publication

    • “Sizing Router Buffers – Guido Appenzeller, Isaac Keslassy and Nick McKeown, to appear at SIGCOMM 2004

The commercial internet today

The Commercial Internet Today

  • Today’s internet differs from our assumptions

    • Core is overprovisioned, almost never congested

    • Access links are usually the bottleneck (DSL, Modem)

    • Flows are usually limited by Maximum Window size

      • Maximum Window is from 6-12 (Windows) to 42 (Unix)

    • Are the results still relevant?

  • Answer: Yes, we were intentionally pessimistic

    • Routers still needs to work in case of congestion

      • Even if this “worst case” scenario is rare

    • Slow access links and small TCP windows reduce buffer requirements further

      • Bursts are smoothed out

      • Converges towards constant rate source, poisson packet arrivals

      • We verified reduced buffer requirements experimentally

How much buffer does a router need1

How much buffer does a router need?

The old “Rule-of-Thumb”

Our Contribution

Impact on router design

Impact on Router Design

  • 10Gb/s linecard with 200,000 x 56kb/s flows

    • Rule-of-thumb: Buffer = 2.5Gbits

      • Requires external, slow DRAM

    • Becomes: Buffer = 6Mbits

      • Can use on-chip, fast SRAM

      • Completion time halved for short-flows

  • 40Gb/s linecard with 40,000 x 1Mb/s flows

    • Rule-of-thumb: Buffer = 10Gbits

    • Becomes: Buffer = 50Mbits





Two tcp flows two flows sharing a bottleneck synchronize

Two TCP FlowsTwo flows sharing a bottleneck synchronize

Impact on protocol design

Impact on Protocol Design

TCP Reno: requires

General: requires

  • The ‘rule-of-thumb’ is due to the nature of TCP

    • We can easily build a TCP that requires less buffers

    • Latency based TCP (Vegas, FAST) suffer from smaller buffers

      • Queuing delay too small to be used as a channel

      • Buffer occupancy similar for different levels of congestion

      • Unclear if these protocols work with aggregated traffic at all

  • For future protocol design, buffer requirements should be considered

Smaller buffers increase loss but this in itself is not a bad thing

Smaller buffers increase lossBut this in itself is not a bad thing

  • Routers drop packets to throttle senders

  • Loss rate is a function of number of flows and sum of TCP windows

  • Rule-of-thumb vs. minimal buffers changes number of outstanding packets by a factor of 2

    • Loss will quadruple with minimal buffers

  • This is just the way how TCP works

    • Quality of service better even with higher loss…

  • Login