Sizing router buffers
This presentation is the property of its rightful owner.
Sponsored Links
1 / 53

Sizing Router Buffers PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Sizing Router Buffers. Guido Appenzeller Thesis Defense May 24 th , 2004. Routers need Packet Buffers. It’s well known that routers need packet buffers It’s less clear why and how much Goal of this work is to answer the question: How much buffering do routers need?.

Download Presentation

Sizing Router Buffers

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Sizing Router Buffers

Guido Appenzeller

Thesis DefenseMay 24th, 2004

Routers need Packet Buffers

  • It’s well known that routers need packet buffers

    • It’s less clear why and how much

  • Goal of this work is to answer the question:

    How much buffering do routers need?

How much Buffer does a Router need?





  • Universally applied rule-of-thumb:

    • A router needs a buffer size:

      • 2T is the two-way propagation delay (or just 250ms)

      • C is capacity of bottleneck link

  • Context

    • Mandated in backbone and edge routers.

    • Appears in RFPs and IETF architectural guidelines..

    • Usually referenced to Villamizar and Song: “High Performance TCP in ANSNET”, CCR, 1994.

    • Already known by inventors of TCP [Van Jacobson, 1988]

    • Has major consequences for router design



  • 10Gb/s linecard

    • Requires 300Mbytes of buffering.

    • Read and write 40 byte packet every 32ns.

  • Memory technologies

    • DRAM: require 4 devices, but too slow.

    • SRAM: require 80 devices, 1kW, $2000.

  • Problem gets harder at 40Gb/s

    • Hence RLDRAM, FCRAM, etc.

Outline of this Work

  • Main Results

    • The rule of thumb is wrong for a core routers today

    • Required buffer is instead of

  • Outline of this talk

    • Where the rule of thumb comes from

    • Why it is incorrect for a core router in the internet today

    • Correct buffer requirements for a congested router

    • Buffer requirements for short flows (slow-start)

    • Experimental Verification

    • Conclusion


  • The Rule of Thumb

    • Where does the rule of thumb comes from? (Answer: TCP)

    • Interaction of TCP flows and a router buffers

  • The buffer requirements for a congested router

  • Buffer requirements for short flows (slow-start)

  • Experimental Verification

  • Conclusion



Only W=2packets may be outstanding



C’ > C


  • TCP Congestion Window controls the sending rate

    • Sender sends packets, receiver sends ACKs

    • Sending rate is controlled by Window W,

    • At any time, only W unacknowledged packets may be outstanding

    • The sending rate of TCP is

Link not fully utilized

Single TCP FlowRouter without buffers

Only Wpackets may be outstanding




C’ > C




  • Rule for adjusting W

    • If an ACK is received: W ← W+1/W

    • If a packet is lost:W ← W/2





For every W ACKs received,

send W+1 packets


Window size


Single TCP FlowRouter with large enough buffers for full link utilization




C’ > C


Required buffer is height of sawtooth




Buffer = rule of thumb

Microscopic TCP BehaviorWhen sender pauses, buffer drains

one RTT


Over-buffered Link

Under-buffered Link

Origin of rule-of-thumb

  • Before and after reducing window size, the sending rate of theTCP sender is the same

  • Inserting the rate equation we get

  • The RTT is part transmission delay T and part queuing delay B/C . We know that after reducing the window, the queuing delay is zero.


  • Rule-of-thumb makes sense for one flow

  • Typical backbone link has > 20,000 flows

  • Does the rule-of-thumb still hold?

  • Answer:

    • If flows are perfectly synchronized, then Yes.

    • If flows are desynchronized then No.


  • The Rule of Thumb

  • The buffer requirements for a congested router

    • Synchronized flows

    • Desynchronized flows

    • The 2T×C/sqrt(n) rule

  • Buffer requirements for short flows (slow-start)

  • Experimental Verification

  • Conclusion

If flows are synchronized

  • Aggregate window has same dynamics

  • Therefore buffer occupancy has same dynamics

  • Rule-of-thumb still holds.


When are Flows Synchronized?

  • Small numbers of flows tend to synchronize

    • In ns2 simulation they are synchronized

    • In at least some cases holds for real networks as well

  • Large aggregates of flows are not synchronized

    • For > 500 flows, synchronization disappears in ns2

    • On a Cisco GSR, 100 flows were not synchronized

    • Measurements in the core give no indication of synchronization

      • C. Fraleigh, “Provisioning Internet Backbone Networks to support Latency Sensisitve Applications”, Ph.D. Thesis, Stanford

      • Hohn, Veitch, Papagiannaki and Diot – “Bridging Router Performance and Queuing Theory”



Buffer Size

If flows are not synchronized



Quantitative Model

model as


  • For many de-synchronized flows

    • We assume congestion windows are independent

    • All congestion windows have the same probability distribution

  • Model congestion window of a flow as random variable

  • Now central limit theorem gives us the distribution of the sum of the window sizes

Buffer vs. Number of Flowsfor a given Bandwidth

  • For a given C, the window W scales with 1/n and thus

  • Standard deviation of sum of windows decreases with n

  • If for a single flow we have

  • Thus as n increases, buffer size should decrease

Required buffer size



  • Flows in the core are desynchronized

    • Substantial experimental evidence

    • Supported by ns2 simulations

  • For desynchronized flows, routers need only buffers of


  • The Rule of Thumb

  • The buffer requirements for a congested router

  • Buffer requirements for short flows (slow-start)

    • M/G/1 Model

  • Experimental Verification

  • Conclusion

Short Flows

  • So far we were assuming a congested router with long flows in congestion avoidance mode.

    • What about flows in slow start?

    • Do buffer requirements differ?

  • Answer: Yes, however:

    • Required buffer in such cases is independent of line speed and RTT (same for 1Mbit/s or 40 Gbit/s)

    • In mixes of flows, long flows drive buffer requirements

      • Short flow result relevant for uncongested routers

A single, short-lived TCP flowFlow length 62 packets, RTT ~140 ms


Flow Completion Time (FCT)




fin ackreceived




Modelling Short Flows

Idea: Find buffer size by modelling queue behaviour

  • Problem: Arrival process is hard to model

    • Simplify by modelling bursts as independent

    • Buffer empties several times during one RTT

Poisson arrivals offlows

Service time is Lflow, the flow length in packets

Poisson arrivals of bursts

Service time is the lengthof the burst: 2,4,8,16…

M/G/1 Model for short flows

  • TCP flows generate independent bursts

    • Service times is burst length :

    • Poisson arrivals of rate

  • To verify if this approach works, let’s compare the average queue length in the model and in simulation

Average Queue length

Buffer B


Packet Loss

P(Q = x)

Buffer Requirements for Short Flows

  • Buffers absorb fluctuations in queue, reduce packet loss

    • Reduce retransmits, Timeouts and thereby flow completion time

    • Utilization not a good measure of QoS as load << 1

  • We can find a good upper bound for loss, if we have the queue length distribution for an infinite buffer

    • If a packet arrives and queue length is shorter than buffer, packet will not be dropped

  • Problem: For M/G/1 there is no closed form expression for the queue distribution

Queue Distribution

We derived closed-form estimates of the queue

distribution using Effective Bandwidth

  • Gives very good closed form approximation

Short Flow Summary

  • Buffer requirements for short flows

    • Can be modeled by M/G/1 model

    • Only depends on load and burst size distribution

    • Example - for bursts of up to size 16 at load 0.8

      • For 1% loss probability B = 115 Packets

      • For 0.01% loss probability B = 230 packets etc.

      • Bursts of size 12 is maximum for Windows XP

    • Independent of line speed and RTT

  • In mixes of flows, long flows dominate buffer requirements


  • The Rule of Thumb

  • The buffer requirements for a congested router

  • Buffer requirements for short flows (slow-start)

  • Experimental Verification

  • Conclusion

Experimental Evaluation Overview

  • Simulation with ns2

    • Over 10,000 simulations that cover range of settings

      • Simulation time 30s to 5 minutes

      • Bandwidth 10 Mb/s - 1 Gb/s

      • Latency 20ms -250 ms,

  • Physical router

    • Cisco GSR with OC3 line card

    • In collaboration with University of Wisconsin

  • Experimental results presented here

    • Long Flows - Utilization

    • Mixes of flows - Flow Completion Time (FCT)

    • Mixes of flows - Heavy Tailed Flow Distribution

    • Short Flows – Queue Distribution

Long Flows - Utilization (I)Small Buffers are sufficient - OC3 Line, ~100ms RTT




Long Flows – Utilization (II) Model vs. ns2 vs. Physical RouterGSR 12000, OC3 Line Card

Mixes of FlowsFlow Completion TimeFCT of 14 packet flows that share a link with long-lived flows.

Heavy-tailed flow length distribution

  • Experiment

    • Flow arrivals are a Poisson process

    • Flow lengths are Pareto distributed

  • Results

    • Buffers in the order of are still sufficient

    • Number of “long-lived” flows n is now defined as number of flows in congestion avoidance mode

Pareto Flow DistributionFinding the number of flows

No. of flows in CA mode

For buffer sizing, pick n = 100

time [seconds]

Pareto Flow Distribution

Flow arrivalson link 1





Short Flows – Queue DistributionM/G/1 Model vs. GSR 12000, OC3 Line Card


  • The Rule of Thumb

  • The buffer requirements for a congested router

  • Buffer requirements for short flows (slow-start)

  • Experimental Verification

  • Conclusion

Related Work

  • Related Publications

    • Buffer sizing

      • “High Speed TCP in ANS Net” - Villamizar and Song, ACCR 1994

      • “TCP behaviour with many flows” – R. Morris, IEEE ICNP 1997

      • “Scalable TCP congestion control” – R. Morris, INFOCOM 2000

    • Queue Modelling

      • “Modelling, Simulation and Measurement of Queuing Delay” – Garetto and Towsley, SIGMETRICS 2003

Original Contributions

  • Main original contributions of this work

    • Routers only require buffers of instead of

    • Models on TCP buffer interaction for

      • Congestion avoidance mode

      • Slow start

    • Experimental Verification

  • Publication

    • “Sizing Router Buffers – Guido Appenzeller, Isaac Keslassy and Nick McKeown, to appear at SIGCOMM 2004

The Commercial Internet Today

  • Today’s internet differs from our assumptions

    • Core is overprovisioned, almost never congested

    • Access links are usually the bottleneck (DSL, Modem)

    • Flows are usually limited by Maximum Window size

      • Maximum Window is from 6-12 (Windows) to 42 (Unix)

    • Are the results still relevant?

  • Answer: Yes, we were intentionally pessimistic

    • Routers still needs to work in case of congestion

      • Even if this “worst case” scenario is rare

    • Slow access links and small TCP windows reduce buffer requirements further

      • Bursts are smoothed out

      • Converges towards constant rate source, poisson packet arrivals

      • We verified reduced buffer requirements experimentally

How much buffer does a router need?

The old “Rule-of-Thumb”

Our Contribution

Impact on Router Design

  • 10Gb/s linecard with 200,000 x 56kb/s flows

    • Rule-of-thumb: Buffer = 2.5Gbits

      • Requires external, slow DRAM

    • Becomes: Buffer = 6Mbits

      • Can use on-chip, fast SRAM

      • Completion time halved for short-flows

  • 40Gb/s linecard with 40,000 x 1Mb/s flows

    • Rule-of-thumb: Buffer = 10Gbits

    • Becomes: Buffer = 50Mbits



Two TCP FlowsTwo flows sharing a bottleneck synchronize

Impact on Protocol Design

TCP Reno: requires

General: requires

  • The ‘rule-of-thumb’ is due to the nature of TCP

    • We can easily build a TCP that requires less buffers

    • Latency based TCP (Vegas, FAST) suffer from smaller buffers

      • Queuing delay too small to be used as a channel

      • Buffer occupancy similar for different levels of congestion

      • Unclear if these protocols work with aggregated traffic at all

  • For future protocol design, buffer requirements should be considered

Smaller buffers increase lossBut this in itself is not a bad thing

  • Routers drop packets to throttle senders

  • Loss rate is a function of number of flows and sum of TCP windows

  • Rule-of-thumb vs. minimal buffers changes number of outstanding packets by a factor of 2

    • Loss will quadruple with minimal buffers

  • This is just the way how TCP works

    • Quality of service better even with higher loss…

  • Login