Building Blocks for Engineering QoS Expectations over Best-Effort Networks

1. 5 I E Logical FIFO 3 5 B I E 2 1 2 3 1 2 E 2 I 1 B F C E A D Building Blocks for Engineering QoS Expectations over Best-Effort Networks David Harrison, Yong Xia, Omesh Tickoo, Hema T. Kaur, Shiv Kalyanaraman : “shiv rpi”

2. Context: Rest-in-Peace QoS ? (SIGCOMM RIPQoS 2003) QOS

3. Our Goals… • HISTORY: TCP offers reliability over unreliable networks • Our focus: • Building blocks for … • … performance expectations … • over (I.e. as an overlay) • …multi-domain inter-networks … • ...that do not guarantee performance… • (I.e. best-effort networks)

4. Expected QoS Using Closed-loop Techniques Part 1: I E 5 Logical FIFO 3 B 5 I E 2 1 2 3 BANANAS Multi-path & TE Framework: incrementally deployable Part 2: 1 E 2 2 I 1 A B E F C D Part 3: E2E Expected QoS Pipe Abstraction: H2H Multimedia Apps Talk Overview [Eg: MSN/Yahoo + Akamai]

5. Priority/WFQ FIFO B  B • Scheduler: differentiates service on a packet-by-packet basis • Loops: differentiate service on an RTT-by-RTT basis using edge-based policy configuration. • Differentiation/Isolation meaningful in steady state only… Part 1: Overlay Closed-loop QoS

6. Bottleneck queue  Edge system End system Architectural Advantages of Closed Loops • Traffic management consolidated at edges (placement of functions in line with E2E principle) • Architectural Potential: • Edge-based (distributed) QoS services, • Edge plays in application-level QoS

7. Expected Min Rate (EMR) Service: Sample Steady State Behavior Flow 1 with 4 Mbps assured + 3 Mbps best effort Flow 2 with 3 Mbps best effort Issues: Transients, steady state oscillations/tracking errors

8. QoS can be viewed as a congestion control problem: extension of the idea of fairness and therefore, QoS can be posed in Kelly’s optimization framework Challenges: 1. What about the non-concavity of QoS utility functions? 2. Can we avoid admission control? Graceful degradation instead. Ans: 1. Define & Solve an Auxiliary Optimization Problem 2. Alternative Convex Constraints in Lagrange Domain can avoid need for admission control, allowing graceful service degradation Overlay QoS w/ Closed Loops

9. weight User s’s send rate User s’s utility function ws can be interpreted as price per unit time or as congestion in the network. Kelly’s Optimization Formulation Network N optimizes Maximize subject to capacity constraints Each user s optimizes Maximize Resulting System optimizes Maximize subject to capacity constraints

10. Two User Utility Functions Maximize optimum

11. Issue: QoS => Non-concave User Utility Functions • A single user with a minimum rate QoS expectation (gracefully degrading into a weighted service) can be modeled with a non-concave utility function. • But this kind of U-function cannot be plugged directly into Kelly’s non-linear optimization formulation! log(xs) log(xs-0.4) 10log(xs) expected minimum =0.4

12. Desired Allocation (oversubscribed) not any of the 2 maxima! • Can use strictly concave functions and define multiple optimization problems for the same QoS problem & • Dynamically choose a different optimization problem when oversubscribed Luckily, the Sum of Non-Concave U-fns is not what we want to Optimize! • U(z) = log(z-0.6)if z > 0.6 (expected minimum rate) 10logz if z <= 0.6 (graceful degradation to weighted svc) Two maxima

13. No Over-Subscription Case: Auxiliary Problem • Let i.e. flow i: two virtual sub-flows on same path • Think of a modified network with modified link capacities Provide proportional fairness on residual network capacity xie Capacity, C1 Capacity,

14. Weighted fairness Effective when: ai = Ai • For aip, xip: (auxiliary problem) Exp. Min Rate: Effective when: ai < Ai If under-subscribed, solve the aux-problem; and the primary problem is automatically solved (note: aip = constant) Handling both under- and over-subscription… • For ai, xi: (primary problem)

15. 1 j j+1 J ingress egress dj fi μij Λi,j+1 μi Λi Accumulation: Per-flow Backlog in a Series of Routers • Assuming: • Define accumulation: • which is a time-shifted, distributed sum of buffered bits of flow i at all routers 1 through J • Note: accumulation is an abstract dynamical concept. Vegas and Monaco attempt to provide estimators for accumulation.

16. 1 j j+1 J dj fi μij Λi,j+1 μi Λi … … time 1 j j+1 J 16 Accumulation: Definition & Physical Meaning

17. 1 j j+1 J dj fi μij Λi,j+1 μi Λi Accumulation-based Control Policy • control objective : keep • if goal , no way to probe increase of available b/w; • control algorithm : • Example control algorithm :

18. 1 j j+1 J dj fi μij Λi,j+1 μi Λi out-of-band in-band ctrl pkt Monaco: Robust Accumulation Estimation … … time 1 j j+1 J

19. out-of-bd ctrl classifier fifo in-band ctrl, data pkt Monaco Accumulation Estimator 1 jf jf+1 Jf djf fi data μij Λi,j+1 μi ctrl Λi Jb jb+1 jb djb 1 ctrl Interior routers provide two priority fifo queues : 1) high priority queue for out-of-band control packet 2) low priority queue for in-band control packet and data packet Can be done w/ IP precedence on existing routers in Internet!!

20. ACC: Fairness & Linux Implementation Results

21. “Accumulation”: Steering Parameter • Accumulation-based congestion control (ACC) is a nonlinear optimization, where user i maximizes: Ui(xi) = ai lnxi • Accumulation (ai) is the weight (wi) of the weighted prop. fair allocation • Accumulation is hence a “steering” parameter: • Equilibrium accumulation allocation => Equilibrium rate allocation! • Dynamics of CC scheme decoupled from equilibrium spec (unlike AIMD) • Accumulation has a physical meaning: sum of buffered bits of the flow in the path • Accumulation is related to the lagrange multiplier, I.e., ai = pl • For two flows i,k sharing the same path, ai / ak = xi / xk • FIFO queues => arrival order decides departure order => buffer occupancy decides rate allocation q1 q2 x1 x2

22. Weighted fairness Effective when: ai = Ai • For aip, xip: (auxiliary problem) Exp. Min Rate: Effective when: ai < Ai If under-subscribed, solve the aux-problem; and the primary problem is automatically solved (note: aip = constant) Recall: Handling both under- and over-subscription… • For ai, xi: (primary problem)

23. Expected Min Rate (EMR) Service Building Block • Accumulation • Control Law di Accumulation limit Target Estimated accumulationin network

24. Range of Expected Minimum Rates

25. Mean Queue Length for EMR No AQM RIO A=30KB A=3000KB AVQ+VD

26. Service Multiplexing Topology 1,0 ~ 1,3 2,0 ~ 2,3 3,0 ~ 3,3 30M 60K … … … 0,0 0,0 web A web B … … 1ms 35M 75K 0,1 0,1 100M 100M 100M 34ms R0 R1 R2 R3 67ms 50M 60K 0,2 1-100ms 1-100ms 0,2 1-100ms … … 100ms 10M 15K 500 web A users 500 web B users 0,3 0,3 … … … 1,0 ~ 1,3 2,0 ~ 2,3 3,0 ~ 3,3 2,0 has weight 3 • Bandwidth for all unlabelled links are 1Gbps; Delay 1ms; • AQM+VD at router R1, no AQM at other routers

27. No oversubscription <m00=30,A00=60KB> <m03=10,A03=15KB> Loop 02,03stop sending Web Moderate oversubscription <m01=35,A01=75KB> Gross oversubscription <m02=50,A02=60KB> Service Multiplexing: Expected Min Rate

28. Queue Lengths Non-AQM: Q-length Q-length w/ virtual accumulation support More details: Y.Xia et al, “Accumulation-based Congestion Control,” to appear in IEEE/ACM Transactions of Networking, early 2005.

29. Expected QoS Using Closed-loop Techniques [Eg: MSN/Yahoo + Akamai] Part 1: I E 5 Logical FIFO 3 B 5 I E 2 1 2 3 BANANAS Multi-path & TE Framework: incrementally deployable Part 2: 1 E 2 2 I 1 A B E F C D Part 3: E2E Expected QoS Pipe Abstraction: H2H Multimedia Apps Talk Overview

30. ISP-1 Internet . . . AS1 ISP-n Part 2/3:Best-EffortPath Multiplicity Phone modem USB/802.11a/b 802.11a Firewire/802.11a/b WiFi (802.11b) Ethernet

31. Multiple E2E Flows Multi-homed interfaces & ISP/AS peers Multi-paths within a routing domain Multi-AS-paths across domains W/o specifying intra-domain paths Multiple exits from an AS w/o specifying intra-domain paths Part 2: Multiplicity: Flows, Paths, Exits/Interfaces BANANAS: A Single Abstract Framework To Exploit These Forms of Multiplicity In the Internet and Future Networks

32. Isn’t multi-path routing an old subject? • Lots of old work: • Multi-path algorithms/protocols [5, 6, 7, 8]*, • Internet signaling architectures [9, 10, 11, 12, 13]* • Novel overlay routing methods [14, 15]* • Transport-level approaches for multi-homed hosts [16, 17]* • But newer goals: • Traffic engineering (reliability, availability, survivability, re-route): • Separation of traffic trunking from route selection • Packet level or aggregate (micro-flow or macro-flow level) • Best-effort e2e service composition… Why hasn’t it happened after 2 decades of work? *[5-17] In SIGCOMM Future Directions of Network Architectures (FDNA) 2003 paper

33. Missing Architectural Concepts • An evolutionary partial deployment strategy • Allows partial deployment, incremental upgrades • Incentives: increasing value with increasing deployment • Fits the connectionless nature of dominant routing protocols, • Must not require signaling (unlike ATM, MPLS etc) • Abstract enough to be applicable to other scenarios: • Overlay networks, last-mile multi-hop (mesh or community) wireless networks, ad-hoc/sensor networks… • Flexible enough to have other realizations/semantics: • Different placement of functions (edge vs core) • Tunneling/Label Stacking • Geographic/Trajectory routing

34. Traffic Engineering (TE) Spectrum … Shortest Path MPLS Signaled TE BANANAS-TE Connectionless TE: Questions • Can we do multi-path & explicit routing ? • without signaling (I.e. in a connectionless context) • without variable (and large) per-packet overhead • being backward compatible with OSPF & BGP • allowing incremental network upgrades • Non-Goals: Monitoring, Traffic trunking/mapping

35. Seattle New York (Egress) 5 San Francisco (Ingress) 1321 120 Miami IP IP IP IP 0 Label 1321 120 What can we learn from ATM and MPLS ? • MPLS label = Path identifier at each hop • Labels is a local identifier… • Signaling maps global identifiers (addresses, path specifications) to local identifiers

36. Seattle 5 New York (Egress) 4 4 18 IP 27 3 10 San Francisco (Ingress) 1 9 Miami 5 IP IP IP IP 27 36 0 PathId Global Path Identifiers? • Instead of using local path identifiers (labels in MPLS), consider the use of “global” path identifiers • Constructed out of global variables a node already knows! • Eg: Link/Router IP addrs, Link weights, ASNs, Area Ids, GPS location • Avoid the need for signaling to establish a mapping!

37. j 2 wm w2 i w1 m-1 1 k Path suffix Global Path Identifier (continued) • Path = {i, w1, 1, w2, 2, …, wk, k, wk+1, … , wm, j} • Sequence of globally known node IDs & Link weights • Global PathID: hash of this sequence: computable w/o signaling! • Canonical method: MD5 hashing of the subsequence of nodeIDs followed by a CRC-32 to get a 32-bit hash value (MD5+CRC) • Low collision (i.e. non-uniqueness) probability • Note: Different PathID encodings have different architectural implications

38. PathID SuffixPathID H{k, k+1, … , m-1} H{k+1, … , m-1} j 2 wm w2 i w1 m-1 1 k Path suffix Abstract Forwarding Paradigm • Forwarding table (Eg; at Node k): • [Destination Prefix, ]  [Next-Hop, ] • [j, ]  [k+1, ] • Packet Header: [j, H{k, k+1, … , m-1} ]  [j, H{k+1, … , m-1} ]

39. IP IP IP IP 27 27 27 0 BANANAS TE: Partial Deployment • Only red nodes are upgraded • “Virtual hop” between upgraded nodes • Black nodes compute single-shortest-path Seattle 5 New York (Egress) 4 4 28 IP 27 30 10 San Francisco (Ingress) 1 9 1 X 2 Miami 3 1

40. Outline • Motivation: Best-effort path multiplicity • BANANAS: Data-plane • BANANAS: Control-plane • BANANAS: Mapping to BGP Not covered: details in SIGCOMM FDNA paper

41. Putting It Together: Integrated OSPF/BGP Simulation

42. Multiple E2E Flows Multi-homed interfaces & ISP/AS peers Multi-paths within a routing domain Multi-AS-paths across domains W/o specifying intra-domain paths Multiple exits from an AS w/o specifying intra-domain paths Multiplicity: Take-Home Message… BANANAS: A Single Abstract Framework To Exploit These Forms of Multiplicity In the Internet and Future Networks

43. Expected QoS Using Closed-loop Techniques [Eg: MSN/Yahoo + Akamai] Part 1: I E 5 Logical FIFO 3 B 5 I E 2 1 2 3 BANANAS Multi-path & TE Framework: incrementally deployable Part 2: 1 E 2 2 I 1 A B E F C D Part 3: E2E Expected QoS Pipe Abstraction: H2H Multimedia Apps Talk Overview

44. Part 3: Multimedia Apps: Leverage E2E Performance Diversity

45. Part 3: Introduction • Motivation: Video over best-effort networks (wired/wireless) • Broadband => more access bandwidth • End-to-end (E2E) => constraints due to path congestion • Virtual extension of broadband access pipe E2E using multi-paths => HOME-to-HOME (H2H) multimedia • Path Diversity: dimensions • Aggregate Capacity • Delay diversity • Loss diversity • Correlations in path performance characteristics • Key: Match inherent content diversity to path diversity

46. Performance Saturation (even w/ many flows/path) Motivation: Internet Path Congestion limits E2E bandwidth Internet Server Access Link Client Access Link Performance Access Link Speed

47. Multi-paths using Peers Overlays or peers can provide path diversity even if multi-paths not available natively Issue: diversity of performance (b/w, delay, loss), possible correlations…

48. Performance Scaling Performance Access Link Speed Smart Multi-path Capacity Aggregation (SMCA): Motivation Path (Flow) Aggregator/ Multiplexer Path (Flow) Aggregator/ De-multiplexer Internet Client Access Link Server Access Link E2E Broadband Virtual Pipe Abstraction!!

49. High Delay/Jitter Low Capacity Lossy Single path issues: capacity, delay, loss… Time • Network paths usually have: • low e2e capacity, • high latencies and • high/variable loss rates.

50. Low Perceived Loss High Perceived Capacity Low Perceived Delay/Jitter SMCA: Leverage Diversity!