Can coarse circuit switching work & What to do when it doesn't?

1. Can coarse circuit switching work & What to do when it doesn't? Jerry Chou Advisor: Bill Lin University of California, San Diego CNS Review, Jan. 14, 2009

2. Outline • Motivation • Overview of new optical networking paradigm • How to provision optical circuits? • What to do when provision circuits not enough? • Conclusions

3. Internet Traffic Ever Increasing

4. Current Packet Routing Scenario • Packets electronically routed hop-by-hop • IP routers interconnected over switched optical backbone • OEO conversion and queuing delays at each hop OXC OXC OXC * OXC OXC

5. Optical Circuit Switching • If optical circuit switching would work, then no intermediate per-hop queuing delays and OEO conversions = much faster OXC OXC OXC * OXC OXC

6. Packet Switching 10 ns Optical Switching Options • Extremely difficult to implement packet buffers and logic in optics • No viable dynamically reconfigurable active optical switches at this time scale

7. Packet Switching Optical Burst Switching 10 ns 1 ms Optical Switching Options • New signaling protocol and electronic control plane required to implement dynamic reservations • Although active optical switches available at this time scale, coordination of such frequent network-wide reconfigurations not easy

8. Packet Switching Optical Burst Switching Quasi-Static Optical Circuits 10 ns 1 ms 1 hr Optical Switching Options • Can we reasonably predict the traffic so that we can provision optical circuits to carry them? • Can we provide a “fall-back” mechanism when circuit capacity is enough? Over 3 Million X

9. Outline • Motivation • Overview of new optical networking paradigm • How to provision optical circuits? • What to do when provision circuits not enough? • Conclusions

10. Observation • Aggregate traffic at the core is relatively smooth and variations are predictable Source: Roughan’03 on a Tier-1 US Backbone

11. Case Study • On high-performance public backbone networks • Abilene (US):11 nodes, 23 links • GEANT (Europe): 23 nodes, 74 links • Public traffic matrices are available • Optical circuits only change on hourly basis • Use historical traffic to “predict” how much traffic will occur in the future • Abilene: 03/01/04-04/21/04, GEANT: 01/01/05–04/10/05 • Provision circuits to maximize likelihood that circuits have enough capacity • Simulated actual traffic (over a week) • Abilene: 04/22/04-04/26/04, GEANT: 04/11/05–04/15/05

12. Circuits • Setup circuits possibly across multiple paths in physical layer Seattle New York Chicago Sunnyvale Denver Indianapolis Los Angeles Washington Kansas City Atlanta Houston

13. Circuits • Logically one (optical) circuit for each OD-pair (origin-destination pair) Seattle New York

14. Abilene Network • Drop rates is the percentage of offering traffic exceeding its circuit capacity • To consider a highly utilized network, traffic is scaled, such that at least one link is saturated under OSPF • Worst-case 6.41%, 0.33% on average, mostly at or near 0% Circuit switching works “most of the time” if carefully provisioned

15. Smaller (simpler) routers Traffic arriving to intermediate node OXC Optical transit traffic New Paradigm • Provision optical circuits that maximize the probability of sufficient capacity to carry traffic • Use optical circuit switching by default • When actual traffic exceeds circuit capacities, route (electronically) over other “pre-configured circuits” with spare capacity

16. To:NY To:HS To:NY Analogy • Direct “non-stop” flights (optical circuits) by default • If overbooked, re-route (electronically) excess demand through alternative multi-hop flights Seattle NY Houston

17. Abilene Network • No packet drops with re-routing (adaptive load-balancing method to be discussed)

18. Advantages of New Paradigm • Minimize queuing delay and latency for packets • Reduce workload on electronic routers • Optical circuits change infrequently, and mechanisms exist to provision circuits • Key idea is to re-route electronically excess traffic rather than “on-the-fly” dynamic optical circuit reconfigurations • Avoid new signaling protocol and frequent coordination of network-wide reconfigurations

19. Outline • Motivation • Overview of new optical networking paradigm • How to provision optical circuits? • What to do when provision circuits not enough? • Conclusions

20. Basic Idea • Use historical traffic data sets to decide on bandwidth allocation • Major ISPs have data collection infrastructure already

21. Seattle New York Chicago Sunnyvale Denver Indianapolis Los Angeles Washington Kansas City Atlanta Houston Ideally, Traffic is Stable • Abilene • 11 nodes connected by 10Gb/s links Seattle/NY: Always 5Gb/s Allocate: 5Gb/s Sunnyvale/Houston: Always 5Gb/s Allocate: 5Gb/s Both flows can be carried by provisioned circuits

22. Seattle New York Chicago Sunnyvale Denver Indianapolis Los Angeles Washington Kansas City Atlanta Houston But, Flows Fluctuate Differently • Abilene • 11 nodes connected by 10Gb/s links Seattle/NY: High traffic mean Low traffic variance Sunnyvale/Houston: Low traffic mean High traffic variance Give more bandwidth to flows with “high mean” or “high variance”?

23. Circuit Provisioning Approach • Use Cumulative Distribution Function (CDF) as “utility function” (predictor of “acceptance probability”) • Acceptance probability • The probability of a provisioned circuit with enough capacity to carry its offering traffic

24. Seattle New York Chicago Sunnyvale Denver Indianapolis Los Angeles Washington Kansas City Atlanta Houston Example • Abilene • 11 nodes connected by 10Gb/s links Seattle/NY: 90% time ≤ 6Gb/s 50% time ≤ 4Gb/s Allocate: 6Gb/s Sunnyvale/Houston: 90% time ≤ 6Gb/s 80% time ≤ 4Gb/s Allocate: 4Gb/s Seattle/NY has 90% acceptance probability Sunnyvale/Houston has 80% acceptance probability

25. Circuit Provisioning Approach • Formulate bandwidth allocation (circuit provisioning) as multi-path utility max-min fair allocation problem • Utility functions represent traffic statistics (generally utility functions can be non-linear) • Max-min fairness reach balance between throughput and fairness • Multi-path circuits provide more freedom and better performance We provide the first solution to the multi-path utility max-min fair allocation

26. Saturated flow Fill-up by with a routing Max utility Multi-path Utility Max-min Algorithm • Allocation based on “water-filling algorithm” and maximum concurrent flow • Steps: • Identify maximum common utility increment • Solve maximum concurrent flow problem to find multi-path routing • Identify saturated flow

27. Multi-Path vs. Single-Path • Significantly lower drop probability • Mean drop rate: 3.56% vs. 20.34% • Max drop rate: 18.25 vs. 34.72%

28. Outline • Motivation • Overview of new optical networking paradigm • How to provision optical circuits? • What to do when provision circuits not enough? • Conclusions

29. Adaptive Load-Balanced Routing • Localized approach: • load-balance on outbound circuits, weighted by spare capacity Optical Circuit 1. r(B) < B[A, B] ? r(B) = 30 B YES 35 r(C) = 20 C Problem1: greedy solution based only one-hop info. Problem2: oscillation of weight changes can happen 35 NO r(D) = 25 D A 2.k = random (wk) 35

30. Adaptive Load-balance Re-routing • Distributed approach: Step1: Compute path cost by Distance-Vector-like protocol Step2: Update weights to reach Wardrop Equilibrium state • Every interval only shift weight by a small fraction δ • Achieve fast converge and prevent oscillation • Based on selfish routing no coordination among nodes path1 cost(C1): (1+4)=5 path2 cost(C2): (1+8)=9 2 1 1 1 t s 1 2 3 1 Current weights: w1, w2 δ = f(C1, C1, w1, w2) w1 = w1 + δ, w2 = w2 - δ 5 4

31. Abilene Network • 90 percentile drop rate comparison • OSPF has 0% drop at scale factor of 1

32. Abilene Network • 90 percentile drop rate comparison • Cisco’s “ecmp” load-balances across equal cost shortest paths and achieve lower drop rate

33. Abilene Network • 90 percentile drop rate comparison • Without rerouting, we suffer small drop rates even at the scale factor of 1 • But show lower drop rates at larger scale factors b.c of greater path diversity and better load-balance

34. Abilene Network • 90 percentile drop rate comparison • Achieve lowest drop rates among all • With rerouting, we don’t have drop until at a factor of 1.75.

35. Abilene Network • Circuit provisioning achieve lower drop rates under high traffic load b.c of load-balanced routing path • Rerouting effectively reduce drop rates under low traffic load by utilizing residual network capacity

36. Outline • Motivation • Overview of new optical networking paradigm • How to provision optical circuits? • What to do when provision circuits not enough? • Conclusions

37. Conclusion • A new paradigm of optical circuit switching by default, packet routing when necessary • Formulate circuit provisioning as an utility max-min fair allocation problem and provide the first solution under multiple paths scenario • Apply a adaptive load-balance protocol on re-routing • Conduct empirical study on two backbone networks, Abilene and GEANT • Show more than 95% of traffic can be carried by the network with carefully static circuit provisioning & all traffic can be routed after re-routing

38. Publication • Jerry Chou, Bill Lin, "Coarse Optical Circuit Switching by Default, Rerouting over Circuits for Adaptation,“ Journal of Optical Networking, vol. 8, no. 1, pp. 33-50 (2009).

39. Thank You

40. Backup Slides

41. Work-In-Progress • Capacity planning • Fault-tolerance • Better adaptive routing algorithms • Joint circuit-provisioning and routability optimization

42. Motivation • Traffic growing nearly twice rate of Moore’s Law • Difficult for electronic packet routers to keep up • On the other hand, optical switching provides abundance of transmission capacity (e.g. WDM) • Rate of increase in optical transport capacity keeping pace with traffic growth (with 100 Gbps per wavelength in next generation), well above Moore’s Law • Rate of decrease in cost per unit of optical transport capacity well below Moore’s Law

43. Networks • Traffic used for prediction (over months) • Abilene: 03/01/04 - 04/21/04, GEANT: 01/01/05 – 04/10/05 • Optical circuits only change on hourly basis (method to be discussed) • Simulated actual traffic (over a week) • Abilene: 04/22/04 - 04/26/04, GEANT: 04/11/05 – 04/15/05 • To consider a highly utilized network, we scaled traffic by a factor, such that at least one link is saturated under OSPF. • Abilene: 4, GEANT: 2

44. Questions • How to decide on circuit provisioning to maximize probability that the circuits provide sufficient capacity to carry traffic? • Formulated as a multi-path utility max-min fair bandwidth allocation problem • What to do when circuit capacity is not enough? • Adaptive load-balancing over circuits that have spare capacity

45. Saturated flow Fill-up by with a routing Max utility Multi-Path Utility Max-Min Algorithm • Based on water-filling algorithm and maximum concurrent flow (MCF) solver • Determine bandwidth allocation that achieves the maximum common utility for all flows • Determine path distribution by MCF routing • Identify saturated flows and fix their utility

46. Binary Search • Find maximum utility by binary search over [0, 1] • Determine flow traffic by utility functions • Find feasible route by querying a MCF solver • If l<1, decrease utility, otherwise increase utility 100 100 100 100 Utility(%) Utility(%) Utility(%) Utility(%) 80 80 80 80 60 60 60 60 40 40 40 40 20 20 20 20 10 20 30 40 50 10 20 30 40 50 10 20 30 40 50 10 20 30 40 50 BW BW BW BW C = 100

47. 20 100 U[1] - U[0] Seg IV 80 Utility(%) Seg III 10 60 40 Seg II 0 20 10 20 30 40 Seg I BW[1]-BW[0] 10 20 30 40 50 BW Piece-Wise Linear Search • Approximate utility functions as piecewise linear functions • Replace binary search by searching through each piecewise linear segment • Query MCF by the inverse of slope as traffic • l is proportional to maximum utility

48. B D A F C E Identifying Saturated Flows • By residual capacity is not enough • Miss-identified saturated flow in earlier iteration would produce smaller bandwidth allocation Let link capacity = 10 Bandwidth requirement: AE = 5, AF = 5 If select path ACDF, AE is saturated If select path ABDF, AE is not saturated

49. Identifying Saturated Flows • A flow is saturated if its utility cannot be increased by any feasible routing • To guarantee optimality, flows have to be re-routed

50. Multi-Path vs. Single-Path • Significantly higher utility • Minimum utility 92.90% vs. 74.74%