15-744: Computer Networking

1. 15-744: Computer Networking L-25 Inter-Domain Traffic, IXPs, and Green Networking

2. Outline • Internet Inter-Domain Traffic (Lebovitz et al., 2010) • Anatomy of a Large European IXP (Ager et al, 2012) • It’s Not Easy Being Green (Gao et al, 2012)

3. Internet Inter-Domain Traffic Craig Labovitz, Scott Iekel-Johnson, Danny McPherson Arbor Networks Jon Oberheide, FarnamJahanian University of Michigan Ack: C. Labovitz

4. Motivation • Measuring the Internet is hard • Significant previous work on • Router and AS-level topologies • Individual link / ISP traffic studies • Synthetic traffic demands • But limited “ground-truth” on inter-domain traffic • Most commercial arrangements under NDA • Significant lack of uniform instrumentation • Goal: longitudinal observations of Internet traffic • Can we instrument representative distribution of ISPs? • Estimate of traffic volume / growth • Analysis of changes in Internet traffic demands

5. Conventional Wisdom • Internet is a global scale end-to-end network • Packets transit (mostly) unmolested • Value of network is global addressability/reachability (Metcalfe effect) • Broad distribution of traffic sources / sinks • An Internet “core” exists • Dominated by a dozen global transit providers • Interconnecting content, consumer and regional providers

6. Methodology • Focus on inter-domain traffic • i.e. distinguish from web hits, tweets, VPN, etc. • Leverage widely deployed commercial Internet monitoring infrastructure • Add export of coarse grain traffic statistics (ASNs, ASPaths, protocols, ports, etc.) • Via anonymous XML forwarded to central servers • Cajole carriers into participation • 110+ ISPs / content providers • Including 3,000 edge routers and 100,000 interfaces • And an estimated ~25% all inter-domain traffic • Wait two years...

7. Change in Carrier Traffic Demands • In 2007, top ten match “tier-1” ISPs • In 2009, global transit carry significant traffic volumes • But non-transit companies Google and Comcast join the list • And a significant percentage of ISP A traffic is Google transit

8. Consolidation of Content (Grouped Origin ASN) • In 2007, thousands of ASNs contributed 50% of content • In 2009, 150 ASNs contribute 50% of all Internet traffic

9. A Google Case Study • Over time Google absorbs YouTube traffic • As of July 2009, Google accounts for more than 5% of all Internet inter-domain traffic • Google the fastest growing ASN group

10. A Comcast Case Study • In 2007, Comcast has typical “eyeball” peering ratios • By 2009, Comcast resembles a transit / content provider • Wholesale transit, cell backhaul, video distribution, backbone, consolidation

11. Market forces intuition

12. Traditional Internet model

13. The New Internet model • Flatter and much more densely interconnected • Disintermediation between content and “eyeball” networks • New commercial models between content, consumer and transit

14. Market forces intuition • Commoditization of IP and Hosting / CDN • Drop of price of wholesale transit • Drop of price of video / CDN • Economics and scale drive enterprise to “cloud” • Consolidation • Bigger get bigger (economies of scale) • e.g., Google, Yahoo, MSFT acquisitions • Disintermediation • Direct interconnection of content and consumer • Driven by both cost and increasingly performance

15. Applications • Growing volume of Internet traffic uses port 80 / 443 • Includes significant video component and source of most growth • Unclassified includes P2P and video • P2P is fastest declining

16. Applications • Growing dominance of web as application front-end • Plus burden of ubiquitous network layer security policies • Results in growing concentration of application traffic over a decreasing number of TCP / UDP ports • Especially port 80 • Especially video

17. Migration of File Sharing to the Web • In 2006, P2P one of largest threats facing carriers • In 2010, P2P fastest declining application group • Significant corresponding growth in direct download and streaming video • Carpathia small hosting company by traffic volume in Fall 2008 • MegaUpload becomes Carpathia customer in November 2008 different continents

18. Discussion • Significant changes in inter-domain traffic patterns • Significant shift from connectivity to content • Aggregation of content / traffic sources • Shift from transit to direct interconnection • Most significant growth in ~150 large content ASN • And concurrent shift in applications to port 80 • i.e. the web may represent the new end-to-end • Implications on engineering and research • ACL/port based security model • Fault tolerance • Routing, traffic engineering, network design • Rapid growth of non-interactive traffic demands (i.e. DC)

19. Outline • Internet Inter-Domain Traffic (Lebovitz et al., 2010) • Anatomy of a Large European IXP (Ager et al, 2012) • It’s Not Easy Being Green (Gao et al, 2012)

20. Anatomy of a Large European IXP Anja Feldmann TU Berlin/T-Labs Steve Uhlig Queen Mary University of London Bernhard AgerETH Zürich Nikos ChatzisNadi Sarrar TU Berlin/T-Labs Walter Willinger AT&T Labs Research Ack: B. Ager and A. Feldman

21. IXPs – Reminder… Accepted industry definition of an IXP (according to Euro-IX): A physical network infrastructure operated by a single entity with the purpose to facilitate the exchange of Internet traffic between autonomous aystems. The number of Autonomous Systems connected should at least be three and there must be a clear and open policy for others to join. https://www.euro-ix.net/what-is-an-ixp

22. Infrastructure of an IXP (DE-CIX) Robust infrastructurewith redundancy http://www.de-cix.net/about/topology/

23. Internet eXchange Points (IXPs) Content Provider 1 Content Provider 2 AS2 AS1 AS3 Layer-2 switch IXPs Offer connectivity to ASes Enable peering AS5 AS4

24. IXPs – Peering • Peering – Why? E.g.: Giganews: “Establishing open peering arrangements at neutral Internet Exchange Points is a highly desirable practice because the Internet Exchange members are able to significantly improve latency, bandwidth, fault-tolerance, and the routing of traffic between themselves at no additional costs.” • IXPs – Four types of peering policies • Open Peering – Inclination to peer with anyone, anywhere • Most common! • Selective Peering – Inclination to peer, with some conditions • Restrictive Peering – Inclination not to peer with any more entities • No Peering – No, prefer selling transit http://drpeering.net/white-papers/Peering-Policies/Peering-Policy.html

25. IXPs – Publicly available information • Sources: euro-ix, PCH, PeeringDB, IXP’s sites • Generally known: # IXPs ~ 350 worldwide http://www.pch.net

26. IXPs – Publicly available information • Generally known: # IXPs ~ 350 worldwide • Somewhat known: # ASes per IXP up to 500 • Less known: # ASes ~ 11,000 worldwide • Even less known: IXPs =~ Tier-1 ISP traffic • Unknown: # of peerings at IXPs

27. Peering links – current estimates?

28. Peering links – current estimates?

29. IXPs • Introduction to IXPs • A large European IXP • IXP peering fabric • IXP member diversity • IXP traffic matrix • Discussion • Summary

30. Data – From collaboration with IXP • Major European IXP • 9 months of sFlow records collected in 2011 • Sampling 1 out of 16K packets • 128 bytes  IP/TCP/UDP headers • Consistency checks and filters • Checked for duplicates • Filtered out IXP management traffic, broadcast and multicast (except ARP) • Eliminated IPv6 (less than 1% of traffic)

31. Fact 1 – IXP members/participants • Traditional classification

32. Fact 3 – IXP traffic • Traffic Volume: Same as Tier-1 ISPs • IXP is interchange for Tier-2 ISPs

33. IXPs • Introduction to IXPs • A large European IXP • IXP peering fabric • IXP member diversity • IXP traffic matrix • Discussion • Summary

34. Fact 4 – IXP peerings • IXP peering link between pair of ASes if • IP traffic exchanged • BGP traffic only (e.g., in case of backup links) • IP otherwise • Potential links • Member ASes in Nov/Dec’11: 396 • 396x395 / 2 = 78,210 P-P links possible • Observed links • > 50,000 peering links • Peering rate > 60%!

35. Fact 4 – IXP peerings Internet-wide • Single IXP > 50,000 peering links • Derivation of new lower bound • 10 large IXPs in Europe: ~160,000 peering links • Remaining 340 or so IXPs: ~ 40,000 peering links • Completely ignoring all other peerings • (Conservative) lower bound on #of peering links • > 200,000 peering links in today’s Internet (as compared to currently assumed ~ 40,000 – 60,000) • Requires a revamping of the mental picture our community has about the AS-level Internet.

36. Fact 4 – IXP peerings Internet-wide

37. Public view of IXP peering links • Peering links at IXP: > 50 K • How come that we did not see them?

38. Visibility of IXP peerings • Even with all available datasets about 70% of IXP peering links remain invisible! • Even with all available datasets about 43 % of exchanged bytes remain invisible!

39. IXPs • Introduction to IXPs • A large European IXP • IXP peering fabric • IXP member diversity • IXP traffic matrix • Discussion • Summary

40. Member diversity – Business type • Classified ASes according to business model • For the remainder of this talk • Large ISPs (LISP) • Small ISPs (SISP) • Hosters and CDNs (HCDN) • Academic and enterprise networks (AEN) • All business models present • Recall: Most member ASes offer multiple types

41. Member diversity – # of peers • Most members have a large # of peers

42. IXP – Fraction of Web-traffic • Individual ASes differ significantly!

43. IXP – Geographic distance • Locality in using the IXP can be observed.

44. IXPs • Introduction to IXPs • A large European IXP • IXP peering fabric • IXP member diversity • IXP traffic matrix • Discussion • Summary

45. Daily pattern – Top-10 tier-2 members • Pronounced time of day effects • Top 10 tier-2 responsible for 33% of traffic • Some ASes fully utilize their capacity

46. IXPs • Introduction to IXPs • A large European IXP • IXP peering fabric • IXP member diversity • IXP traffic matrix • Discussion • Summary

47. Internet: Mental model (before 2010) http://conferences.sigcomm.org/sigcomm/2010/slides/S3Labovitz.pdf

48. Most recent mental model – a 2011 Google, Akamai, RapidShare, … http://conferences.sigcomm.org/sigcomm/2010/slides/S3Labovitz.pdf • Flattening of the AS topology

49. Question – What about IXPs Google, Akamai, RapidShare, … IXP • Flattening of the AS topology • What about IXPs impact

50. Network map 2012+ Global Transit/National Backbones „Hyper Giiants“ Large Content, Consumer, Hosting CDN Global Internet Core IXP IXP AS 1 AS 2 Regional / Tier2Providers IXP Leaf IPNetworks • IXPs central component • Lots of local peering – rich fabric • Even flatter AS topology than assumed