Outline

1. ECODE FP7 Project Training Seminar: Session 2aInternet architecture (incl. Topology structure, and models)Dimitri Papadimitriou and Olivier BonaventureAlcatel-Lucent BELL - Universite catholique de Louvain (UCL)September 1, 2008Alcatel-Lucent BELLAntwerpen, Belgium

2. Outline 1.Organization of the Internet • Topology • Types of domains • Transit domain • Stub domain • Example of domains • Internet Routing 2. Evolution of the Internet • Number of Hosts • IP address allocation • IPv4 address allocation • IPv6 address allocation • Number of AS • Routing tables • Size of the IPv4 BGP routing tables • Size of the IPv6 BGP routing tables • IP traffic flows • Bandwidth

3. Outline 3. Internet Topology modelling • Network properties • Random Graphs models and generators • Structural models and generators • Topology measurements • Power Law relationships • Degree-based models and generators • Internet topology metrics

4. Outline 1.Organization of the Internet • Topology • Types of domains • Transit domain • Stub domain • Example of domains • Internet Routing 2. Evolution of the Internet 3. Internet Topology modeling

5. Organization of the Internet • Internet: infrastructure composed by an interconnected set of (heterogeneous) networks architected around a distributed routing system that is partitioned into independently administrated domains (autonomous systems) • A domain is a set of routers, links, hosts and local area networks under the same administrative control • A domain can be very large... • AS568: SUMNET-AS DISO-UNRRA contains 73154560 IP addresses • A domain can be very small... • AS2111: IST-ATRIUM TE Experiment a single PC running Linux... • Internet is composed of ~ 30.000 autonomous systems (AS)

6. Organization of the Internet • Domains are interconnected in various ways • The interconnection of all domains should in theory allow packets to be sent anywhere • Usually IP datagram will need to cross a few ASes (3 to 4, average 3.4) to reach its destination

7. Evolution of the Internet Topology (1) • 1986: NSF builds NSFNet as backbone, links 6 supercomputer centers, 56 kbps; huge increase of connections, especially from universities • 1987: 10,000 hosts - 1989: 100,000 hosts - 1992: 1 million hosts • 1988: NSFNet backbone upgrades to 1.5Mbps • 1991: NSF lifts restrictions on the commercial use of the Net; • 1994: NSF reverts back to research network (vBNS); the backbone of the Internet consists of multiple private backbones • Before ‘95: Strict hierarchical network with single central backbone NSFNet Backbone Regional Regional Regional Campus Campus Campus Campus

8. Evolution of the Internet Topology (2) • Between 1995-1999: increased meshedness between ISP backbones and customers • Decentralization: from a single backbone network to a conglomeration of 100s of backbone and 1000s ISP • Loss of hierarchy and abstraction: from hierarchical network to increasingly meshed interconnection • Significant bandwidth increase: from T3 (45MB) and T1 (1MB) to OC48 (2.5GB) and OC12 (622MB) link capacity AS1 AS2 R2 R1 AS4 AS3 R3 R4

9. Can be viewed as structured into tiers Tier-1 ISPs a.k.a backbone providers Dozen (12 to 20 AS) of large international or large national ISPs interconnected by multiple private peering points (shared cost) Provide transit service (no “upstream” provider) Examples: AT&T, Verizon, Sprint, Level 3, etc. Tier-2 ISPs Regional or National ISPs (order 1k AS) Customer of T1 ISP(s) - at least 1 and often 2 - and Provider of T3 ISP(s) Shared-cost with other T2 ISPs Examples: France Telecom, BT, Belgacom Tier-3 ISPs a.k.a stub AS Smaller ISPs, Corporate Networks, Content providers (order 10k AS) Customers of T2 or T1 ISPs (no transit service to other ISPs) Shared-cost with other T3 ISPs Interconnections An ISP runs (private) Points of Presence (PoP) where its customers and other ISPs connect to it ISPs also connect at (public) Network Access Point (NAP) called public peering Evolution of the Internet Topology (3) 9

10. Tier-1 ISP Tier 1 ISP NAP Tier 1 ISP Tier 1 ISP “Tier-1” ISPs(a.k.a. backbone providers e.g., AT&T, Verizon, Sprint, Level 3, Qwest): national/ international coverage treating each other as equals (peers) Tier-1 providers also interconnect at public network access points (NAPs) = public peering Tier-1 providers interconnect privately = multiple private peering

11. Tier-2 ISP Tier 1 ISP NAP Tier 1 ISP Tier 1 ISP “Tier-2” ISPs (often regional-national): ISPs that connect to one or more Tier-1 ISPs, possibly other Tier-2 ISPs Tier-2 ISPs also peer privately with each other, and publicly interconnect at NAP Tier-2 ISP Tier-2 ISP Tier-2 ISP pays Tier-1 ISP for connectivity to rest of Internet Tier-2 ISP is customer of Tier-1 ISP Tier-2 ISP Tier-2 ISP Tier-2 ISP PoP

12. Tier-3 ISP Tier 1 ISP NAP Tier 1 ISP Tier 1 ISP “Tier-3” ISPs: last hop (“access”) network (closest to end systems) Tier-3 ISP Tier-3 ISP Tier-3 ISP Tier-3 ISP Tier-3 ISP Tier-2 ISP Tier-2 ISP Tier- 3 ISPs are customers of higher tier ISPs connecting them to rest of Internet Tier-2 ISP Tier-2 ISP Tier-2 ISP PoP Tier-3 ISP Tier-3 ISP Tier-3 ISP Tier-3 ISP

13. Organization of the Internet Tier-1 ISPs • Dozen of large ISPs interconnected by shared-cost • Provide transit service • Uunet, Level3, Sprint, ... Tier-2 ISPs • Regional or National ISPs • Customer of T1 ISP(s) • Provider of T2 ISP(s) • Shared-cost with other T2 ISPs • France Telecom, BT, Belgacom Tier-3 ISPs • Smaller ISPs, Corporate Networks, Content providers • Customers of T2 or T1 ISPs • Shared-cost with other T3 ISPs

14. AS Ranking • Proposing two ranking methods: • Degree-based: ASes are ranked by their degrees in the AS topology graph: http://as-rank.caida.org/ • AS-relationship-based: ASes are ranked by their customer cone sizes See http://as-rank.caida.org/data RouteViews BGP AS links annotated with inferred relationships Dataset date: 20080818 Alpha parameter of inference algorithm: 0.01000 Format: <AS1> <AS2> <relationship> where <AS1> and <AS2> are AS numbers, and <relationship> is -1 if AS1 is a customer of AS2, 0 if AS1 and AS2 are peers, 1 if AS1 is a provider of AS2, and 2 if AS1 and AS2 are siblings (the same organization)

15. Summary • Based on AS connectivity and relationships, the Internet routing infrastructure can be viewed as a three tier hierarchy • Core: consisting of a dozen or so Tier-1 providers forming the top level of the hierarchy • Middle: consisting of few thousands of ASes (Tier-2 providers) that provide transit service but are not part of the core • Edge: 10 thousands of stub ASes that do not provide transit service. Usually, local ISP, ASP and CSP

16. Outline 1.Organization of the Internet • Topology • Types of domains • Transit domain • Stub domain • Example of domains • Internet Routing 2. Evolution of the Internet 3. Internet Topology modeling

17. Types of domains • The Internet consists of routing domains: Autonomous Systems (AS) interconnected with each other: • Transit domain: provider, hooking many AS together • Stub domain: smaller corporation/domain: • At least one and usually two connections to other domain • No transit service to other domains • Two-level routing: • Intra-domain: administrator responsible for choice of routing protocol within network (usually link-state routing protocol) • Inter-domain: standard for interdomain routing: BGP

18. Types of domains (1) S1 S2 S3 S4 T1 T2 T3 • Transit domain • A transit domain allows external domains to use its own infrastructure to send packets to other domains • Examples • UUNet, OpenTransit, GEANT, Internet2, RENATER, EQUANT, BT, Telia, Level3,...

19. Types of domains (2) S1 S2 S3 S4 T1 T2 T3 Stub domains • A stub domain does not allow external domains to use its infrastructure to send packets to other domains • A stub is connected to at least one transit domain • Single-homed stub : connected to one transit domain • Dual-homed stub : connected to two transit domains • Content stub domain (Content Service Provider) • Large web servers : Yahoo, Google, MSN, TF1, BBC,... • Access-rich stub domain (Access Service Provider) • ISPs providing Internet access via CATV, ADSL, ...

20. Multihomed domains S3 T2 T3 T1 • Definition: use of redundant network links/connections to the same or different domain for the purposes of external connectivity • Objective: • Robustness in case of failure (link, upstream domain) • Performance (load balancing) • Cost • Multi-homed stub AS: connectivity to multiple immediate upstream transit domains • Multi-homed transit AS

21. A transit domain : Easynet

22. A transit domain : GEANT

23. A transit domain : BT/IGnite

24. A large transit domain : UUNet

25. $ $ Customer-provider $ $ $ $ Shared-cost (peering) $ $ Composition of Internet paths AS7 AS8 AS9 AS4 AS3 AS1 AS2 • Most Internet paths contain a sequence of • 0 or more Customer->Provider relationships • 0 or 1 Peer-to-Peer relationships • 0 or more Provider->Customer relationships

26. Outline 1.Organization of the Internet • Topology • Types of domains • Transit domain • Stub domain • Example of domains • Internet Routing 2. Evolution of the Internet 3. Internet Topology modeling

27. Internet Routing Internet domains comprises devices called routers comprising a routing and a forwarding engine (and a management agent) Routing engine: • Process routing information (exchanged between routers using a routing protocols such as BGP) so as to compute routes (using a shortest path algorithms) • Routes entries (composed by a destination, a next-hop interface, and a metric) are stored in routing information bases (RIB) • Routing entries are subsequently used by the forwarding engine Forwarding engine: • Transfer incoming IP datagram to an outgoing interface directed towards a router closer (next-hop) to the traffic destination by performing a longest match prefix lookup on forwarding entries stored in forwarding information base (FIB) using the incoming IP datagram destination address

28. Control Forwarding Pol Pol Shap. Shap. Architecture of a normal IP router Class. Class. Routing table Routing protocol The "best" paths selected from the routing table built by the routing protocols are installed in the forwarding table IP packets Forwarding Table IP packets IP packets Forwarding decision based onlongest prefix match Update of TTL and checksum fields in IP datagrams (packets)

29. Internet Routing Protocols • Exterior Gateway Protocol (EGP) • Routing of IP packets between domains • Each domain is considered as a blackbox • Interior Gateway Protocol (IGP) • Routing of IP datagrams inside each domain • Only knows topology of its own domain (all routers within given AS managed by a single admin unit) Domain4 Domain2 Domain1 Domain3

30. Inter vs Intra-domain Routing Protocols IGP: Intra-domain routing (within AS) • Allow routers to transmit IP packets towards their destination along the best path = shortest-path (metrics: #hops, link cost) • IGP routing protocols: distance vector or link state • All routers exchange routing information: each domain router can obtain routing information for the whole domain EGP: Inter-domain routing (between AS) • Routing policies based on business relationships • No common metrics, and limited cooperation • Policy-based, path-vector routing protocol: external/internal Border Gateway Protocol (eBGP/iBGP) eBGP eBGP iBGP IGP eBGP eBGP eBGP eBGP

31. Session 2a - Outline 1.Organization of the Internet 2. Evolution of the Internet • Number of Hosts • IP address allocation • IPv4 address allocation • IPv6 address allocation • Number of AS • Routing tables • Size of the IPv4 BGP routing tables • Size of the IPv6 BGP routing tables • IP traffic flows • Bandwidth 3. Internet Topology modelling

32. Growth in number of Internet hosts Number of Hosts on the Internet: Aug. 1981 Oct. 1984 Dec. 1987 Oct. 1990 Jul. 1993 Jul. 1996 Jul. 1999 Jul. 2004 Jul. 2005 Jul. 2006 Jul. 2007 213 1,024 28,174 313,000 1,776,000 19,540,000 56,218,000 285,139,000 353,284,000 439,286,000 489,774,000

33. Growth in number of Internet users Number of Users over Time (from Dec’95 to Mar’08)

34. Issues with the current Internet architecture Limited size of IPv4 addressing space • NAT, CIDR and IPv6 have been proposed to overcome this limitation Projected Address Consumption (/8s) Source http://www.potaroo.net/tools/ipv4/index.html

35. Issues with the current Internet architecture • Exhaustion date of first RIR available pool of addresses (and no further numbers available in IANA unallocated pool to replenish RIR's pool) - Best fit predictive model predicts occurrence on Dec 2011 • Exhaustion of IANA unallocated number pool - Model predicts occurrence on Feb 2011 Source http://www.potaroo.net/tools/ipv4/index.html

36. IPv6 usage: advertised prefixes source http://bgp.potaroo.net/v6/v6rpt.html

37. Current IPv6 usage: ASes using IPv6 Ratio: prefix/AS ~ 1 source http://bgp.potaroo.net/v6/v6rpt.html

38. Issues with the current Internet architecture (2) Interdomain routing scalability • Growth of BGP IPv4 routing tables Growth is back again ! Internet bubble: growth is back Classless Inter-domain routing (CIDR) as reaction to running out of class B: RFC 1338 (Jun.92) - RFC 1519 (Sep.93) CIDR works well Bubble explosion pre-CIDR fast growth Source : http://bgp.potaroo.net

39. Growth of Active BGP Entries in FIB (from Jan’89 to Mar’08) Jan.1 2006 – FIB Size: 176,000 prefixes – Update Rate: 0.7M prefix updates / day – Withdrawal Rate: 0.4M prefix withdrawals / day – 250Mbytes memory – 30% of a 1.5Ghz processor ~25% ~15-20% RIB/FIB ratio (779057/266725): 2.9208 (*) Jan.1 2009 - FIB size: [275,000;300,000] prefixes - Update Rate: 1.7M prefix updates / day - Withdrawal Rate: 0.9M withdrawals / day - 400Mbytes Memory - 75% of a 1.5Ghz processor Jan.1 2011 (low-end predictions) - FIB Size: [370,000;400,000] prefixes - Update Rate: 2.8M prefix updates / day - Withdrawal Rate: 1.6M withdrawals per day - 550Mbytes Memory - 120% of a 1.5Ghz processor 09 (*) - RIB/FIB ratio can vary from ~3 to 30 (function of the number of BGP peering sessions at sample point) Source: BGP Routing Table Analysis Reports on AS65000 - http://bgp.potaroo.net

40. Issues with the current Internet architecture (3) Reasons for the BGP growth • Number of distinct ASes ? Number of unique ASN advertised in BGP routing table over Time 29.227 post-boom period sharp growth during the Internet boom period from 1999 until early 2001 Ratio: prefix/AS ~ 10 pre-Internet boom prior to 1999 Source: http://www.potaroo.net/tools/asn32/

41. Issues with the current Internet architecture (3) Unadvertised ASN count = Assigned ASN count - Advertised ASN count Number of advertised and advertised ASs over Time Ratio of unadvertised to advertised ASN over Time

42. Expansion of Internet between 2005 and 2006 IPv4 in 2006 Total BGP FIB entries over Time Prefixes: 173,800 – 203,800 (+17%) AS Numbers: 21,200 – 24,000 (+13%) Addresses: 87.6 – 98.4 (/8) (+12%) Average advertisement size: smaller (8,450 – 8,100) Average prefixes per update: smaller (2.1 - 1.95) Average address origination per AS: smaller (69,600 – 69,150) Average AS Path length: steady (3.4) AS transit interconnection degree: growing (2.56 – 2.60)  IPv4 network becomes • denser (more interconnections) • finer levels of advertisement granularity (more specific advertisements) Higher levels of path exploration before stabilization on best available paths Source: IEPG, <http://www.potaroo.net>

43. 194.100.10.0/23 194.100.0.0/16 200.0.0.0/16 I can reach 194.100.0.0/16 I can reach 200.0.0.0/16 I can reach 194.100.10.0/23 Issues with the current Internet architecture (4) Client : AS4567 R1 R3 R2 I can reach 194.100.10.0/23 Reasons for the BGP growth • Multihoming and 194.100.10.0/23 Provider AS123 and 194.100.10.0/23 Internet Provider AS789

44. 200.0.0.0/16 194.100.10.0/23 194.100.0.0/16 I can reach 194.100.11.0/24 I can reach 194.100.0.0/16 I can reach 200.0.0.0/16 and 194.100.10.0/23 Issues with the current Internet architecture (5) Client : AS4567 R3 R1 R2 I can reach 194.100.10.0/24 and 194.100.10.0/23 Reasons for the BGP growth • Traffic engineering and 194.100.11.0/24 and 194.100.10.0/23 Provider AS123 and 194.100.10.0/24 and 194.100.10.0/23 Internet Provider AS789

45. Issues with the current Internet architecture (6) BGP messages processed by routers • Hourly average prefix update rate (per second) http://bgpupdates.potaroo.net/instability/bgpupd.html

46. Issues with the current Internet architecture (7) BGP messages processed by routers • Hourly peak of per second prefix update rate http://bgpupdates.potaroo.net/instability/bgpupd.html

47. Issues with the current Internet architecture (8) Interdomain routing security • Only Best Current Practices from network operators prevent a customer network from using BGP to announce the prefix of someone else • Misconfigurations (fat fingers) are frequent http://www.ripe.net/news/study-youtube-hijacking.html Evolution-Internet-Architecture/2008

48. Internet architectural evolution: Sequence of reactive updates • 1978-1983 • 1982 • 1980s • 1988 • 1989 • 1992 TCP split into TCP and IP - Cutover from NCP to TCP/IP as a reaction to the limitations of NCP DNS as a reaction to the net getting too large for hosts.txt files EGP, and OSPF as reactions to scaling problems with earlier routing protocols TCP congestion control in response to congestion collapse BGP as a reaction to the need for policy routing in NSFnet CIDR as a reaction to running out of class B

49. Internet architectural evolution: Sequence of reactive updates … as the Internet become bigger, it becomes a lot harder to change while the Internet is accumulating problems faster than they are being fixed NA Internet in March 2006 ARPANET in 1974 62 host computers (37 nodes) Red: Verizon, Blue: AT&T, Yellow: Qwest, Green is other backbone players like Level 3 & Sprint Nextel, Black: entire cable industry together, Gray: everyone else Source: http://som.csudh.edu/cis/lpress/history/arpamaps/

50. Outline 1.Organization of the Internet 2. Evolution of the Internet 3. Internet Topology modelling • Network properties • Random Graphs models and generators • Structural models and generators • Topology measurements • Power Law relationships • Degree-based models and generators • Internet topology metrics