The Next Wave of Massive Disruptions to the Peering Ecosystem

1. The Next Wave of Massive Disruptions to the Peering Ecosystem William B. Norton Co-Founder & Chief Technical Liaison Equinix, Inc. Asia Pacific Peering Forum Singapore, October 5, 2006 Slide set v0.9

2. Internet Operations White Papers Name: William B. Norton Internet Researcher What is Peering? When does it make sense?

3. On the Internet Everyone is a Publisher

4. Internet Operations White Papers • “Interconnection Strategies for ISPs” • “Internet Service Providers and Peering” • “A Business Case for Peering” • “The Art of Peering: The Peering Playbook” • “The Peering Simulation Game” • “Do ATM-based Internet Exchanges Make Sense Anymore?” • “Evolution of the U.S. Peering Ecosystem” • “The Asia Pacific Internet Peering Guidebook” • “The Folly of Peering Traffic Ratios?” Freely available. See Web site or send e-mail to wbn@equinix.com Internet makes anyone a publisher, similar effect now emerging for video

5. Massive Disruption in U.S. Peering Ecosystem  Short Videos • YouTube – founded 2005 • Short video clips – 50 million view per day! • 20Gbps of peering traffic Feb 2006 • $1M/month in Sept 2006! • Entering Peering Ecosystem • 30 Other competitors600Gbps peerable? • DoveTail • Video may dwarf current peered traffic • 2010 – 80-90% Internet is Video • Inculcate video guys into peering ecosystem On the Internet Everyone is a Broadcaster Short video clips…Full TV shows… Source: http://digg.com/tech_news/YouTube_Gets_Bandwidth_Boost_from_Level_3 Source: http://www.nanog.org/mtg-0606/norton.html

9. Massive Disruption in U.S. Peering Ecosystem  Full Episodes • “Desperate Housewives” – 210MB/hour • For 320x240 H.264 Video iTunes image • 10,000,000 households • 2,100,000,000 MB = 2.1 peta-Bytes • How long will that take to download? 3 days @ 64Gbps non-stop ! Just one show Try 250M*180 Channels*HDTV Historical Perspective…review 5yr disruptions… Source: http://www.pbs.org/cringely/pulpit/pulpit20060302.html

10. T1 ISPs T2 ISPs Content 2002: Evolution #1 Cable Companies Peer Evolution #1 Significant Evolution… • Volume of traffic is huge • Cable Cos Open Peering • “Kazaa Effect” amplifies • peering benefits • Scale: O(20Gbps) peered

11. 2002: Evolution #2 Large Scale Content Players Peer T1 ISPs Significant Evolution… • Volume of traffic is huge • Content is Open Peering • Improves End-User • Experience • 4) Leading Players are • paving the way • Scale: O(100Gbps) peered T2 ISPs T2 ISPs Content Content

12. 2002: Evolution #3 Cable Cos Peer w/Large Scale Content Players T1 ISPs Significant Evolution… • Volume of traffic pulled away • from T1s is huge • 2) Reduces perceived need for • T1s (for local delivery anyway) • 3) T1s still needed for distance •  Content Literally directly on • The Cable Company Network • Scale: O(100Gbps) T2 ISPs T2 ISPs T2 ISPs Content Content Content

13. 2006: Evolution #4 VideoPeering T1 ISPs Video Service Providers Significant Evolution… • Volume of traffic is huge • 2) Most Traffic is Regional • 3) Massive Growth • 4) Many Emerging players • 5) Video size growth • Scale: > O(600Gbps) T2 ISPs T2 ISPs T2 ISPs Content Content Content Notes: Questionable if aggregate capacity exists TBD Impact of CDN/P2P/Satellite/caching/etc. Net Neutrality Issues not considered here

14. Research Topic • Massive Wave of Internet Traffic • 90% of all Internet bits by 2010 • How will Video Service Providers distribute this massive amount of Video Traffic over the Internet?

15. Modeling the Video Service Provider Distribution Networks Four Models • Commodity Transit • CDN • Transit/Peering/DIY CDN • Peer2Peer Four Load Models A: Small Load B: Medium Load C: Large Load

16. Load Model A • Load Model A – Light Load: Every 5 minutes, 10 customers each start to download a 1.5 GB movie, resulting in an average 15GB five minute load. Adjust load to sinusoidal customer traffic demand curve Jeff Turner: 6.6:1 peak-to-mean

17. Load Model B • Load Model B – Medium Load: Every 5 minutes, 100 customers each start to download a 1.5 GB movie, resulting in an average 150GB five minute load. Adjust load to sinusoidal customer traffic demand curve Jeff Turner: 6.6:1 peak-to-mean

18. Load Model C • Load Model C – Large Load: Every 5 minutes, 1000 customers each start to download a 1.5 GB movie, resulting in an average 1500GB five minute load. Adjust load to sinusoidal customer traffic demand curve Jeff Turner: 6.6:1 peak-to-mean

19. Assumptions • 1000 full length 1.5GB videos • Equipment Costs • Transit Costs • Colo Costs • Staff Costs • Software=LAMP • Colo@IX • Multi-homed

20. Model 1: Commodity Transit Business Premise: • VSP focuses on core competence • Transit Providers handle traffic better and cheaper • Economies of scale, Aggregation, Expertise, Billing, Peering, etc.

21. Model 1A: Transit Light Load

22. Model 1B: Transit Medium Load

23. Upstream ISPs Model 1C Router4 Router2 10G Router2 8 * 10GE to upstreams each Server1 GigE Switch1 Router1 : Server24 : 10G : GigE Switch14 : : Server262 Server263 Server264 : Distribution GigE Switch 48 port GigE for servers 2 10GE for upstream $10,000 Add another every 24 servers Routers Cisco 6509Sup720-3bxl w/4*4-port 10GE, $150,000 80Gbps from switches, 80Gbps to upstreams

24. Model 2: Content Delivery Networks (CDNs) for the Distribution of Video Content • Business Premise: Single-site transit traffic traverses potentially many network devices, increasing latency and the potential of packet loss: • By spreading web objects closer to the eyeball networks latency is reduced • Fewer network elements are traversed so reliability is improved • Congestion points in the core of the Internet are avoided • CDNs have the expertise, deployed infrastructure, economies of scale from aggregation efficiencies.

25. Assumption CDN Price Points

26. Model 2A: CDN Light Load

27. Model 2A: CDN Light Load

28. Model 2B: CDN Medium Load

29. Model 2C: CDN Large Load

30. Model 3: Transit/Peering/DIY CDN • Business Model Premise: Operation of the Internet distribution is seen as strategic to the VSP: • End-user experience is mission-critical so outsourcing the end user experience to a transit provider or CDN puts the VSP at risk. • The VSP has visibility into what video are being released, which ones are likely to be hot and which ones don’t require special infrastructure adjustments. • Internet Video distribution is so new that the VSP prefers control. This is a strategic focus of the VSP: ensuring reliability, scalability, through the constant monitoring and evolution of the infrastructure to ensure the end user experiences during these early phases of Internet Video Distribution. • The traditional CDN may be ill-suited to distribute very large video object, therefore we have to do it yourself.

31. Model 3: Transit/Peering Light Load

32. Model 3A: Peering/Transit Light Load

33. Model 3B: Transit/Peering Medium Load

34. Model 3C: Transit/Peering Heavy Load

35. Model 4: Peer2Peer • Business Model Premise: The current Internet Service Providers and CDNs at the core can not handle the load across single or even multiple locations: • Backbone, peering interconnects, and the hundreds of thousands of routers deployed can not handle the load of today and tomorrows video. • the leaf nodes (i.e. Grandma’s PCs left on) in aggregate have the cycles and network capacity, if shared, to distribute popular content today. • Popular content can be chopped up into small chunks such that many downloaders become sources, and topologically close downloaders will prefer the topologically close sources. This ‘swarmcasting’ requires only a source ‘seed’, and a lookup mechanism for the first downloaders to find the seed, and then to direct future downloaders to topologically closer sources.

36. Model 4A: P2P Light Load

37. Model 4A: P2P Light Load

38. Model 4B: P2P Medium Load

39. Model 4B: P2P Medium Load

40. Model 4C: P2P Large Load

41. Model 4C: P2P Large Load

42. Summary Per Video Cost Of delivery

43. Acknowledgements • Vish Yelsangikar (NetFlix), Peter Harrison (NetFlix), Aaron Weintraub (Cogent), Jon Nistor (TorIX), Barrett Lyon (BitGravity), Dave Knight (ISC), Aaron Hughes (Caridien), David Filo (Yahoo!), Jim Goetz (Sequoia Capital), Jason Holloway (DoveTail), Matt Peterson, Richard Steenbergen (nLayer), Lane Patterson (Equinix), Eric Schwartz (Equinix), Pete Ferris (Equinix), David Cheriton (Sun), Andy Bechtolsheim (Sun), Jeffrey Papen (Peak Web Consulting), KC Broberg (Rackable), Henk Goosen (Sun), Geoffrey Noer (Rackable), Jeff Turner (InterStream/nuMetra), Vab Goel (NorWest Venture Partners), Phil Thomas (Quad)