Scaling Internet Hubs into the Next Decade

1. Scaling Internet Hubs into the Next Decade LESSONS FROM A MOUSE, AND OTHER DATA POINTS

2. Outline • Internet Growth Trends • How good is our crystal ball? • Next Gen Core Network Equipment • Exciting and Scary • Data Center, Power, and Cooling • Challenges, Strategies, and Hacks Scaling Internet Hubs into the Next Decade

3. Internet Growth Trends • Yearly traffic growth of 75-100% has been common for last 5+ years • Growth slowing to 55-65% in “broadband-penetrated” regions • However, much of the world population still underserved: parts of Eastern Europe, Middle East, Africa, South America, India, Asia • In already-penetrated markets, is growth just taking a breather before uptake of new technologies (GPON, 10GPON, 3G, LTE, DOCSIS3.0)? • Demand drivers are still there • Video over Internet – cacheable at edge via CDN • Recession may boost acceptance of Video Conferencing over Internet? • Up to 30 mbps and not cacheable! • Cisco Visual Networking Index – Forecast and Methodology 2007-2012 • Reasonable industry forecast, although plenty of fodder for argument Scaling Internet Hubs into the Next Decade

4. Next Generation Network Equipment • For largest Tier1 carriers, it appears Cisco CRS1-16 and Juniper T1600 will continue to dominate • Cisco CRS1-16 • 96 x 10G or 16 x 40G per chassis, future upgrades coming • Multichassis up to 8, already in deployment, eliminates intra-POP router trunks • Juniper T1600 • 80 x 10G or 20 x 40G per chassis • Multichassis TX Matrix (mature yet?) • DWDM—Continued improvement of serial speeds and λ densities • 40G OC768 cost now reasonable, dropping • 160 x 10G systems • Unrepeatered subsea, wider wave bands (> 200 λ), On-router 2000km optics Scaling Internet Hubs into the Next Decade

5. Future Path – Near Term • Routers/switches will reach 100-200+ Gbps per slot • Cisco CRS1-16 • Gen2 line cards probably 100GbE, N x 40G • Larger multichassis • Juniper – rumors of a T3200? • Force10, Foundry, Cisco Nexus: 100-230+ Gbps / slot • DWDM—Continued improvement of serial speeds and λ densities • Mass market 40G cost/density • Example: Infinera next-gen PIC will be 10 x 40G in a single slot • 100GbE in data center and metro ~ 2010? • DP-QPSK squeezes more bits per clock cycle using polarity and phase Scaling Internet Hubs into the Next Decade

6. Future Path – Longer Term • Next Next-Gen: 500Gbps / slot ~ 2011-2012 • 4-port 100GbE, 10-port 40GbE / OC768, and 48-port 10GbE line cards! • < 25watts per 10G port • 16-slot chassis ~ 19 kW • Still copper backplane • Next Next Next-Gen: 6+ years away ~ 2014/2015 • 1+ Tbps/slot enabled by short-reach silicon photonics = optical backplane • 25 x 40Gbps cheap fabric chip mass manufacturable VCSEL by Intel photonics fabs • Suitable only for < 1m distances across backplanes • Capability to dramatically lower power per Gbps • Deeper electron channels (using new chip substrates) reduce leakage • Silicon photonics greatly reduces copper trace distances • These technologies already lab-proven Scaling Internet Hubs into the Next Decade

7. Star Trek Stuff – R&D Frontiers • OOAM “twisted light” • http://billstarnaud.blogspot.com • http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=04388855 • [“Already researchers are using … quadrature phase shift modulation to achieve data rates in excess of 560 Gbps on a single wavelength… Optical Orbital Angular Momentum (OOAM) has the potential to add an almost infinite number of phase states to the modulated signal and further increase the capacity to thousands of terabits. Up to now the challenge has been how to couple OOAM modulated signals into single mode fiber. • Hollow Fiber • At least a decade away for Telco use! Currently in limited use in surgical applications • Honeycomb-lattice fibers allow photons to travel through a vacuum 95% of the time • Higher power input possible, much lower attenuation great repeaterless distance! • You can change the speed of light (e.g. get rid of 46% latency penalty for glass index of refraction @ 1550nm) Scaling Internet Hubs into the Next Decade

8. Network Kit Draw and Capacity Next gen gear to be < 25W per 10G port Scaling Internet Hubs into the Next Decade

9. Power and Cooling Trends, Tier1 Peering Presence • Current Observations • 2 – 4 chassis at 7.5kW average draw = 30kW Planned, by 2010 • 6 chassis at 10kW average draw = 60kW Planned, ~ 2012 • 10 chassis at 15kW average draw = 150kW Scaling Internet Hubs into the Next Decade

10. Managing Space, Power, and Cooling… …Strategies and Hacks Scaling Internet Hubs into the Next Decade

11. Data Center Campuses: Lessons from a Mouse • Disneyland: 160 Acres • Disney World: 30,500 Acres Scaling Internet Hubs into the Next Decade

12. Managing Space • In 2008, we started a 5 year planning process, • even though most participants only forecast 6-18 months out (yes, even the Tier1s) • In constrained / full sites, we prioritize peering customers over others for available space and power • We like campus plans, and so do our customers • Incremental, contiguous investment over time: think “Ashburn” • Permits private interconnects for large peers • Better critical mass • Space and power for caching servers • Eliminates metro builds and multiple POPs • Examples: • US: Ashburn, Secaucus, Great Oaks (planned) • EU: Equinix London Slough, Frankfurt, Paris for now Scaling Internet Hubs into the Next Decade

13. Perf Tuning…Managing Power and Cooling • Bolt on more DC Power where you can: • For 2008, we added total of 450kW to “full” sites • Another 500kW planned in 2009 • Improve PUE (cool gear more efficiently): • Less power to cooling equipment more power for customers  • Raised floor and slab floor solutions • Liquid cooling • Cold-aisle containment • Heat exhaust capture • Warmer data centers Scaling Internet Hubs into the Next Decade

14. Concentrated load (CRS-1) and overhead supplemental cooling Enclosed area – similar to shipping container approach, but much more flexible Energy savings of 15%+ Can be used in low-ceiling facilities Enclosed Space – CFD Analysis – Feasibility Evaluation Scaling Internet Hubs into the Next Decade

15. High Power Density … Liquid Cooling • 15-18% efficiency boost, leak detection and double-piping Scaling Internet Hubs into the Next Decade

16. Cold Aisle Containment • Cold-aisle containment at AM1 (Amsterdam) and PA2 (Paris) Containment ceiling (at top of cabinets) Containment doors at end of cold aisle Entire contained space can handle contiguous 15kW cabinets. Blanking plates essential for sealing airflow. Scaling Internet Hubs into the Next Decade

17. Heat Exhaust Capture • Funneling heat exhaust remains another retro-fittable way to improve cooling efficiency • Good for high-ceiling hot/cold aisle configurations, slab floors, to reduce air mixing (e.g. most US IBX locations) Scaling Internet Hubs into the Next Decade

18. Challenge: CRS heat exhausts at 2,000 linear feet/minute – hard to capture Solution: In high-ceiling data centers, 45 degree exhaust baffle (left) deflects heat upward toward return ducts Reduces air mixing significantly Also reduces cross-cage contamination Vendor Partnering Scaling Internet Hubs into the Next Decade

19. Considerations for Greener (and More) Power • Industrial Fuel Cells • Off-grid power • Available but not yet economical • Solar • Long payback, even with Tax credits • Less than 8% of load • But great for extra cooling power on summer heat peaks Scaling Internet Hubs into the Next Decade

20. Thank You.

21. Death of the Broadcast Segment • Flooding unknown unicast is so 1998 • Other topologies needed • Various options: • TRILL / Cisco DCE • MAC-in-MAC / PBT • MPLS / VPLS • Proxy ARP/TTL hack • Future wisdom Scaling Internet Hubs into the Next Decade

22. Thank You.