Operating a Large Scale Multi-Location Internet Exchange Point Andre Els – Senior Network Engineer

1. Operating a Large Scale Multi-Location Internet Exchange Point Andre Els – Senior Network Engineer SAFNOG 1 – 22 April 2014

2. The London Internet Exchange Introduction • The London Internet Exchange facilitates Internet interconnection in the UK through Public Peering, for carriers worldwide. • LINX operates as a neutral & mutually-owned membership association; owned and operated for the members that it serves. • Currently serves members based in around 60 countries. • 65 of the top 100 global networks: • Almost all UK ISPs • Content • BBC, Microsoft, Yahoo!, Google, Facebook, Twitter,… • Content Delivery • Akamai, Limelight, Edgecast, BitGravity,… • Education and Research • Janet, NORDUnet, TENET, Eduserve

3. Southern African Members … to name a few

4. AS Count, by RIR 03/2003-10/2013

5. AfriNIC members at LINX 2003-2014

6. LINX London LANs – Quick Stats 511 members. 1262 connected member ports. 747 member-facing 10G ports. 2 member-facing 100G ports. 60 member countries. +/- 900 pairs of PI fibre. Around 400k prefixes.

7. Locations North Virginia Manchester Scotland London

8. London PoPs • Slough • Equinix LD4 • London West • TeleCity Powergate • London City • InterXion • London Docklands • TeleHouse North • TeleHouse East • TeleHouse West • TeleCity 6/7 HEX • TeleCity 8/9 HEX • TeleCity Sov House • TeleCity MillHarbour

9. United States PoPs – LINX NoVA • Manassas • EvoSwitch • Ashburn • DuPont Fabros (DFT) • Reston • CoreSite LINX NoVA – First IX in the world to be awarded Open-IX Certification. http://www.open-ix.org

10. Our challenges of running a large IX • Traffic growth – We need to accommodate for ~40% growth each year. Almost doubles every 2 years. • Density – ~1200 connected ports across 10 sites with high traffic flows (~2Tbps). • Scalability – Bigger ISLs, more routers/switches. Rack space and power. • PoP locations – Longer fibre distances, optical budgets.

11. Traffic Growth • LINX used to be limited to a few major data centres in the Docklands area. • This provided easy access to diverse, abundant and inexpensive dark fibre. • Running at 8x1G and later 10G, now 100G • The need for bigger LAGs and more sites, required more capacity. • Passive DWDM solution was best suited to our needs, due to simplicity.

12. Passive DWDM Core Network 32 x 10G

13. Passive DWDM Edge Network

14. 100G at LINX • We are moving away from the bundles of Nx10Gb and introducing 100Gb in the core of our Primary LAN (Juniper-based) • 64 x 10Gb link between two busiest POPs was recently replaced with a 12 x 100Gb bundle • Optical transport (MRV DWDM) replaced with diverse dark fibre

15. Passive DWDM 10G to 100G

16. Scalability Primary LAN Juniper (VPLS) - MX480 - MX960 - PTX5000

17. Scalability Secondary LAN Extreme (L2) - X460 - X480 - X670 - BDX8

18. Fibre lengths vs Optical budgets Typical 80km DWDM Optical Transceiver 24db optical budget 78km dark fibre ~ -20db Total loss end-to-end: ~ 20db

19. Fibre lengths vs Optical budgets Typical 80km DWDM Optical Transceiver 24db optical budget data centre data centre -3/4db -3/4db 78km dark fibre ~ -20db Total loss end-to-end: ~ 26db

20. Fibre lengths vs Optical budgets Typical 80km DWDM Optical Transceiver 24db optical budget data centre data centre -3/4db -3/4db -4.5db -4.5db MUX MUX 78km dark fibre ~ -20db Total loss end-to-end: ~ 35db

21. Operational Challenges • Membership – Many members do not know who or what LINX does, but then, we do have members from 60 countries. • Communication – Ill-informed member NOCs. • Support – Providing “expected” 24/7/365 technical support.

22. Operational Challenges • Membership – Many members do not know who or what LINX does, but then, we do have members from 60 countries. • Communication – Ill-informed member NOCs. • Support – Providing “expected” 24/7/365 technical support. • Beer – Not to drink too much beer at LINX social events :-)

23. Questions ? Andre Els andre@linx.net http://www.linx.net