Gigabit Ethernet: Shared Risks and Future Developments

1. Exchange Point TechnologyNanog 20, Oct 22-24, 2000 Lane Patterson Member Research Staff <lane@equinix.com>

2. Intro Framework

3. IX Technology Universe • 10/100 Ethernet, FDDI • ATM OC3/12 (SAR limitations) • GigE • Private Copper/Fiber Cross Connect • SONET ADM/DCS • Frame Relay on POS OC48/192 • LSR (over POS, Ethernet, …) • 10GigE • Optical Switching (w/ signaling of some sort)

4. IX Connection Models • Layer 2 Non-Broadcast Multi-Access (NBMA): • TE and connection-oriented = more control at the expense of management and operational overhead • Frame Relay or ATM PVCs • MPLS LSPs • Even 802.1p/q in a “VLAN per /30” model • Layer 2 Shared Media • FDDI, Ethernet, DPT/SRP • Layer 1: • SONET DCS style: STS-1, OC-N • Wavelength, Dark Fiber • Nailed Up TDM Today; GMPLS tomorrow?

5. IX Applications: Current and Potential • Unicast Peering Aggregation • Private or Public • Multicast Peering Aggregation • Interdomain Interconnection Policy • Human Enforced Clue • Route Servers • Interdomain MPLS/GMPLS? • QoS Brokering? • Not focusing outside of interconnection in this talk (e.g. Stratum 1 servers, other services)

6. What We’re Doing

7. Gigabit Ethernet • The one big Unicast VLAN model • But private VLANs and tagged interfaces OK • Must Mitigate Shared Risk: • IX Policy and Enforcement • Switch Filters and Knobs • Cost, simplicity, and operational advantages over ATM • More traffic control features now on GigE lowers the contrast with ATM • MTU a problem? • Lots of “core Internet MTU” debate • But more router data helpful—flow cache packet distributions, frag stats, better analysis of packets that are >1500, etc. • Waiting on vendor support: MTU per VLAN tag • Will be announced as a product • I’m the R&D guy not the product guy—usual non-Nanog channels apply

8. GigE Shared Risks • No Brainers: Participants • ARP: no ip proxy-arp • Broadcast: no ip directed-broadcast • ICMP Redirects: no ip redirects • CDP/other noise: no cdp enable • No IGP: passive interface • No 3rd party switches, BPDUs • Multicast: keep off unicast exchange VLAN

9. GigE Shared Risks (cont’d) • IX-side • STP tuning a must • Upcoming 802.1w and STP improvements from today’s 2-4 second reconvergence to milliseconds • Block BPDUs on non-IX-trunk ports • Layer 2/3 filters per port • Trend and alarm bridge tables (# and identity of MACs on each port) • Some controls still needed that are easy • Some controls may not be worth it: • Too much maintenance overhead for ISPs • Static ARP, MAC per PORT/VLAN • Shaping/Policing policies • Filter Maintenance • If we really want this, go back to NBMA models

10. Future GigE Developments • Vendors are active • 10GigE • Other proprietary features for control, scaling • Active 802.1 projects • Still Need Multicast IX Features • PIM snooping for Port/Group state

11. What We’re Tracking/Testing

12. SONET Cross-Connects • Aggregate private circuits through SONET DCS • Assumes OC48/192 Ports Channelizable to OC-N (both on Router and DCS) • Recent DCS Products: up to 512 OC48 down to STS-1 • Worthwhile for co-located routers? • Already precedent: FloridaMIX • Good combo with DWDM into building • Extends current SONET aggregation at DS3 & OC12 • Cost of SONET DCS ports vs. Other Alternatives • Policed Ethernet dot1q alternatives • Cost/density/flexibility trade-offs versus continuing to burn DS3/OC3/OC12 router ports • Will report back on future testing progress • Input is welcome

13. NBMA Migration Paths • Frame Relay at OC48/OC192 most promising “ATM Upgrade” today • Mature standards • Switches becoming available • MPLS LSRs need more momentum • Still intra-domain focused and limited set of ISPs • Inter-domain signaling issues • Promising longer term for new inter-provider features • MPLS/BGP VPN • Inter-provider QoS • Dynamic B/W provisioning: circuit-em, signaling • None of these proven yet

14. NBMA Migration Paths (cont’d) • Equinix committed to more MPLS LSR Testing • Current Inter-Domain Models • Non-terminating IX LSR: • Static Labels • RSVP+ERO but no IGP—statics across directly-connected IX /30’s • Terminating IX LSR: • Add CCC-style to above combinations • Need more robust Inter-Domain Signaling models • MBGP NLRI for label exchange a first step • Bilateral policy directly on LSRs or real 3rd party gateway infrastructure? • Same issues carry over to GMPLS and optical world

15. Optical Switching • Dense numbers of signaled wavelength and fiber cross connects • Testing one OXC vendor near term • Testing open to interested parties • Still early in development • For now, wavelength exchange is just a private cross connect between customer-owned DWDM equipment • Fiber/wavelength density and signaling-driven applications will drive this onto OXCs • Then Bill Norton can do some whiz-bang cost benefit graphs • “Multiservice Exchange”

16. Virtual Device Trends… • …or lack of them • Useful for customer-driven, on-demand provisioning • Today not many real virtual devices, just NMS/OSS integration through secure web front-end. • Future boxes that can push this: • 3rd party policy brokers • Register policy • Integrate statistics for operation, billing • Admission control in secure signaling domain • Real virtual devices: • Log in to your virtual instance on a shared box • Bandwidth trading systems

17. Going Forward • Large IX Facilities: • Focus on removing space, power, and interconnection constraints from scaling equation • Continued ability to accept new participants • Only limit should be strands*wavelengths into the building/campus • Today, • Public Fabric: 10/100/1000 Ethernet moving to 10Gig • Private Fabric: dark fiber cross connects • Under Consideration: SONET cross-connects and aggregation onto OC12/48/192

18. Last Thoughts • When future cores are 80 wavelengths of OC192, and migrating to 320 wavelengths of OC768, what do exchange points need to look like? • Other than more bandwidth, what inter-domain services will take hold? • Will we have any hair left by then?