SWITCHlambda - Experiences with national dark fibers @ SWITCH

1. SWITCHlambda - Experiences with national dark fibers @ SWITCH Willi Huber / Urs Eppenberger huber@switch.ch / eppenberger@switch.ch

2. The SWITCHlambda Project Project goals: • Provide fore front network services to the Swiss Universities at low recurring cost • Network scalable to multiples of 10 Gbps with low additional cost • Long living infrastructure, no need to change provider every couple of years • Independence of carrier market

3. SWITCHlambda Project Basic decisions: • Replacement of the STM-1 links by GigabitEthernet connections based on dark fiber • Long term lease of dark fiber, indefeasible right of use (IRU) for at least 10 years • Use of DWDM on main trunk lines • Redundant wide area links • Simple network design (no SDH, no ATM): Use of Ethernet-Interface all over the place

4. SWITCHlambda Network Design • Hierarchy of technologies • DWDM on main trunk lines (16 l possible today, expandable if later needed) • CWDM or single GE to smaller sites off the backbone • Transparent optical path between Zurich and Geneva • No regeneration between the two main sites of SWITCH • Optical path on main trunks designed for 10 Gbps / l • Chromatic dispersion compensated • Polarization mode dispersion under control • Bi-directional use of fibers • lower fiber costs

5. SWITCHlambda Network end of 2003 350 km Basel Zurich Bern Lausanne Geneva dark fiber based links (approx. 1200 km) leased GE link

6. Bi-directional Use of Fiber • Only one fiber used for the DWDM system • Second fiber free to connect smaller sites inbetween Universities with cheap GE technology DWDM node DWDM node n * 1 GE / 10 GE implemented with DWDM Router Router P2P GigabitEthernet- connections Router One pair of fiber Router

7. Typical SWITCHlambda node Bi-directional transmission mode: i.e. a single fiber is used for signals in both directions, leaving the second fiber free for future use.

8. Sorrento Gigamux

9. Sorrento CWDM JumpStart-400 CWDM System • 8 wavelengths (4 receive, 4 transmit) • Bi-directional operation • Optional backup path SWITCH uses currently 1 pair of Jumpstarts for two p2p GE connections over on single fiber.

10. Switch / Router Splitter Signal reflection (e.g. bad connector) Coloured GBIC Switch / Router Splitter OADM Coloured GBIC OADM Bi-directional GigabitEthernet Cheap solution for p2p GE connection over one single fiber with colored GBICs and passive optical devices

11. Testing 10 Gbps/channel capability Goals • verify equipment capability under real life conditions • verify optical design (dispersion compensation) • verify fiber quality for 10 Gbps application (PMD and other adverse characteristics) Realization • DWDM interface cards on loan from Sorrento (10GE WAN-PHY) • IP equipment from Force10 (2002) and Cisco (2003) • traffic generation with Linux based workstations and the free software “mgen”

12. 10 Gbps tests 10 Gbps/l, 360 km, bi-directional transmission • Test with signal regeneration optical - electrical - optical in the middle (at Bern) • Test with transparent optical signal Zurich - Geneva • Same as 2., but with additional attenuation of 3 dB of optical signal at the sender Each test: 8 Gbps of traffic during 35 hours, 0 bit errors • BER < 8 * 10-15 • SWITCHlambda ready for 10 Gbps !

13. Pros and Cons: Technical View • Stable, long living infrastructure • High future potential • This new stuff is fun • none

14. Is this reliable? • Stable infrastructure with occasionally long interruptions • Interruptions mainly due to planned maintenance • True fiber cuts are rare, but last for hours if not days (e.g. fiber on high voltage power lines, fiber along railway) • Experience of a big Swiss carrier: Fiber cuts happen mostly in the local loop (city) • Our experiences so far • Several interruptions due to planned maintenance (lasting up to 1 day) • Broken optical amplifier (interruption of 4 hours) • Redundant links are mandatory for a good sleep!

15. No connection, what now? • 24/7 monitoring by Ascom Netwatch • ping, SNMP • alarm SWITCH staff • basic analysis • SWITCH NOC for analysis and action decision • University IT staff • power problems • physical equipment check • Onsite support by Ascom Netwatch • localize cable problems with OTDR (optical time domain reflectometer) • Cisco hardware • Onsite support by Deltanet for Sorrento DWDM

16. Pros and Cons: Operational View • we are the boss • Big effort for negotiations with a lot of contract partners • no single provider to blame in case of connection loss • need a car to fix problems instead of telnet

17. Network Cost, general Remarks • Cost for wide area links • Well known in advance • Local loop cost • Vary a lot from place to place • Good (say cheap) solutions need a lot of planning and patience • Building own infrastructure is economically better than renting, but not possible in all cities • In some cities there is a de facto monopole of the city’s industrial works • The railways are an excellent partner for wide area links: low local loop cost because the schools are mostly located near railway stations

18. Network Cost, general Remarks (2) • Cost for optical equipment • DWDM is expensive • Long 10 Gbps/l DWDM trunks are very expensive (dispersion compensation, amplification) • CWDM stuff is getting cheap • Routers / Switches • Ethernet interfaces are cheaper by factors than SONET ones

19. Cost Structure Depreciation period used for the calculations: Fiber optic infrastructure (wide area & local loop): 10 years Fiber optic equipment DWDM & CWDM: 5 years 12 % wide area fiber 6 % 31 % local loop fiber DWDM equipment CWDM equipment maintenance & support 7 % 44 % Total yearly cost: 2.05 MEUR Cost include: Investment years 2001-2004 10 Gbps between the major sites maintenance & support

20. Comparison with predecessor SWITCHng SWITCHlambda Years covered: 1999-2002 2001-2004 Technology: ATM/STM-1 dark fiber, GE & 10 GE Sites connected: 13 universities 21 universities Yearly cost: 2.95 MEUR 2.05 MEUR

21. Isn’t it cheapter to buy it? We have not done a serious market review lately or even negotiated for capacity to get the real prices But we have a figure we assume is the best current offer on the market for high capacity links: 100 kEUR • limited to 300 Mbit/s • local loop must exist For 21 sites: 2.1 MEUR • Similar cost as SWITCHlambda,but less flexibility and performance.

22. Pros and Cons: Economic View • Low recurring cost • Low marginal cost for additional bandwidth • High investment • Uncertainity about development of bandwidth prices