Hawaii IPv6 Deployment Experiences

1. Hawaii IPv6 Deployment Experiences • Alan Whinery U. Hawaii Chief Internet Engineer President, IPv6 Forum Hawaii alan.whinery@ipv6hawaii.org Webcast for IPv6Hawaii.org (IEEE GLOBECOM 2009 IPv6 FORUM, November 30, 2009)

2. Acknowledgements • Antonio Querubin • Systems Engineer, Lavanet • Bill Becker • Network Administrator, Honolulu Community College • Pacific Center for Advanced Technology Training • Latif Ladid • President, Global IPv6 Forum • David Lassner • U. Hawaii VP for Info Tech • UH ITS Network Team

3. IPv6 is not a “project” • We don't have an “IPv6 person”, or an “IPv6 team”, or an “IPv6 initiative”. • It is our policy to deploy IPv6 where we deploy IPv4 • As upgrades or maintenance or changes are scheduled, IPv6 is on the to-do list.

4. IPv6 is not a “project” • Start now • Don't forklift • Consider IPv6 in the course of your design and purchasing decisions • Work toward including IPv6 in what you do.

5. Most things support IPv6 Now • Clients • Windows (XP,Vista,7) • Mac OS X • Router/Switch • Cisco, Juniper, Brocade (Foundry) • Server • Linux, Solaris, Win2003/2008, MacOS • Apache, BIND, Postfix, Sendmail

6. UH System Network • 18 Campuses and learning centers on 6 islands • ~50 separate research/outreach facilities • ISP for • State Government • Bishop Museum • Internet2 connector for • Dept. Of Education • NOAA NWS and NMFS

10. U. H. IPv6 History • January 2004: “turned on” peerings with 3 RENs • IPv6 Demos to Korea/Japan in 2004 • ITS network engineers and Honolulu Community College v6 enabled ever since • Addressing moved from “provider” to self in December 2007 • Turned on State-wide OSPFv3 and IPv6 addressing in 2008

11. U.H. IPv6 History • Initial campus deployment (2004) was through Cisco 7500 running an experimental IOS load • (Juniper supported in main release from 2003) • Distribution to user vlans was through a trunk into (inter-)campus VLANs • Although operational DNS servers running BIND were capable early, we waited for Infoblox appliance to do IPv6 records • For user interface

12. U.H. IPv6 History • Support for Cisco Catalyst 6500 was late in the game • Our production, mission-critical IPv4 multicast infrastructure had us running non-standard IOS in various places • Cisco 3550's are still not capable, but we anticipate IPv6 and other factors will drive replacement of those devices

13. Watershed Moments • Acquired address block 12/2007 • (2008) Long-awaited Cisco IOS Catalyst 6500 and 3750, permitting us to “go native” on most core infrastructure • In June 2009, we added IPv6 to our TWTC peering, which made “commodity” IPv6 viable • Prior paths to commercial sites were circuitous • Netflix “Instant” over IPv6 works well

14. Current UH IPv6 Deployment Status • DNS Stores and answers forward and reverse IPv6 records • Forward currently served in IPv4 packets • Reverses currently served in IPv4 and IPv6 • State-wide, 99% of facility gateway routers have IPv6 • 3 campuses have deployed IPv6 in some form • Manoa, Honolulu CC and Maui CC • UH Manoa has native v6 turned on to all user segments

15. v6 Peerings • UH • TW Telecom • Pacific Northwest GigaPoP • DREN (Hawaii Intranet Consortium) • Hawaii Internet eXchange • AARNET (Seattle, Sydney, [LA]) • Lavanet • Hurricane Electric • Sprint • AT&T • Verizon Business • TowardEX • Hawaii Internet eXchange

16. Lava.net Status • Multiple v6 peers • network core (unicast and multicast) • public servers • DNS hosting • Lavanet web site • mail (SMTP/IMAP/POP) • NTP • most office workstations • DSL or frame-relay (on request)

17. UH Short Term Goals • DNS • workflow • DNS • Address management • full services in IPv4 packets • full services in IPv6 packets • Dual stack all IP routing • Currently @ ~ 99% • Dual-stack all public-facing services • Currently @ ~ 5%

18. UH Short Term Goals • Provide v6/SLAAC to dual stack clients • Provide Tunnel endpoints for 6to4, Teredo, ISATAP • Currently have in-state 6to4 with Lavanet • Teredo is harder to affect • ISATAP is a “good idea”

19. UH Medium Term Goals • Develop single stack v6 capability • DHCP6 • Translation • Proxying • Move email infrastructure to IPv6

20. Mail

21. Mail • Careful planning will be necessary • Training the email sysadmins will be necessary • When email is not flowing, a crisis is occurring

22. The Big Island Router Memory Thing ($$$) • Problems on the south REN path from Mauna Kea Observatories on the Big Island of Hawaii • Cisco 3750's with BGP tables of ~12,000 prefixes • Adding IPv6 overwhelmed the fixed-config memory • We were required to re-design and buy an extra Juniper M7i to support v6 there • This would have occurred regardless of IPv6 • This is an expense to expedite, not simply to deploy

23. Cost • IPv6 is not value-added software • Cisco now has “feature parity” • Juniper has stopped charging for it • Most of our costs, Lavanet's costs are in staff time and training. • Lavanet has participated in Opensource projects and contributed IPv6 code • Cost can be controlled if you simply place IPv6 on your requirements list, start requiring it, and don't panic • The Big Island router memory re-design is so far the highest-cost IPv6 deployment measure (by far).

24. List of Problems: Native IPv6 Deployment To User Networks • Honest: not a single one.

25. UH Client OS Distribution Volume of HTTP GETs categorized by User-Agent

26. Out-Of-Box V6 Readiness

27. Tunneled Forms of IPv6 • Teredo (Significant incidental traffic) • Included with Windows XP, Vista, 7 • Used from behind NAT device (real/virtual) • 6to4 (some incidental traffic) • Included with Windows/Mac OS • tunnels via well-known address (i.e. 192.88.99.1) • ISATAP (?? traffic) • Included in Windows, some Cisco IOS • Tunnels through guessable domain name

28. Tunneled v6 In The Wild • Sources of incidental 6to4, Teredo seem to be applications which require IPv6, e.g. P2P clients • Teredo can be used as an indicator of NAT • There may be more insidious things present • Setting up local tunneling services can mitigate cost and issues for tunneled clients • Native IPv6 deployment should stop 6to4, but Teredo will persist from behind NAT • Un-managed tunnels can represent increased attack surface and firewall by-pass.

29. UH Teredo Traffic • All clients use one of three Teredo servers: • 207.46.48.150 (Microsoft Asia) • 213.199.162.214 (Microsoft Europe) • 65.55.158.80 (Microsoft USA) • NAT causes Teredo traffic • Virtual machine NATs cause Teredo traffic • Exceedingly complicated • Presumably initiated by an application install

30. SNMP • Cisco devised interim MIBs • Which persist in our current IOS • Cisco's MIB support notes are fiction • CISCO-IETF-IP-MIB

31. Graphing v4/v6 • The old MRTG model of graphing interface Octet-counts doesn't do per protocol accounting • Various non-optimal things can be done • ACLs feeding counters, etc • The following graphs were by using 8 “bpf” counters fed by individual filter expressions • No packet was examined • Not a scalable approach • Data represents 1 day on our TWTC v6/v4 peering

35. Performance • High-throughput transfers can exhibit throughput differences between v4 and v6 • Usually because v6 is being processor-switched (configuration?) • v4 and v6 paths to a resource often differ. • We have been happy if it worked at all • Now we need it to perform well • The time for optimization is at hand.

36. Comparing v6/4 paths (UH)

37. Comparing v6/4 paths (LavaNet)

38. Every Firewall, ACL,etc • Web server access controls • acls • firewall setups • PHP code to return restrict content based on IP address • MaxMind GeoIP • is v6 capable

39. IPv6 Deployment Scenarios • Laissez-faire (ignoring your destiny) • Your resources will be unreachable via IPv6 • external IPv6 resources will either be unreachable or tunneled per client • You probably have significant tunneled traffic now • Your existing IPv6 traffic will be high latency, poor performance • Possibly without the end-users' knowledge, they will simply blame you for bad performance • Or they will blame IPv6 and turn it off • Requires IPv4 addresses • “I don’t believe the v6 transition is occuring” • Means “I choose denial instead of participation”

40. IPv6 Deployment Scenarios • IPv6 Only ☺ - Client support sketchy - Translation necessary to reach IPv4 Internet + Some value in enabling v6-only servers • Dual-stack ☺☺☺ + Client support good + No translation necessary + Serves potential v6-only groups - Requires IPv4 addresses

41. Stateless Auto-configuration (SLAAC) • Many operating systems have IPv6 turned on by default • With SLAAC, if your router interface is using v6, then you are too. You may use v6 without realizing it • Your machine determines your IPv6 address, and adds it to the prefix advertised by the router • Some OS build the RH 64 bits using the MAC address • Others will make up random (currently only Vista and W7) • complicates address accounting/management

42. Getting a DNS Server address • Stateless auto-configuration gets you an address and gateway • But no DNS server • Of course, if you have DNS through IPv4, you will learn v6 addresses through that DNS server • Currently, the only way for a v6-only host to auto-learn the name server address is DHCPv6 • Attachments to SLAAC are proposed • RFC 5006 (IPv6 Router Advertisement Option for DNS)

43. IPv6: Apple OSX 10.4+ • On by default • Missing DHCP6 • Can't specify v6 address for networked printer, because the preferences pane for printer set-up considers a colon ‘:’ as preceding a port number (? 10.6) • Printer can, however, be specified by name

44. Apple OS X Applications • Firefox – once required v6 “turn on” • This seems to have changed • Safari – does browse IPv6 • ping – works with separate “ping6” • traceroute – works with separate “traceroute6” • SSH client – works • telnet – works to router: fe80::209:7bff:fedc:400%en0 • email – no server to test to yet

45. IPv6: Windows XP (SP2+) • You can add it to an interface with the interfaces “Properties” pane, just like IP(v4) or IPX/SPX or NetBIOS • Once added, there is no GUI config, although some things can be accomplished with the command line • Will not do DNS queries in IPv6 packets • Will receive IPv6 info from DNS in IPv4 packets • Is Ultimately doomed.

46. Windows XP Applications • Firefox – will browse IPv6 • IE7 – will browse IPv6 • ping – works • Tries first address as returned by DNS • tracert – works • Tries first address as returned by DNS • Telnet – doesn’t appear to work • Thunderbird – no server to test to yet

47. IPv6: Windows Vista and 7 • On by default • Does DHCP6 • There have been some problems • Passing of ICMP6 messages to applications

48. Windows Vista Applications • Firefox – will browse IPv6 • IE7 – will browse IPv6 • ping – works • Tries first address as returned by DNS • tracert – works • Tries first address as returned by DNS • Telnet – doesn’t appear to work • Thunderbird – no server to test to yet

49. IPv6: Ubuntu 8 • On by default • Does DHCP6, if you install it • Since Linux (and BSD OS) are typically used for reference implementations, support is pretty good

50. Ubuntu Linux Applications • Firefox – will browse IPv6 • ping – works as “ping6” • traceroute – works as “traceroute6” • Telnet – doesn’t appear to work • Linux is a kernel. • Linux distributions are operating systems. They differ as to what apps they provide for various roles. • “Distributions” means, Red Hat, Ubuntu, Suse, Debian, Slackware, etc.