CA*net 4 Optical Networking and Third Wave of the Internet

1. CA*net 4 Optical Networking and Third Wave of the Internet Bill St. Arnaud CANARIE Inc – www.canarie.ca Bill.st.arnaud@canarie.ca

2. Overview • 1. CA*net 4 Network Architecture • 2. Applications that may drive require a new network architecture

3. CA*net 4 Drivers-1Reduce cost of Internet • Set up lightpaths to no cost peering exchanges • Most lambda sales in Canada and USA are for “Remote peering” to no cost peering points • Allows for considerable savings in Internet transit costs • Each lightpath is directly connected to a high volume peer and bypasses peering router • Good example is “STAR LIGHT” where high volume peers have direct connect and small volume peers use a router • CA*net 4 “customer controlled patch panel” allows peers to change peering relationship remotely without contacting technical staff at peering exchange • Very similar in concept to WorldCom “Peermaker” at MAEs which use e-mail to setup peers

4. CA*net 4 Drivers-2Reduce cost of routers • Eliminate expensive high end routers and replace them with lower cost optical switches • But circuits are NOT intended to replace packet networks • Use rich mesh of circuits between edge routers to eliminate high cost of 10GbE core routers • 10Gbe routers ~ $500K with interfaces at ~$200k each • 10Gbe switches ~$25K with interfaces at ~$20k each • Trade off between cost of multiple lightpaths and loss of multiplexing versus cost of high end core routers • 10Gbe wavelengths ~$1000/km for 5 years (lifetime of router) • Assume 1 GbE lightpaths per edge institution then • One 5000km Gbe lightpath (or 8 x 600km GbE) lightpaths per institution is cheaper than routers

5. CA*net 4 Drivers-3A VPN alternative to MPLS • Allows customer to create “customer owned and managed” networks with resource heterogeneity • Integration of wavelengths and dark fiber from different carriers • Create customer controlled VPNs for downstream users and overlay networks across multiple suppliers • Customers can manage their own restoral and protection schemes • Allows for inter-domain end to end setup of VPNs • End users do not need to to signal carrier for VPN management • Create VPNs • Cross connect VPNs from independent users • Partition or spawn VPNs • Establish VPNs across multiple management domains

6. CA*net 4 Drivers-4Application Specific Networks • Lambda Grids - “Underlay” networks to support Grids and overlay projects like PlanetLab and Oceanstore • A lot of exciting research into overlay networks • At some point in time when traffic volume is sufficient in overlay network to setup its own direct path • Soon high end grid applications will have sufficient traffic volume to require their own underlay networks ”Complementing” routed networks • Not a replacement for routed networks – only increasing the direct peering mesh of the routed network for specific applications • But peering may be more dynamic (and not globally advertised) than traditional IP BGP peering • Discipline or applications specific networks • VBLI grids like European EVN • High energy physics grid – Ultralight • NEES grid, Bio-informatics Grid, etc

7. NORDUnet JANET SURFnet PSNC DFN GARR Example – EVN traffic flows over GEANT SE UK JIVE PL CZ BE NL DE2 DE1 FR CH AT 2.5G 10G IT Provided courtesy of Dai Davies

8. Issues • How do you charge for bandwidth and usage when single application traffic dwarfs all other IP traffic? • Who pays for the traffic volume when it sinks into one NREN? • Possible solutions: • GMPLS (with QoS) • Requires expensive routers and complex coordinated central management to setup and tear down tunnels • Does not address issue of traffic charging • Interdomain still unproven • Optical overlay/underlay –ASON • same problems as GMPLS • Application specific optical BGP networks • Increase BGP mesh for specific applications or disciplines

9. NORDUnet JANET SURFnet PSNC DFN GARR OBGP applied to EVN Express route SE JIVE UK PL CZ BE NL DE2 DE1 FR CH AT • EVN sites will see 2 BGP routes to SURFnet: • the normal IP route over GEANT • Express route using dedicated lightpaths (in green) IT

10. CA*net 4 Drivers-5QoS • Spatial QoS • TCP throughput over long fat pipes very susceptible to packet loss, MTU, TCP kernel, Buffer memory, AQM optimized for commodity Internet, Auto negotiating Ethernet, etc • May also require consistent and similar TCP throughput for multiple sites to maintain coherency for grids and SANs • Some exciting new TCP protocols like FAST, XCP, etc • Mice and Elephant problem • Without careful design may look like a DOS attack on a router network • Many commercial SAN/Grid products will only work with QoS network • Some users want to have super jumbo MTU (64K) or protocols other than IP

11. Spatial QoS Normal BGP path y.y.y.1 x.x.x.1 Optical “Peermaker” Only y.y.y.1 advertised to x.x.x.1 via OBGP path Only x.x.x.1 advertised to y.y.y.1 via OBGP path OBGP path Application or end user controls peering of BGP optical paths for transfer of elephants!!!

12. CA*net 4 Drivers-6Extend E2E principle to circuits • Extend the Internet end to end principle to circuit based networks • The success of the Internet is largely attributable to the classic e2e principle • Allowed development of exciting new applications or services • E2E principle presumes all users behave honorably • Otherwise you get spam and DDoS attacks • This ungentlemanly behavior can be reasonably contained on commercial Internet but pose big challenge on research networks • What happens if you have a malicious implementation of FAST or XCP on a big pipe network? • Can the Internet e2e principle be applied to circuit based networks? • Will it engender the same creativity in new applications and services? • MPLS and ASON are classic network state based solutions for VPNs • CA*net 4 architecture is an alternate approach • All VPNs are BGP direct static routes using lightpaths that are setup and controlled by end user

14. CA*net 4 is NOT a network • It is an aggregation of point to point 10 Gbps wavelengths from a number of carriers • CA*net 4 is made up of may parallel application or discipline specific networks that may (or may not) BGP peer with each other • The wavelengths and switches are partitioned into smaller lightpaths with user control of the switch partition which are used for a variety of applications particularly grids • International Grid Testbed – 10 Gbe server to server to CERN • WESTgrid – 1 Gbe lightpaths for distributed backplane • Numerous lightpaths to support direct peering between regional networks and universities • Lightpaths to support TransLight projects between North America, Europe and Asia • Many, many more coming – Virtual Astronomy, HDTV video walls, etc

15. Applications for E2E Lightpaths

16. The three waves • The first wave of the Internet consisted primarily of text and data services such as e-mail and FTP. • The second wave was the web which improved ease of use and facilitated the transfer of images, sound and video. • The third wave is the integration of applications, p2p networking, open source, distributed computing enabled by next generation web services, semantic web and high speed networks

17. What is the Third Wave? • Before the Web on-line information was only available through a small number of information providers who charged high fees • Compuserve, Dialogic, etc • The Web allowed millions of creators of information to make it easily accessible to all others at very low cost, bypassing the information middleman • The Third Wave proposes to extend the WEB paradigm to processes, applications and content • Third Wave is about creation of tools and applications (i.e. services) in variety of fields such as eLearning, eBusiness, eScience, eHealth, etc that can make these services easily available to all others • At there are millions of web sites, there will be millions of Third wave services

18. Today’s Network The network is subservient to the computer The application is tightly bound to the OS Network Application Application User User OS OS The network is a mechanism for applications to communicate with each other Data Data

19. Application and data exist on the network and are uncoupled from any specific machine or location Third Wave Third Wave Network OS OS The computer is subservient to the network Application and Data Third Wave Third Wave Third Wave OS OS OS Third Wave Data Data Data OS Data Third Wave Network

20. A new way of doing science • Science used to about test tubes, wet labs and big instruments • But increasingly science is moving to networks and computers • Science is now longer bound by bricks and mortar or geography • NSF has announced “Cyber Infrastructure” initiative • https://worktools.si.umich.edu/workspaces/datkins/001.nsf • DOE SciDAC “Scientific Discovery through Advanced Computing” • http://www.er.doe.gov/feature_articles_2001/august/SCIAC/SciDAC_announcement.htm • Recognition that more and more science is network and computationally based • Grids using web services will be foundation of this new research methodology

21. Sloan Digital Sky Survey ALMA Many e-Research Projects Coming LHC ATLAS

22. Online System Tier2 Centre ~1 TIPS Caltech ~1 TIPS Tier2 Centre ~1 TIPS Tier2 Centre ~1 TIPS Tier2 Centre ~1 TIPS HPSS HPSS HPSS HPSS HPSS 1 TIPS is approximately 25,000 SpecInt95 equivalents Physicists work on analysis “channels”. Each institute will have ~10 physicists working on one or more channels; data for these channels should be cached by the institute server Pentium II 300 MHz Pentium II 300 MHz Pentium II 300 MHz Pentium II 300 MHz International Grid Testbed ~PBytes/sec ~100 MBytes/sec ATLAS Offline Processor Farm ~20 TIPS ~100 MBytes/sec Tier 0 CERN Computer Centre ~622 Mbits/sec per channel Tier 1 France Regional Centre Germany Regional Centre Italy Regional Centre FermiLab ~4 TIPS ~622 Mbits/sec Tier 2 ~622 Mbits/sec per channel Institute ~0.25TIPS Institute Institute Institute Physics data cache ~1 MBytes/sec Tier 4 Physicist workstations

23. International Grid Testbed • First production use of international e2e lightpaths to transfer Forward Calorimeter data from CERN to Canada • Experiments planned to directly transfer low level trigger data from CERN to Canadian computers • 10 GbE server to server experiments • RDMA, TCP/IP offload, etc

24. Virtual Observatory • http://www.us-vo.org/ • Discovery process will rely on advanced visualization and data mining tools • Not tied to a single brick and mortar location • Will cross correlate existing multi-spectral databases petabytes in size • Web services will integrate data and applications No new telescopes or radio dishes. Just big networks interconnecting large databases

25. Canada Virtual Observatory Data Flows • Flow to Terapix : 1TB/month • Flow to science centres: 0.5 TB/month • Return from Terapix: 2 TB/month • Over 5 years userswill download full dataset (100 TB) ~ 30 times: 50 TB per month • Average flow 154 Mbps for five years • User network load is the largest, least predictable, and least manageable component of the network traffic • Accessible to students at schools

26. Canadian Forestry Grid • SAFORAH (System of Agents for Forest Observation Research with Automation Hierarchies). • SAFORAH connects five locations across the country to support the monitoring of Canada's forests • Together, all five locations will generate data equaling 40 terabytes (TB) per month

27. Grids for Kids • The ultimate goal of Grids for Kids is to allow students and eventually members of the general public to be full participants in scientific discovery and innovation. • Will allow increasing number of computationally or networked research experiments to be seamlessly integrated with the computer capabilities of thousands of PCs located at our schools • Some early primitive examples…

28. FightAIDS@Home • Scientists at The Scripps Research Institute (TSRI) are using computational methods to identify drugs that have the right shape and interaction characteristics to fight diseases such as AIDS. • Once such candidates are identified, they can be synthesized in a laboratory, tested according to FDA guidelines, and released as prescription drugs to benefit the public. • Such computations require a vast number of trial dockings, testing variations in the target protein and the trial drug molecules

29. Folding@home • This "virtual supercomputer" uses peer-to-peer technology to make unprecedented amounts of processing power available to medical researchers to accelerate the development of improved treatments and drugs that could potentially cure diseases. • Rapid new discoveries in cancer research • Two projects in Canada: • Smallpox cure at UWO • Leukemia research at Mt Sinai

30. Climate Prediction • Predict future climate due to greenhouse affect • Distribute climate model to thousands of PCs worldwide • www.climateprediction.com

31. ALTA Cosmic Ray eScience • Collaborative scientific research project involving the University of Alberta Center for Subatomic Research and over 50 high schools across Canada in the area of cosmic ray detection. • Teachers and students actively contribute to the physics research while learning about an exciting area of modern science.  • Distributed computing at schools required to analyze data from sensors in near real time

32. Neptune/Venus Grid • Joint US-Canadian project for undersea dark fiber network off west coast of USA and Canada • Undersea network will connect instrumentation devices, robotic submarines, sensors, under sea cameras, etc • All devices available to students and researchers connected to CA*net 4 and Internet 2 networks Distributed computing and data storage devices on CA*net 4 and Internet 2 will be used to analyze and store data

33. Faulkes Telescope • Provide UK schools with access to a research class telescope in Hawaii • Provides an exciting resource for teachers to use via the Web • To provide a real-time experience of astronomy, through live use of a telescope • To allow students to participate in real research programs, mentored by professional astronomers • Provides other public interest groups, such as amateurs, access to high quality astronomical data • http://www.faulkes-telescope.com/

34. More Information • http://www.canarie.ca/canet4/library/canet4design.html • http://www.canarie.ca/canet4/obgp/index.html • http://www.canarie.ca/canet4/library/customer.html • Thanks to the design teams at Carleton U, Ottawa U, CRC, UQAM, UoWaterloo, Montague, etc

35. Background Slides

36. Grid Application User Access Layer Grid Service Interface J2EE Application Server LPO Service EJBHome GT3 Hosting Environment LPO Grid Service LPO Factory Service LPO Service Implemen- tation RMI LPO Service EJBRemote LPO Delegate Service RMI JDBC RMI DB Resource Management Layer Create service Access service User Controlled Service Provisioning Layer Using Third Wave Service to manage a network

37. OGSI-conformant services OGSI-conformant services • LPO advertisement • LPO query • LPO termination • LPO access • LPO reconfiguration • LPO spawning • LPO concatenation • End-to-end LPO establishment

38. RMI Service Provisioning Layer Resource Agent Request Controller LPO Controller LPO Controller LPO Controller LPO Controller Programmable Controller LPO Space LPO Controller RMI Switch Interface TL1 CA*net 4 Resource Management Layer