html5-img
1 / 45

WAN in Lab NSF Site Visit

WAN in Lab NSF Site Visit. John Doyle , CDS/EE/BE Steven Low (PI) , CS/EE Harvey Newman , Physics Demetri Psaltis , EE/CNS Steven Yip , Cisco. March 5, 2003. Reviewer concerns. Narrow focus on TCP/AQM A range of IST research at Caltech

lukas
Download Presentation

WAN in Lab NSF Site Visit

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. WAN in LabNSF Site Visit John Doyle,CDS/EE/BE Steven Low (PI),CS/EE Harvey Newman, Physics Demetri Psaltis, EE/CNS Steven Yip, Cisco March 5, 2003

  2. Reviewer concerns • Narrow focus on TCP/AQM • A range of IST research at Caltech • Spanning theory, implementation, experiment, deployment • WAN in Lab a critical component • Alternatives not discussed • Use spectrum of tools at different stages • How to manage and share WAN in Lab • Part of Federated Emulab • Both demand and excellent support for global sharing • Experience in global collaboration, e.g. Newman’s VRVS netlab.caltech.edu

  3. Agenda • EAS, IST Initiative, Theory program, FAST • Intellectual environment in which WAN in Lab fits (Murray, Doyle, Low) • WAN in Lab • Design, capabilities, alternatives, management (Low) • Cisco example & collaboration (Yip) • Education, outreach, poster session • Research talks • Projects that will use WAN in Lab • International collaboration, leverage & impact on HENP & Grids netlab.caltech.edu

  4. WAN in Lab Steven Low netlab.CALTECH.edu NSF Site Visit March 5, 2003

  5. A value of testbeds is “… building and maintaining research collaborations and communities” • NSF Workshop on Network Research Testbeds (Nov 2002) Why Testbed in IST “Prototypes & testbeds are required to gain acceptance of new concepts with potential user communities” “A lack of wide-area testbeds would contribute to a growing tendency towards paper solutions to thesis-factory problems, leaving the real networking world short of new ideas and technologies”

  6. Outline • Proposal summary • Basic design, equipment, costs • Unique features • Alternatives • Spectrum of tools • Emulated delay • Community resource • Demand • Management software • Why Caltech • Leverage on Abilene, HENP, CalREN, TeraGrid • Summary • Reviewer concerns • Review criteria netlab.caltech.edu

  7. Goal State-of-the-art WAN • High speed • 2.5G  10G • Large distance • 50 – 200ms • Controlled & repeatable experiments • Reconfigurable & evolvable netlab.caltech.edu

  8. OPM l1 l1 fiber spool S S R R l20 l20 S S S S R R R S S 500 km electronic crossconnect H : server : router EDFA EDFA Max path length = 10,000 km Max one-way delay = 50ms

  9. Equipment • 26 Servers • GbE cards ( 10GbE cards) • 12 routers • 10 Cisco 15454 with router blades • 2-port GbE, 8-channel OC48 • 2 Force10 E600 • 24-port GbE, 2-port OC48 • DWDM gears • 500km fiber • 6 EDFA • 2 Dispersion compensation modules • 2 optical mux/demux • Tektronix TDS7404 Oscilloscope • Integration with global network netlab.caltech.edu

  10. Costs • 26 Servers: $104K • 12 routers: $1.03M • 2 Force10 E600: $280K ($340K if OC192) • 10 Cisco 15454 with router blades: $750K ($810K if OC192) • DWDM gears: $148K • 500km fiber: $8K • 6 EDFA: $60K • 2 Dispersion compensation modules: $40K • 2 optical mux/demux: $40K • Tektronix TDS7404 Oscilloscope: $50K • Integration with global network: $110K • Personnel, software, service & maintenance • Total: $2M (NSF) + $0.67M (cost sharing) netlab.caltech.edu

  11. Yearly costs • Year 1: $1.128K • 10 servers, 5 routers, 2.5Gbps • Year 2: $564K • 20 servers, 8 routers, 2.5Gbps • Year 3: $124K • Software development • Year 4: $733K • 26 servers, 10 routers, 2.5Gbps • Year 5: $120K • Software development • Total: $2M (NSF) + $0.67M (cost sharing) netlab.caltech.edu

  12. Jorgensen Lab Networking Lab • WAN in Lab • 3 racks, 2 consoles • Networking Lab • 424 sq ft • Next to CACR • Easy connection to global network • Renovation (cost sharing) • New IST Building NetLab netlab.caltech.edu

  13. l1 l1 R1 R2 l2 l2 l3 l3 l4 l19 l18 l19 R10 l20 l20 (a) Physical network Unique capabilities • WAN in Lab • Capacity: 2.5 – 10 Gbps • Delay: 0 – 100 ms round trip • Configurable & evolvable • Topology, rate, delays, routing • Always at cutting edge • Risky research • MPLS, AQM, routing, … • Integral part of R&A networks • Transition from theory, implementation, demonstration, deployment • Transition from lab to marketplace • Global resource netlab.caltech.edu

  14. R1 R2 R3 l1 l2 l20 l3 l19 l4 R10 (b) Logical network Unique capabilities • WAN in Lab • Capacity: 2.5 – 10 Gbps • Delay: 0 – 100 ms round trip • Configurable & evolvable • Topology, rate, delays, routing • Always at cutting edge • Risky research • MPLS, AQM, routing, … • Integral part of R&A networks • Transition from theory, implementation, demonstration, deployment • Transition from lab to marketplace • Global resource netlab.caltech.edu

  15. WAN in Lab Caltech research & production networks Chicago CERN StarLight Calren2/Abilene Geneva Multi-Gbps 50-200ms delay SURFNet Experiment Amsterdam Unique capabilities • WAN in Lab • Capacity: 2.5 – 10 Gbps • Delay: 0 – 100 ms round trip • Configurable & evolvable • Topology, rate, delays, routing • Always at cutting edge • Risky research • Dynamic recovery, AQM, MPLS, routing, … • Integral part of R&A networks • Transition from theory, implementation, demonstration, deployment • Transition from lab to marketplace • Global resource • Federated Netlab (Emulab) netlab.caltech.edu

  16. Outline • Proposal summary • Basic design, equipment, costs • Unique features • Alternatives • Spectrum of tools • Emulated delay • Community resource • Demand • Management software • Why Caltech • Leverage on Abilene, HENP, CalREN, TeraGrid • Summary • Reviewer concerns • Review criteria netlab.caltech.edu

  17. ? DummyNet EmuLab ModelNet WAIL HENP Abilene CalREN WAIL PlanetLab CAIRN NLR NS SSFNet QualNet JavaSim Mathis formula Optimization Linear model Nonlinear model Stocahstic model Spectrum of tools log(cost) log(abstraction) live nk WANiLab emulation simulation math …we use them all netlab.caltech.edu

  18. live nk emulation simulation math WANiLab Critical in development e.g. Web100 Spectrum of tools netlab.caltech.edu

  19. l1 l1 S High speed electronic memory R l20 l20 S S R R S Emulated delay • Available technology inadequate • Spirent SX/14 Link Simulator: 1ms (155Mbps) – 10s (100bps) • Adequate technology too expensive • 2.5Gbps, 100ms delay: IC expert at least 2 man-years & $200K • Less realistic S R S S S netlab.caltech.edu

  20. HENP testbed CWND: 5801-5815 Ins. RTT Sylvain Ravot (Caltech/CERN) netlab.caltech.edu

  21. instantaneous RTT average RTT Example1: end-to-end delay Y: RTT (us) CWND: 5801-5815 Ins. RTT Delay between Geneva & Chicago X:Real Time (us) netlab.caltech.edu

  22. instantaneous RTT Example1: end-to-end delay Y: RTT (us) CWND: 5801-5815 Ins. RTT RTT=270ms 12450 pkts!? 1500 pkt time without buildup!? Delay between Geneva & Chicago X:Real Time (us) netlab.caltech.edu

  23. Example1: end-to-end delay Y: RTT (us) Y: RTT (us) CWND: 8700->4000 Ins. RTT RTT=980ms!? Ins. RTT Avg RTT Passive monitoring in WANiLab can help debug X:Real Time (us) X:Real Time netlab.caltech.edu

  24. inst RTT Example2: 10G Expt avg RTT ms Losses & retransmissions Real time ms Delay between Geneva & Sunnyvale netlab.caltech.edu

  25. inst RTT Retransmission without loss!? Example2: 10G Expt avg RTT ms Losses & retransmissions Real time ms netlab.caltech.edu

  26. inst RTT Retransmission without loss!? Example2: 10G Expt avg RTT ms Real time ms Passive monitoring in WANiLab can help debug netlab.caltech.edu

  27. Network debugging • Performance problems in real network • Simulation will miss • Emulation might miss • Live network hard to debug • Enable or speed up FAST development • 10GExpt: 20 people in 8 organizations for 3 months • Complete facility available only for a week • Many mysteries unresolved • WAN in Lab • Passive monitoring inside network • Active debugging possible netlab.caltech.edu

  28. GPS DAG Timestamp Header RAID Monitor Passive monitoring • No overhead on system • Can capture full info at OC48 • UofWaikato’s DAG card captures at OC48 speed • Can filter if necessary • Disk speed = 2.5Gbps*40/1500 = 66Mbps • Monitors synchronized by GPS or cheaper alternatives • Data stored for offline analysis Fiber splitter David Wei (Caltech) netlab.caltech.edu

  29. router router monitor monitor monitor monitor monitor monitor GPS DAG Timestamp Header RAID Monitor Passive monitoring Web100, FAST monitor Fiber splitter Server Server David Wei (Caltech) netlab.caltech.edu

  30. Outline • Proposal summary • Basic design, equipment, costs • Unique features • Alternatives • Spectrum of tools • Emulated delay • Community resource • Demand • Management software • Why Caltech • Leverage on Abilene, HENP, CalREN, TeraGrid • Summary • Reviewer concerns • Review criteria netlab.caltech.edu

  31. DataTAG link CWND: 5801-5815 Ins. RTT Sylvain Ravot (Caltech/CERN) netlab.caltech.edu

  32. DataTAG link • Funded by EU (CERN), USA (DoE, NSF, Caltech) • OC48 circuit StarLight-CERN • Upgrade to OC192 by August 2003 • Linux farms • StarLight: 20 CPU (P4), 20 Syskonnect • CERN: 12 CPU (P4), 12 Syskonnect • 50 users, 13 institutes, 7 countries (Feb 2003) • Heavy utilization • European hours: 100% reservation • US hours: 25% reservation, but busy netlab.caltech.edu

  33. Netbed (Emulab) • Funded by NSF with Cisco donations • Integrates simulation, emulation, live Internet • Dummynet & VLAN • Emulab Classic • University Utah: 168 PC, 5 100M Ethernet cards • Connected by 4 Cisco 6409 • Testbed backplane limited to 2Gbps • University of Kentucky: 48 PC, similar setup • Netbed: Federated Emulab • 32 nodes in 25 sites • Heavy utilization • July2002 • 65 user accounts (40 external) • 54 projects • Feb 2003 • 400 user accounts • 94 projects (10 Utah, 78 US, 6 Int’l) University of Utah netlab.caltech.edu

  34. Management software • Part of Federated Emulab • Tailor Emulab management software • Jay Lepreau’s team consult on setup • Complementary to existing federated Emulab • WANiLab • High speed large distance (Gbps WAN) • Small network (30 nodes) • Emulab • Low speed (100Mbps LAN, 10M WAN) • Large network (200+ nodes) • Instantly available to Emulab community • Web accessible anywhere any time • Virtual machine for network experimentation netlab.caltech.edu

  35. Experiment life cycle (White et al) • Experiment creation • Web based sign-up form by project lead • Approved by Emulab team • Experiment specification • ns script or Java GUI • Can download own OS, host algorithms, etc • Links emulated by Dummynet nodes with specified rate, delay, loss • Experiment realization • Map target configuration to physical resources • Reserve resources for each experiment • Oversubscription • dynamic reallocation, swap in, swap out netlab.caltech.edu

  36. Why Caltech: synergies • Caltech networking research • FAST project: the missing experimental facility (Doyle, Low) • IST Initiatives: testbed tied to rich theory program (Murray, Psaltis) • Combination of theory, implementation, experiment & deployment • Synergy in research • Caltech’s leadership role in IT for global HENP (Newman) • Vibrant research in HENP, astronomy, geological sci, biology, visualization, CACR • Early testing ground & adopter of FAST (Newman) • Availability of real data for ultrascale networks • Synergy in facility • Integration with HENP networks, Abilene, CalREN XD, TeraGrid (see Newman’s talk) • Synergy with Cisco • See Yip’s talk netlab.caltech.edu

  37. Why Caltech: experience • Hardware • Cisco’s testbed • Psaltis, Yip, Hajimiri, DeHon • Software • Netbed management software • Operation • Newman’s group • Testbed driven by networking research • IST, Theory Program, FAST, optics, scientific computing, network coding, … netlab.caltech.edu

  38. Team • Hardware • Yip’s team: Doraiswami (Cisco) • Psaltis (EE/CNS), DeHon (CS), Hajimiri (EE) • Caltech Information Tech Services, CACR • Software • Lepreau’s team • Low’s team: Almsberger (CS), Jin (CS), Wei (CS), Hu (CS) • Operation • Newman’s team: Bunn (Physics), Ravot (Physics/CERN), Suresh (CACR) • Testbed driven by networking research • Caltech IST Institute netlab.caltech.edu

  39. NewYork ABILENE UK STARLIGHT SuperJANET4 ESNET NL GENEVA SURFnet Wave Triangle GEANT CALREN It STAR-TAP GARR-B Fr Renater Global research network WAN in Lab Caltech Newman (Caltech) netlab.caltech.edu

  40. Outline • Proposal summary • Basic design, equipment, costs • Unique features • Alternatives • Spectrum of tools • Emulated delay • Community resource • Demand • Management software • Why Caltech • Leverage on Abilene, HENP, CalREN, TeraGrid • Summary • Reviewer concerns • Review criteria netlab.caltech.edu

  41. Reviewer concerns • Narrow focus on TCP/AQM • A range of IST research at Caltech (Murray, Doyle) • Spanning theory, implementation, experiment, deployment • WAN in Lab a critical component • External projects in HENP, Grid & Emulab communities • Alternatives not discussed • Use spectrum of tools at different stages • Each complementary but not replaceable • DWDM gears more realistic and cheaper • How to manage and share WAN in Lab • Part of Federated Emulab • Both demand and excellent support for global sharing • Experience in global collaboration, e.g. VRVS • How much hardware development needed • Mostly off-the-shelf (Yip) • Sample system & experience from Cisco • Local expertise: Psaltis (Optics), Yip (Cisc), Hajimiri (high speed IC), DeHon (VLSI) netlab.caltech.edu

  42. Review Criteria • Intellectual merit • Theory, implementation, experiment, deployment • Must inform and influence each other intimately • Approach validated by pilot project • Experimental facility tied to rich theory program • Broader impacts • HENP’s global collaborations a model for future corporations & society • FAST protocols enabling technology • Shared by & stimulate external research that need high speed large distance • HSTCP, Scalable TCP, TCP Westwood, AVQ, REM/PI, … • Internet as simplest complex system netlab.caltech.edu

  43. Review Criteria • Integration of research & education • Excellent projects for undergraduates and graduates • During & after development • Unique teaching platform for advanced networking, distributed systems, complex systems, optics course • Bruck, Chandy, Doyle, Hickey, Low, Psaltis • Diversity • 33% women grad students in Netlab • 50% women postdocs and grad students in Doyle’s group • Synergy among projects • Bring together 4 CISE projects (1 ITR, 1 STI, 2 pending) • Leverage for additional funding and industry collaborations netlab.caltech.edu

  44. NSF Workshop Criteria • Tested driven by research agenda • Rich and strong networking effort • “A network that can break” • Multi-user experimental facility • With a clear research focus and foreseeable impact • Federated testbed • Leverage on Netbed’s management software • Integrated monitoring & measurement facility • Fiber splitter passive monitors • Technology transfer • Strong leadership in FAST user community (Newman) netlab.caltech.edu

  45. Some potential projects • TCP: FAST, HSTCP(Floyd, ICIR), TCP Westwood(Gerla, UCLA), Scalable TCP(Kelly, Cambridge/CERN), XCP (Dina, MIT) • AQM: REM(Low), PI(Misra/Towsley), AVQ(Srikant, UIUC) • Protocol decomposition (Doyle, Low, Caltech) • Network self-management (Yemini, Columbia) • Content distribution (Bruck, Low, Caltech, Xu, Washington U) • Optical switching (Low, Psaltis, Caltech) • Network separation theory (Doyle, Low, Caltech Paganini, UCLA) • Real-time control over high performance networks (Dolye, Low, Murray, Caltech) • Simple Optics Smart Router (SOSR) (Yates, AT&T Research) • Optical protection, recovery (Yates, AT&T Research; Nirmalathas, Melbourne U) • Dynamic lightpath configuration & provisioning (Tucker, Melbourne U) • Active probing (Veitch, CUBIN) • Passive monitoring (Veitch, CUBIN) • Building & testing firewalls (Hoffman, U of Victoria) • High performance active network node (Turner, Washington) netlab.caltech.edu

More Related