Gopher giganet a next generation campus network
1 / 43

- PowerPoint PPT Presentation

  • Uploaded on

Gopher GigaNet A Next Generation Campus Network. David Farmer ([email protected]) Winter 2005 Joint Techs February 14 th 2005. Alternate Titles. How I spent my summer Without any Vacation Firewalls every where But not a Policy to implement Why MPLS Policy, Policy, Policy

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about '' - erika

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Gopher giganet a next generation campus network l.jpg

Gopher GigaNetA Next Generation Campus Network

David Farmer ([email protected])

Winter 2005 Joint Techs

February 14th 2005

Alternate titles l.jpg
Alternate Titles

  • How I spent my summer

    • Without any Vacation

  • Firewalls every where

    • But not a Policy to implement

  • Why MPLS

    • Policy, Policy, Policy

    • Or, I want to build a broken network,But still manage it

Agenda l.jpg

  • About UMN

  • The Old Network

  • Design Goals

  • Key Technologies We Picked

  • Architecture Components

  • The “Big Picture”

Twin cities campus vital statistics l.jpg
Twin Cities CampusVital Statistics

  • 897 surface acres

    • East Bank, West Bank, St. Paul

  • 251 Buildings

    • 20 story Office Towers to Garden Sheds

  • Nearly 13M Assignable ft2

    • Nearly 22M Gross ft2

  • 50,954 Student Enrollment – Fall 2004

    • Second Largest Nationally (first only 41 more)

  • Ranked 10th in total research

Twin cities campus network statistics l.jpg
Twin Cities CampusNetwork Statistics

  • More than 200 on-net Buildings

  • 1730 Wire Centers (Closets or Terminal Panels)

    • 842 With Network Electronics

  • 2774 Edge Access Switches (3750G-24TS)

  • 312 Aggregation Switches (3750G-12S)

  • 29 Core Switches (6509-NEB-A)

  • 5000 Virtual Firewall Instances

The old network l.jpg
The Old Network

  • Originally installed Sept ’97 – Dec ’99

    • Took way too long

  • 10Mb Switched Ethernet to desktop

    • Small amount of 100Mb for high-end desktops and servers

  • Typically multiple 100Mb building links

  • Partial-Mesh OC3 ATM backbone

The old network7 l.jpg
The Old Network

  • Cisco 1924 Closet Switches

    • 4 switches per 100Mb uplink

  • Cisco 2924M-XL Closet Switches

    • Used for small amounts of 100Mb for servers and desktops

    • single switch with two 100Mb uplinks

  • Cisco 5500 Core Switches

    • With RSMs for routing

    • 25 Core Nodes

  • FORE ASX-200 and ASX-1000 ATM switches for Core network

The old network midlife upgrade l.jpg
The Old Network – Midlife Upgrade

  • Installed Aug ’00

  • Added GigE Backbone

  • Cisco 5500 Core Switches

    • Upgraded to Sup3s with GigE uplinks & MLS

  • Foundry BigIron

    • Center of Star Topology GigE Backbone

Design goals l.jpg
Design Goals

  • Divorce Logical and Physical Topologies

  • Provide more than 4096 VLANs network wide

  • “Advanced” Services

  • Routed (L3) Core, Switched (L2) Aggregation and Edge

  • Network Policy – AKA Security

  • Network Intercept

  • Other Stuff

Design goals10 l.jpg
Design Goals

  • Divorce Logical and Physical Topologies

    • Administrative Topology

    • Policy Topology

      • Security or Firewalls

      • Bandwidth shaping or Usage

      • QOS

    • Functional or Workgroup Topology

Design goals11 l.jpg
Design Goals

  • Provide more than 4096 VLANs network wide

    • More than 1000 VLANs now

    • Micro segmentation for Security and other Policy could easily require 4X over the next 5 years

    • Even if we don’t exceed 4096 VLANs, the VLAN number space will be very full

Design goals12 l.jpg
Design Goals

  • “Advanced” Services

    • Native IPv4 Multicast

      • PIM Sparse Mode, MSDP, BGP for Routing

      • IGMP v3 (SSM support) for L2 switching

    • IPv6

      • Unicast for sure

      • Multicast best shot

    • Jumbo Frame

      • 9000 Clean

Design goals13 l.jpg
Design Goals

  • Routed (L3) Core, Switched (L2) Aggregation and Edge

    • How many L3 control points do you want to configure

    • Limit scope of Spanning Tree

      • If possible eliminate Spanning Tree

      • Minimally, limit it to protecting against mistakes, NOT an active part of the Network Design

Design goals14 l.jpg
Design Goals

  • Network Policy – AKA Security

    • Security is, at least partly, the network’s problem

      • Let’s design it in to the network, rather than add it in as an after thought

    • The network needs to enforce Policies

      • Only some of these are actually truly related to Security

        • Rate Shaping, COS/QOS, AAA, just to name a few

    • Firewalls with state-full inspection are necessary in some locations

    • Network Authentication (802.1x)

Design goals15 l.jpg
Design Goals

  • Network Intercept

    • Intrusion Detection and Prevention

    • Trouble shooting

    • Measurement and Analysis

    • Legal Intercept and Evidence collection

    • Sinkhole Routing

Design goals16 l.jpg
Design Goals

  • Other Stuff

    • Core Services

      • DNS

      • DHCP

      • NTP

    • Measurement

    • Localized Logging

      • Syslog

      • Netflow

Design goals17 l.jpg
Design Goals

  • Other Stuff

    • Data Centers

      • Intend to support 6 – 12 Data Centers on campus

      • Create Separate Infrastructure

        • Allows different maintenance windows

        • Provide Higher SLA/SLE

        • Provide things that can’t scale to the rest of campus

          • Server load balancing

          • Dual fiber entrances

          • Single L2 Domain

          • Redundant Routers

Design goals18 l.jpg
Design Goals

  • Other Stuff

    • Management Network

      • Console Servers

      • Remote Power Control

      • Redundant GigE network

        • Allow access to critical Core Network equipment at all times

      • Dial-up Modem on Console Server for Emergency Backup

Key technologies we picked l.jpg
Key Technologies We Picked


  • Cisco StackWise Bus on 3750s

    • Cross Stack EtherChannel provides redundancy without creating loops in the Spanning Tree topology

  • Cisco FWSM with Transparent Virtual Firewalls

    • Policy as L2 bumps on the wire

    • Let the Routers Route

How to scale l.jpg
How to Scale

  • A network with those numbers doesn’t fit in your head

    • My mind is to small to hold it all

    • How about yours

  • “consistency is the hobgoblin of little minds”

    • Emerson

  • Consistency is the answer to Scaling

Mpls vpns short tutorial l.jpg
MPLS VPNs – Short Tutorial

  • RFC 2547 defines layer 3 routed MPLS VPNs

  • Uses BGP for routing of VPNs

  • Routers create a VRF (VPN Routing & Forwarding) Instance

  • VRFs are to Routers as VLANs are to Ethernet Switches

Mpls vpns short tutorial22 l.jpg
MPLS VPNs – Short Tutorial

  • P – “Provider” Router

    • No knowledge of customer VPNs

    • Strictly routes MPLS tagged packets

  • PE – “Provider Edge” Router

    • Knowledge of customer VPNs & provider network

    • Routes packets from customer network across the provider network by adding VPN MPLS tag and tag for the remote PE

Mpls vpns short tutorial23 l.jpg
MPLS VPNs – Short Tutorial

  • CE – “Customer Edge” Router

    • No knowledge of provider network

    • Strictly routes IP packets to PE

  • Only PE routers are necessary in the MPLS VPN Architecture

    • This is important in a Campus Network

Example campus mpls vpn architecture l.jpg
Example Campus MPLS VPN Architecture

Architecture components l.jpg
Architecture Components

  • Campus Border

  • Core Network

  • Aggregation Networks

  • Edge Nodes

Campus border l.jpg
Campus Border

  • Border Routers

    • Redundant routers in diverse locations

    • Act as CE routers for all VRFs that need Internet Access

    • Cisco 6509

      • Dual SUP720-3BXL

      • Dual Power Supplies and Fans

      • All 6700 Series Interface Cards

Campus border27 l.jpg
Campus Border

  • Border Policy Enforcement

    • Layer 2 bumps on the wire

      • Cisco FWSM

      • Packeteer 9500

      • Home grown ResNet Authentication Control & Scanner (RACS)

    • Attach to or contained within Border Router

      • Packets get a little dizzy passing through Border Router L2 or L3 switching fabric several times

Core network l.jpg
Core Network

  • Backbone Nodes

    • 2 Backbone Nodes producing a Dual-Star Topology

    • Collocated with the Border Routers

    • 10Gb interconnection between Backbone Nodes.

    • 10Gb connection to each Core Node

    • Cisco 6509

Core network29 l.jpg
Core Network

  • Core Nodes

    • Located at 16 Fiber aggregation sites around campus

    • 10Gb connection to each Backbone Node

    • 2 or 3Gb to Aggregators or Edge Nodes

    • Cisco 6509-NEB-A

Core network30 l.jpg
Core Network

  • Core Nodes

    • Layer 3 routing provide for End User Subnets

      • Layer 3 MPLS VPNs provide separate Routing Domains

    • Virtual Firewalls provided per Subnets as needed

    • Root of a VLAN Domain

      • 802.1q tags have local significance only

      • VLANs connected between Core Nodes using Layer 2 MPLS VPNs as needed

Aggregation networks l.jpg
Aggregation Networks

  • Layer 2 only

  • Aggregates Edge Nodes & connects them to a Core Node

  • Cisco 3750G-12S

Aggregation networks32 l.jpg
Aggregation Networks

  • Regional Aggregator

    • 3Gb Connection to Core Node

  • Area Aggregator

    • 3Gb Connection to Regional Distribution Node

  • Building Aggregator

    • 2 or 3Gb Connection to Regional or Area Dist. Node or directly to Core Node

Edge nodes l.jpg
Edge Nodes

  • Connects users and servers to the Network

  • Connects to a Building Aggregator

    • If more than one closet in a building

    • Otherwise connects to

      • Core Node

      • Regional Aggregator

      • Area Aggregator

  • Cisco 3750G-24TS

Data center networks l.jpg
Data Center Networks

  • Data Center Core Nodes

    • Redundant Routers servicing all Data Centers on Campus

    • Collocated with the Border Routers and Backbone Nodes

    • 10Gb interconnection between Data Center Core Nodes.

    • 10Gb connection to each Backbone Node

    • 2Gb up to 10G connection to each Data Center

    • Cisco 6509-NEB-A

Data center networks36 l.jpg
Data Center Networks

  • Data Center Aggregator

    • Connected to both Data Center Core Nodes

    • Two 3750G-12S or two 3750G-16TD

    • Feeds Data Center Edge Nodes within a single Data Center

Data center networks37 l.jpg
Data Center Networks

  • Data Center Edge Nodes

    • Min Stack of two 3750G-24TS

    • Connects to Data Center Aggregator

      • Or directly to Data Center Core Node if a single stack serves the Data Center

    • Want hosts to EtherChannel to separate switches in the Stack for redundancy

Management network l.jpg
Management Network

  • Management Node

    • 3750G-24TS collocated with each Core Node

    • Routed as part of Control Plane & Management network

    • Cyclades Console server and Remote Power Control

  • Management Aggregator

    • Connects all the Mgmt Nodes

Management network39 l.jpg
Management Network

  • Measurement Server collocated with each Core Node

  • Log Server Collocated with each Core Node

  • DNS, DHCP, NTP Server Collocated with each Core Node

    • Using Anycast for DNS Redundancy

Analysis network l.jpg
Analysis Network

  • Analysis Node

    • All switches collocated in single location

    • Provides access to every Core Node for testing and Analysis

    • Provides for remote packet sniffing of any traffic on campus

    • Provides Sinkhole Drains for each Core Node

That s enough l.jpg
That’s enough

  • That’s enough rambling for now!

  • I real want to do more, but!

  • Find me and lets talk more!