Use of bgp and mpls vpns a case study
1 / 53

- PowerPoint PPT Presentation

  • Uploaded on

Use of BGP and MPLS VPNs: A Case Study. Fred P. Baker CCIE#3555. Contents. Current Network The MPLS VPN project Routing Objectives What we did How we tested. Current Network. Current Environment. Hub and spoke to 4 data centers

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about '' - zagiri

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Contents l.jpg

  • Current Network

  • The MPLS VPN project

  • Routing Objectives

  • What we did

  • How we tested

Current environment l.jpg
Current Environment

  • Hub and spoke to 4 data centers

    • Sites do not in general connect to 2 data centers due to cost and OSPF issues

  • Generally place servers by geography

    • You servers are in the data center your links are in

  • Mostly Frame Relay to ATM interworking with some private lines

    • 70 of some 350 remote sites have 2 links

  • ATM PVC dual mesh between the data centers

  • 12000 agent location network done by MCI with combination of DSL and Fractional T1

Address space l.jpg
Address Space


    • Mostly inside

    • Some BP


    • Used all over


    • Extranet


    • Public address space

    • Used mostly by extranet

    • Some legacy inside

Slide6 l.jpg

  • ATM PVCs

  • 2 10meg between each pair of data centers

  • 2 routers on the core

  • So 2 meshes

10 0 0 0 address allocation 11 for core 1 per data center l.jpg address allocation/11 for core 1 per data center

Routing protocol l.jpg
Routing Protocol

  • Single OSPF AS

  • Cisco and OS/390 based routers only

  • Firewalls now static routed

  • Peer authentication soon

Remote sites l.jpg
Remote sites

  • AT&T frame relay at the site

  • ATM into the data center

  • Some ISDN backup

  • A remote site is connected to a single data center (for now)

  • Servers and applications tend to have geographic affinity

Agent broadband l.jpg
Agent Broadband

  • 10,000 locations

  • Connected via IPSEC VPN

  • WorldCom managed routers

  • NO split tunneling

  • IPSec Transport with GRE tunnel to Dallas and Hudson

  • Agent PCs are 10.*.*.*

  • Agent access is via Allstate Internet Proxy

Internet extranet l.jpg

  • We do not use the default route

  • There are 3 data center with ISP connections

  • We code static routes to the firewalls (we don’t trust firewalls running dynamic routing protocols) and redist to OSPF

The project20 l.jpg
The project

  • We use a single data network provider

  • This is a single point of failure of that providers ATM/Frame networks

  • Add a second data provider

    • Initially to use for the dual attached sites

    • Then convert 1 of the core ATM meshes to the second provider

Layer 2 vs layer 3 provider l.jpg
Layer 2 vs Layer 3 provider

  • Frame Relay is layer 2 connectivity

    • The routers have a direct peering relationship

  • Many providers are offering Layer 3

    • Costs are the same or even less

    • MPLS VPN is the data transport

      • Many providers are using MPLS to move even layer 2 networks

    • You have a routing relationships with the provider not with yourself

      • So More complex to configure and fix

      • Not a simple OSPF network anymore

Which one we picked l.jpg
Which one we picked

  • Layer 3…

    • DR becomes free do not need to run more PVCs to a DR data center

    • The data center placement of servers assumption is changing

      • Apps are being put to 1 DC

    • Also there is more site to site traffic than we expect

    • So we can reduce traffic on the ATM core

    • And increase response time

    • Do dual homed sites first convert 1 link to L3

    • Single homed late

Mpls vpn l.jpg

VPN A/Site 2


VPN B/Site 1






VPN B/Site 2













VPN A/Site 3


VPN A/Site 1

VPN B/Site 3


Route types l.jpg
Route types

  • CE customer Edge

    • your router

    • run BGP to provider

    • Knows nothing about other customers or provider routes

  • PE provider Edge

    • Knows about all local customer VPNS

    • Has multiple routing tables

  • P providers

    • Transport only

    • No customer routes

Routing objectives l.jpg
Routing objectives

  • Support load share from the home DC

  • Remote site goes direct to non home DC over L3

  • Remote site directly to remote site

  • Reduce transit of the core

  • Support a L3 provider in the core replacing 1 ATM mesh

  • Do not use remote sites to transit traffic

Technical objectives l.jpg
Technical Objectives

  • Limit the number of bgp attributes used

  • Keep the remote site configuration simple

  • Do not inject the default route unless you must

  • How to inject the Internet routes

Don t forget the 3 rules of routing l.jpg
Don’t forget the 3 rules of routing

  • Longest subnet mask

  • Lowest distance

  • Best metric

Bgp features we used l.jpg
BGP features we used

  • As path

  • Path length filters

  • No export

  • Backdoor

  • If AS Paths are equal then router uses eBGP route

How to route l.jpg
How to route

  • Must look at the routes going BOTH ways

    • Routes to

    • Routes from

  • The routes you advertise drags traffic to you

  • The routes you take in is how you route back

  • We load share by having each router use a different path, then send equal cost into IGP

Result l.jpg

  • Use MPLS VPN based L3 provider

  • Remote sites 2nd link to L3

  • Each data center connects to L3

  • Will not use L3 to route between DCs due to QoS concerns

Routing l.jpg

  • Use BGP at remote sites

    • Can use OSPF with SOME providers but not all

    • BGP works much better

    • Each site is 1 AS

  • EACH data center is 1 AS

    • This allows us to put an L3 provider in later

    • BGP routes BETWEEN ASes

  • Address ASes from private space

  • This is ok because provider is a VPN

Route injection to from bgp l.jpg
Route injection to/from BGP

  • Allstate Data Center

    • Explicit network statements to BGP

    • Redist BGP to OSPF

  • Remote site routes

    • Redist from OSPF

      • Decided that using network statements to complex

    • BGP routers send just default route to any switches

      • We will accept the extra LAN transit

  • Internet routes

    • Redist static

Internet routes l.jpg
Internet routes

  • There will be non BGP L3 switches between Inet and allstate core

  • Redist static into OSPF already

  • So just redist into BGP also

  • Put internet router in same AS as datacenter (have to as no direct path)

  • Use sync

  • Send to L3 provider and to sites over L3

Bgp to l3 provider and then remote sites l.jpg
BGP to L3 provider (and then remote sites

  • Data center side

    • Send data center /11s

    • Send internet routes

    • Take routes from L3 provider

    • Do not forward other eBGP learned routes

  • Remote site side

    • Send all local routes

    • do not forward other learned eBGP routes

    • Remember the no export to kill transit

    • Receive all routes

      • Want to take L3 when I can

Dc to remote site fr l.jpg
DC to Remote site FR

  • Send all bgp derived routes

  • Do as prepend of the data center AS

  • This makes AS path =2 for DC on FR and L3 paths

  • This makes AS Path=3 for DC to DC via ATM core so site to remote DC traffic over L3

Remote site to dc on fr l.jpg
Remote site to DC on FR

  • Do as prepend of 1 AS at remote end

  • Need this so FR and L3 paths have AS Path=2 so we load share

  • Filter routes with AS Path >1

    • I only want to send the local site routes up the FR link

    • Do not want DC to send transit traffic to site

Ibgp in the remote site l.jpg
IBGP in the remote site

  • Set next hop self

  • Routers must have a shared Enet

  • No redist of BGP to OSPF

  • So cant use sync so cant transit a L3 switch

  • Do not forward routes I learn via FR

  • Do not want a transit from L3 up the FR link

  • Do not want a transit to L3 from FR link

  • Set no export attribute on routes from DC over the FR link

  • This prevents site from passing them to L3

  • Cannot AS path filter on IBGP because I want to pass the DC route via iBGP

    • Why I use no export

Dc to dc l.jpg
DC to DC

  • Each site learns over ATM network with AS Path = 1

  • Cannot route over L3 provider

Remote site to non home dc l.jpg
Remote site to non home dc

  • Non home DC sent via L3 AS Path = 2

  • Home data sends via FR AS Path = 3 due to prepend

    • Use if L3 down

Non home dc to remote site l.jpg
non home dc to remote site

  • Non Home DC learns remote site routes from L3

  • Home data center sends only the /11 summary

  • so longest match says L3

Home dc to remote site l.jpg
home dc to remote site

  • Load share

  • Routes from L3 have AS Path = 2

  • Routes from FR have AS Path = 2 due to prepend

  • So each router uses eBGP route

Remote site to home dc l.jpg
remote site to home dc

  • Don’t care as much about load share

  • Routes from L3 have AS Path = 2

  • Routes from FR have AS Path = 2 due to prepend

  • So each router uses eBGP route

Remote site to remote site l.jpg
remote site to remote site

  • Use L3 network

  • Learn site specific routes directly from site

  • Learn /11 summaries from DCs

Agent routes l.jpg
Agent routes

  • Only dual DC connected things that don’t use BGP

  • Many routes summarized as /19s

  • I get these from MCI as OSPF externals

  • Have not decided how to inject them

  • They go to two data centers for redundancy

  • So I need to send them via BGP

  • So a router will get an OSPF external from the local MCI connection and the other data center via BGP

  • eBGP < OSPF so BOOM

  • Use backdoor on core routers to set distance on the agent routes to > than OSPF

  • So if local MCI connection up use it, else transit core

Local testing l.jpg
Local Testing

  • Use 7 routers

  • 1 remote site OSPF route not shown

  • Paths

    • iBGP at remote

    • L3

    • FR to home DC

    • Inter DC

Slide49 l.jpg

  • Cisco Proof Of Concept

  • In Raleigh and San Jose

  • Lab use is free (if you are big enough)

  • Send in specific test plan

  • Your SE goes in a week ahead of time

  • Lab is all setup when you arrive

Testing50 l.jpg

  • Test migrations

  • Test routing

    • based on our policies

    • failovers

  • Measure convergence

  • Test a migration of a core ATM mesh to L3

  • Get some data and experience on the MPLS side

  • Try multicast over MPLS/VPN

Cpoc learnings l.jpg
CPOC Learnings

  • Inject all links both ATM core and L3 into BGP as they will source pings

  • Turn sync off due to code defect

  • You must explicitly code send community in iBGP

  • If you reference a non-existent as-path statement NO ROUTES

  • OSPF LSAs stay in the data base up to 90 minutes due to timer jitter

    • This is a migration issue

  • Do lots of clear routes/clear ip bgp in the migration

  • Need to change the BGP timers as default convergence is 3 minutes

  • iBGP only sends the best route

Going forward l.jpg
Going forward

  • Already run BGP to some remote sites

  • Migrate the core to bgp first

    • Do a dress rehearsal

    • Will be a big scary change so plan well

  • Examine tools

    • May not be able to assume we will get traps

    • May have to watch the BGP tables for changes

  • Get a test connection in place