Surviving Large Scale Internet Outages - PowerPoint PPT Presentation

Surviving large scale internet outages l.jpg
Download
1 / 93

Surviving Large Scale Internet Outages Dr. Krishna Kant Intel Research Acknowledgements: Work supported by National Science Foundation Collaborative work with A. Sahoo & P. Mohapatra Outline Overview Routing and Name resolution infrastructures Some large scale failures

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Surviving Large Scale Internet Outages

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Surviving large scale internet outages l.jpg

Surviving Large Scale Internet Outages

Dr. Krishna Kant

Intel Research

Acknowledgements:

Work supported by National Science Foundation

Collaborative work with A. Sahoo & P. Mohapatra


Outline l.jpg

Outline

  • Overview

    • Routing and Name resolution infrastructures

    • Some large scale failures

  • Routing Vulnerabilities

    • Routing algorithms & their properties

    • Improving inter-domain routing

  • Dealing with Name Resolution Failures

    • Name resolution preliminaries

    • DNS vulnerabilities & Solutions

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


The problem l.jpg

The Problem

  • Internet has two critical elements

    • Routing (Inter & intra domain)

    • Name resolution

  • How robust are they against large scale failures/attacks?

  • How do we improve them?

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Internet routing l.jpg

inter-domain router

intra-domain router

Internet Routing

  • Not a homogeneous network

    • A network of autonomous systems (AS)

    • Large variation in AS sizes – typical heavy tail.

  • Inter-AS routing

    • Border Gateway Protocol (BGP)

    • Complex configuration parameters

    • Flexible but serious stability, recoverability and configurability issues

  • Intra-AS routing

    • Usually easier to manage

      • Central control, smaller network, …

    • But, can suffer from similar problems

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Internet name resolution l.jpg

Internet Name Resolution

  • Domain Name Server (DNS)

    • Translates names to IP addresses.

    • Critical for all networking services

    • Hierarchical structure

    • Caching of data in proxy servers & resolvers

  • DNS Vulnerabilities

    • Complex dependencies & easy to “poison”

    • Can lead to large scale “failures”

      • Inability of access or diversion to malicious sites.

ftp acme.com

application

Resolver

acme.com

10.7.196.31

DNS proxy

server

Auth. DNS

server

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Large scale failures l.jpg

Large Scale Failures

  • Characteristics

    • Large service impact.

    • Usually non-uniformly distributed, e.g., an affected geographical area, hijacked .com domain, etc.

  • Why study large scale failures?

    • Several moderate sized incidents already.

      • Larger failures will happen

    • Can cause other undesirable impacts

      • Secondary failures due to large recovery traffic,

      • Substantial imbalance in load, …

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Routing failures l.jpg

Routing Failures

  • Physical Damage

    • Earthquake, hurricane, high BW cable cuts, …

  • SW bugs & configuration errors

    • Incorrect input or output filtering rules

    • Aggregation of large un-owned IP blocks

    • Incompatible policies among AS’es

  • Network wide congestion (DoS attack)

  • Malicious route advertisements via worms

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Name resolution failures l.jpg

Name Resolution Failures

  • Compromising name resolution

    • Poisoning (altering/insertion) of address records

      • Doesn’t even require compromising the server

      • Extensive caching  More points of entry

    • Substitution of rogue DNS server

    • Security holes due to configuration errors

  • Potential large scale effects

    • Poisoning at higher levels  Large scale disruption

      • Example: March 2005 .com attack

    • Redirection to malicious sites to collect sensitive info

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Some significant failure events l.jpg

Some Significant Failure Events

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Taiwan earthquake dec 2006 l.jpg

Taiwan Earthquake Dec 2006

  • Major outage in SE Asia, 60% drop in traffic

  • Issues:

    • Global traffic passes through a small number of seismically active choke points.

      • Luzon strait, Malacca strait, South coast of Japan

    • Satellite & overland cables  Inadequate backup capacity

    • Several countries depend on 1-2 landing pts

  • Outlook: Potential repeat performance

    • Economics makes change unlikely.

    • May be exploited by pirates + terrorists

  • Reference: http://master.apan.net/meetings/xian2007/publication/051_Kitamura.pdf

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Hurricane katrina aug 2005 l.jpg

Hurricane Katrina (Aug 2005)

  • Major local outages. No major regional cable routes through the worst affected areas.

  • Outages persisted for weeks & months. Notable after-effects in FL (significant outages 4 days later!)

  • Reference:

    http://www.renesys.com/tech/presentations/pdf/Renesys-Katrina-Report-9sep2005.pdf

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Ny power outage aug 2003 l.jpg

NY Power Outage (Aug 2003)

  • No of concurrent network outages vs. time

    • Large ASes suffered less than smaller ones.

  • Many ASes all routers down for >4 hours.

  • Very similar power outage in Italy, sept 2003.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Slammer worm jan 2003 l.jpg

Slammer Worm (Jan 2003)

  • Worm started w/ buffer overflow of MS SQL.

    • Very rapid replication, huge congestion buildup in 10 mins

    • Korea falls out, 5/13 DNS root servers fail, failed ATMs, …

    • High BGP activity to find working routes.

  • Reference: http://www.cs.ucsd.edu/ savage/papers/IEEESP03.pdf

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Dns attack jan 2006 l.jpg

DNS Attack (Jan 2006)

  • Attack Type

    • Authoritative TLD DNS servers attacked using ~100 zombie clients & 51K recursive servers.

    • 55 Byte zombie query  4.2KB response.

    • Responses directed to target name server (w/ spoofed IP address).

  • Impact

    • Failures in networks in the path including transit providers to authoritative TLD DNS servers

  • Graph

    • #Unanswered queries (Y-axis) vs. Time (X-axis)

    • Red: failure, yellow: slow

Reference: http://www.oecd.org/dataoecd/34/40/38653402.pdf

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Infrastructure induced failures l.jpg

Infrastructure Induced Failures

  • En-masse use of backup routes by 4000 Cisco routers in May 2007 (Japan)

    • Routing table rewrites  7 hr downtime in NE Japan

    • Ref: http://www.networkworld.com/news/2007/051607-cisco-routers-major-outage-japan.html

  • Akamai CDN failure – June 2004

    • Probably widespread failures in Akamai’s DNS.

    • Ref: http://www.landfield.com/isn/mail-archive/2004/Jun/0064.html

  • Worldcom router mis-configuration – Oct 2002

    • Misconfigured eBGP router flooded internal routers with routes.

    • Ref: http://www.isoc-chicago.org/internetoutage.pdf

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Routing infrastructure l.jpg

Routing Infrastructure

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Routing basics l.jpg

Routing Basics

  • Distance vector based (DV)

    • RIP (Routing Information Protocol).

    • IGRP (Interior gateway routing Protocol).

  • Link State Based (LS)

    • OSPF (Open shortest path first)

    • IS-IS (Intermediate system to IS)

  • Path Vector Based (PV)

    • BGP (Border Gateway Protocol)

    • Intra-domain (iBGP) & inter-domain (eBGP) versions.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Distance vector ds protocols l.jpg

A-D=3

E-D=2

F-D=1

E-D=2

C-D=3

Distance Vector (DS) Protocols

  • Build RT using successive path advertisements.

  • May use stale info used to handle failures

    • “count to infinity” problem; Several versions to fix this.

  • Difficult to use policies

Routing Table for A

B

D

E

A

F

C

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Link state ls protocols l.jpg

Link State (LS) Protocols

  • Each node keeps complete adjacency/cost matrix & computes shortest paths locally

    • Any failure propagated via flooding

    • Expensive in a large network

  • Loop-free & can use policies easily.

3

B

D

1

4

6

A

2

5

E

C

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Path vector protocols l.jpg

B/E/F/D:3

E/F/D:2

F/D:1

B

D

E/F/D:2

E

A

Link_cost=2

F

C

C/E/F/D:4

Path Vector Protocols

  • Each node initialized w/ a set of paths for each destination

  • Active paths updated much like in DV

    • Explicitly withdraw failed paths (& advertise next best)

  • Filtering on incoming/outgoing paths, path selection policies

  • Paths A to D:

    • Via B: cost 3

    • Via C: cost 4

  • Entire path not stored (only cost, next hop)

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Intra domain routing under failures l.jpg

Intra-domain Routing under Failures

  • Routing algorithms

    • Link state (OSPF)

      • Flooding can handle failures quickly.

    • Path vector (iBGP)

      • iBGP routers are fully meshed in small networks (routing not much of an issue)

      • In large network, route reflectors may be used for scalability

  • Can recover rather quickly

    • Single domain of control

      • High visibility, common management network, etc.

      • Easy to configure consistent values at all routers

    • iBGP with route reflection shown to suffer from oscillations, but can be remedied.

  • Reference: A. Rawat & M.A. Shayman, “Preventing persistent oscillations and loops in IBGP configuration with route reflection”, Computer Networks, Vol 50, No 18, Dec 2006, pp 3642-3665

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Inter domain routing l.jpg

R5

R

R4

R1

R3

R2

AS1

I-BGP

IGP

AS3

border router

internal router

A

AS2

E-BGP

announce B

B

Inter-domain Routing

  • BGP: Default inter-AS protocol (RFC 1771)

  • Path vector protocol, runs on TCP

    • Scalable, “rich” policy settings

  • But prone to long “convergence delays”

    • High packet loss & delay during convergence

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Inter domain routing bgp specifics and vulnerabilities l.jpg

Inter-domain routingBGP specifics and vulnerabilities

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Bgp routing table l.jpg

BGP Routing Table

  • Prefix: origin address for dest & mask (eg.,207.8.128.0/17)

  • Next hop: Neighbor that announced the route

  • One active route, others kept as backup

    • Only active route can be advertised

  • Route “attributes” -- may be conveyed outside

    • ASpath: Used for loop avoidance.

    • MED (multi-exit discriminator); preferred incoming path

    • Local pref: Used for local path selection

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Bgp messages l.jpg

Withdrawn route lengths (2 octets)

Withdrawn routes (variable length)

Length of all path attributes (2 octets)

Advertised path attributes (variable length)

Reachability Information (variable length)

BGP Messages

  • Message Types

    • Open (establish TCP conn), notification, update, keepalive

  • Update

    • Withdraw zero or more old routes

    • Optionally advertise exactly one new route.

    • May need to also advertise sub-prefix

      • E.g., 207.8.240.0/24 which is contained in 207.8.128.0/17

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Routing process l.jpg

Fwd/not-fwd

Set MEDs

Accept, deny,

set preferences

BGP decision

process

Route pkts

Routes

received

from

peers

IP Routing

Table

BGP routing

Table

Routes

sent to

peers

Input

Policy

Engine

Output

policy

engine

Routing Process

  • Input policy engine

    • Filter routes by path attributes, prefix, etc.

  • Output policy engine

    • Manipulate attributes, e.g. Local pref., MED, etc.

  • Multiple points for possible configuration errors & mismatch between AS’es

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Bgp recovery l.jpg

BGP Recovery

  • BGP Convergence Delay

    • Time for all routes to stabilize following an event

    • Four durations of interest

      • Tup, Tshort, Tlong, Tdown

  • Min. Route Advertisement Interval (MRAI)

    • Applies only to adv., not withdrawals

    • Intended – per destination, Implemented – per peer

    • Damps out oscillations

Convergence Delay

MRAI

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Impact of bgp recovery l.jpg

Impact of BGP Recovery

  • Long Recovery Times

    • >3 min. for 30% of isolated failures

    • > 15 min. for 10% of cases

    • Longer for larger failures

  • Consequences

    • Connection attempts over invalid routes fail.

    • Big increase in pkt loss (30X) and delay (4X)

    • Compromised QoS

Graphs taken from ref #2, Labovitz, et.al.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Bgp illustration 1 l.jpg

H

E

F

G

I

D

2

3

A

B

10

C

H

E

F

G

I

D

2

3

A

B

10

C

BGP Illustration (1)

  • Best path PSD=(N, cost) [X]

    • S,D: Source & destination nodes

    • N: Next hop

    • X: Actual path (for illustration only)

  • Sample starting paths to C

    • PBC=(D,3) [BDAC], PDC=(A,2) [DAC], etc.

    • Paths shown using arrows (all share seg AC)

  • Failure of A

    • BGP does not attempt to diagnose problem or broadcast failure events.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Bgp convergence delay analysis l.jpg

BGP Convergence Delay Analysis

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Known analytic results l.jpg

Known Analytic Results

  • Lots of work for isolated failures, none on large scale failures.

  • Labovitz [1]: Convergence delay bound for full mesh networks

    • O(n3) for average case, O(n!) for worst case.

  • Labovitz [2], Obradovic [3], Pei[8]:

    • Convergence delay  Length of longest path involved

    • Applies only for unit cost hops

  • Griffin and Premore [4]:

    • V shaped curve of convergence delay wrt MRAI.

    • #Messages wrt MRAI decreases at a decreasing rate.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Evaluation of ls failures l.jpg

Evaluation of LS Failures

  • Evaluation methods

    • Primarily simulation. Analysis is intractable

  • BGP Simulation Tools

    • Several available, but simulation expense is the key!

    • SSFNET – scalable, but max 240 nodes on 32-bit machine

  • SSFNet default parameter settings

    • MRAI jittered by 25 % to avoid synchronization

    • OSPFv2 used as the intra-domain protocol

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Topology modeling l.jpg

Topology Modeling

  • Topology Generation: BRITE

    • Enhanced to generate arbitrary degree distributions

      • Heavy tailed based on actual measurements.

      • Approx: 70% low & 30% high degree nodes.

    • Mostly used 1 router/AS  Easier to see trends.

  • Failure topology: Geographical placement

    • Emulated by placing all AS routers and ASes on a 1000x1000 grid

    • The “area” of an AS  No. of routers in AS

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Convergence delay vs failure extent l.jpg

Convergence Delay vs. Failure Extent

  • Initial rapid increase & then flattens out.

  • Delays & increase rate both go up with network size

     Large failures can pose a problem!

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Delay msg traffic vs mrai l.jpg

Delay & Msg Traffic vs. MRAI

  • Small networks in simulation 

    • Optimal MRAI for isolated failures small (0.375 s).

  • Main observations

    • Larger failure  Larger MRAI more effective

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Convergence delay vs mrai l.jpg

Convergence Delay vs. MRAI

  • A V-shaped curve, as expected

  • Curve flattens out as failure extent increases

  • Optimal MRAI shifts to right with failure extent.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Impact of as distance l.jpg

Impact of AS “Distance”

  • ASes more likely to be connected to other “nearby” ASes.

  • b indicates the preference for shorter distances (smaller b higher preference)

  • Lower convergence delay for lower b.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Improving bgp convergence delay l.jpg

Improving BGP Convergence Delay

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Reducing convergence delays l.jpg

Reducing Convergence Delays

  • Many schemes – mostly evaluated for isolated failures

  • Some popular schemes

    • Ghost Flushing

    • Consistency Assertions

    • Root Cause Notification

  • Our work (Large scale failure focused)

    • Dynamic MRAI

    • Batching

    • Speculative Invalidation

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Ghost flushing l.jpg

H

E

F

G

I

D

2

3

A

B

10

C

Ghost Flushing

  • Bremler-Barr, Afek, Schwarz: Infocom 2003

  • An adv. implicitly replaces old path

    • GF withdraws old path immediately.

  • Pros

    • Withdrawals will cascade thru ntwk

    • More likely to install new working routes

  • Cons

    • Substantial addl load on routers

    • Flushing takes away a working route!

  • Install BC 

  • Routes at D, F, I via B will start working

  • Flushing will take them away.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Consistency assertion l.jpg

Consistency Assertion

  • Pei, Zhao, et.al., Infocom 2002

    • If S has two paths S:N1xD & S:N2yN1xD, & first path is withdrawn, then second path is not used (considered infeasible).

  • Pros

    • Avoids trying out paths that are unlikely to be working.

  • Cons

    • Consistency Checking can be expensive

S

N2

N1

y

x

D

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Root cause notification l.jpg

Root Cause Notification

  • Pei, Azuma, Massy, Zhang: Computer Networks, 2004

  • Modify BGP messages to carry root cause (e.g., node/link failure).

  • Pros

    • Avoid paths w/ failed nodes/links  substantial reduction in conv. delay.

  • Cons

    • Change to BGP protocol. Unlikely to be adopted.

    • Applicability to large scale failures unclear (diagnosis difficult)

H

E

F

G

I

D

2

3

A

B

10

C

  • D, E, G diagnose if A or link to A has failed.

  • Propagate this info to neighbors

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Large scale failures our approach l.jpg

Large Scale FailuresOur Approach

  • What we can’t or wouldn’t do?

    • No coordination between ASes

      • Business issues, security issues, very hard to do, …

    • No change to wire protocol (i.e., no new msg type).

    • No substantial router overhead

    • Solution applicable to both isolated & LS failures.

  • What we can do?

    • Change MRAI based on network and/or load parms

      • e.g., degree dependent, backlog dependent, …

    • Process messages (& generate updates) differently

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Key idea dynamic mrai l.jpg

Key Idea: Dynamic MRAI

  • Increase MRAI when the router is heavily loaded

    • Reduces load & #of route changes.

  • Relationship to large scale failure

    • Larger failure size  Greater router loading  Larger MRAI more appropriate.

    • Router load directed MRAI caters to all failure sizes!

  • Implementation:

    • Queue length threshold based MRAI adjustment.

Decrease th1

Decrease th2

Increase th1

Increase th2

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Dynamic mrai effect on delay l.jpg

Dynamic MRAI: Effect on Delay

  • Change wrt fixed MRAI=0.375 secs.

  • Improves convergence delay as compared to fixed values.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Key idea message batching l.jpg

Key Idea: Message Batching

  • BGP default: FIFO message processing 

    • Unnecessary processing, if

      • A later update (already in queue) changes route to dest.

      • Timer expiry before a later msg is processed.

  • Relationship to large scale failure

    • Significant batching (and hence batching advantage) likely for large scale failures only.

  • Algorithm

    • A separate logical queue/dest. – allows processing of all updates to dest as a batch.

    • >1 update from same neighbor  Delete older ones.

B

B

C

A

A

A

A

B

A

A

B

C

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Batching effect on delay l.jpg

Batching: Effect on Delay

  • Behavior similar to dynamic MRAI w/o actually making it dynamic

  • Combination w/ dynamic MRAI works somewhat better.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Key idea speculative invalidation l.jpg

Key Idea: Speculative Invalidation

  • Large scale failure

    • A lot of route withdrawals for the failed AS, say X

    • #withdrawn paths w/ AS X e AS_path > thres  Invalidate all paths containing X

  • Implementation Issues

    • Going through the routes for invalidation is inefficient

      • Use output route filters at each node

    • Threshold estimation  Computed (see paper)

    • Reverting routes to valid state  time-slot based

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Effect of invalidation l.jpg

Effect of Invalidation

  • Avoids exploring unnecessary paths

    • Reduces conv. delay significantly, but …

    • May affect connectivity adversely.

  • Implement only at nodes with degree 4 or higher

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Comparison of various schemes l.jpg

Comparison of Various Schemes

  • CA is the best scheme throughout!

  • GF is rather poor

  • Batching & dynamic MRAI do pretty well considering their simplicity

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Routing recovery metrics l.jpg

Routing Recovery Metrics

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


What s the right performance metric l.jpg

What’s the right performance metric?

  • Convergence delay

    • Network centric, not user centric

    • Instability in infrequently used routes is almost irrelevant

  • User Centric Metrics

    • Packet loss & packet delays

  • Convergence delay does not correlate well with user centric metrics

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


User centric metrics l.jpg

User Centric Metrics

  • Frac of pkts lost or frac increase in pkt delay

    • Pkt delay needs E2E monitoring  Impractical

  • Metric computation

    • Single number: Overall avg over routes & time

    • Distribution wrt routes, time dependent rate, etc.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Comparison between schemes l.jpg

Comparison between Schemes

  • Comparisons

    • Consistency assertion (CA), Ghost Flushing (GF), Speculative Invalidation (SI)

  • All 3 schemes reduce conv delay substantially, but only CA can really reduce the pkt losses!

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


How schemes affect routes l.jpg

How Schemes affect routes

  • Cumulative time for which there is no valid path

    • T_noroute: Time for which there is no route at all

    • T_allinval: Time for which all neighbors advertise an invalid route

    • T_BGPinval: Time for which BGP chooses an invalid route (even though some neighbor has a valid route).

  • GF increases T_noroute the most, CA reduces T_allinval the most

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Changes to reduce pkt losses l.jpg

Changes to Reduce pkt Losses

  • GF: Difficult to reduce T_noroute. Not attempted.

  • CA: Use “best route” even if all of them are “infeasible”, but don’t advertise infeasible routes.

    • Improves substantially

  • SI: Mark the route invalid probabilistically depending on fail count (instead of deterministically)

    • Improves substantially

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Routing misconfiguration l.jpg

Routing Misconfiguration

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Configuration checking l.jpg

Route Validity

Path Visibility

Configuration Checking

  • Fault checking by a tool called RCC

    • N. Feamster & H. Balakrishnan, Detecting BGP faults with static analysis, NSDI ’05

    • Config faults in every AS, most related to lack of coordination

    • Some faults could have global consequences

  • Consistency checking required for each change in policies – hard!

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Conclusions open issues l.jpg

Conclusions & Open Issues

  • Inter-domain routing does not perform very well for large scale failures.

    • Considered several schemes for improvement. Room for further work.

  • Convergence delay is not the right metric

    • Defined pkt loss related metric & a simple scheme to improve it.

  • Open Issues for large scale failures

    • Analytic Modeling of convergence properties.

    • What aspects affect pkt losses & can we model them?

    • Account for policies & AS relationships.

    • Effective & efficient methods for misconfiguration detection.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Domain name system basics vulnerabilities l.jpg

Domain Name SystemBasics & Vulnerabilities

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Dns infrastructure l.jpg

DNS Infrastructure

root

Browser

FTP

E-Mail

au

nz

sg

Application

and O/S

Resolver

edu

gov

DNS Proxy

Server

ips

gb

sa

Cache

Authoritative DNS Server

  • Three levels of name resolution:

    • Client side (OS provided resolver)

    • DNS proxy server (organization level)

    • Authoritative server (serves a “zone”)

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Dns basics l.jpg

DNS Basics

  • DNS manages “zones”

    • A set of names that are under the same authority

      • E.g., ftp.acme.com and www.acme.com under acme.com

  • DNS is a "lookup service"

    • Simple queries  No search or 'best fit' answers

    • Limited data expansion capability

  • TTL (Time To Live): The time an RRSet can be cached/reused by a non- authoritative server

  • Best matching records

  • Iterative vs. recursive resolution

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Ttl values used by dns proxies l.jpg

TTL Values Used by DNS Proxies

  • 2.7 Million names on dmoz.org

    • Some values, e.g. 1 hr, 1 day, 2 days dominate

    • Some extremely small values

  • Large TTL

    • Low overhead, more likely to be stale/incorrect

  • Small TTL

    • High overhead, but up to date.

CDF of TTLs

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Dns resource record rr l.jpg

Domain name

Type

Class

TTL

RL

RDATA

Variable 2 2 42Variable

DNS Resource Record (RR)

  • Domain name:

    • (length, name) pairs, eg., cisco.com  05cisco03com00

  • Record Types

    • DNS Internal types

      • Authority: NS, SOA; DNSSEC: DS, DNSKEY, RRSIG, …

      • Many others: TSIG, TKEY, CNAME, DNAME, …

    • Terminal RR:

      • Address records: A, AAAA

      • Informational: TXT, HINFO, KEY, … (data carried to apps)

    • Non Terminal RR:

      • MX, SRV, PTR, … w/ domain names resulting in further queries.

  • Other fields

    • RL: Record length, RDATA: IP address, referral, …

    • TTL: Time To Live in a cache

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Dns attacks l.jpg

DNS Attacks

  • Inject incorrect RR into DNS proxy (poisoning)

    • Capture query & send fake response before the real response

    • Randomly send fake responses

  • Query interception relatively easy …

    • UDP based  Don’t need any context!

    • DNS query uses 16-bit “trans-id” to connect query w/ response.

      • Randomized in newer implementations, but attacker can generate a large number of replies.

    • Response can include additional RR’s

  • Intercept updates to authoritative server

    • Technically not poisoning, but a problem

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Poisoning consequences l.jpg

root

au

sg

nz

gov

edu

ips

gb

sa

gb

Poisoning Consequences

  • Can be exploited in many ways:

    • Disallow name resolution

    • Direct all traffic to small set of servers

      • DDOS attack!

    • Direct to a malicious server to collect info or drop malware

  • Scale of attack depends on the level in the hierarchy!

    • Poison propagates downwards

    • Set large TTL to avoid expiry

    • Actual scenario in Mar ’05 (.com entry poisoned)

Proxy Cache

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Kaminsky dns attack l.jpg

Kaminsky DNS Attack

  • Attack target: www.abc.com

    • Poisoning of auth server response possible on TTL expiry in the proxy  hard

  • Generate queries for fake x.abc.com (x=1, 2, 3, …)

    • Supply fake responses with guessed TXID (ahead of auth server response)

    • In fake response, delegate www.abc.com to a server of attackers choosing

    • Source port guessing is necessary for this attack

  • Repeat until something works, say 83.abc.com

    • The proxy-server now has a valid DNS record for 83.abc.com.

    • Queries to www.abc.com are now directed to attacker’s site

      Reference: http://www.doxpara.com/DMK_BO2K8.ppt

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Dnschanger attack l.jpg

DHCP-discover

Good

client

Bad DNS

Server

DHCP-

offers

Real DHCP

Server

Bad

client

DNSchanger Attack

  • Attack via DHCP

    • Doesn't exploit any security vulnerability

    • Depends on “ndisprot.sys” driver installed on infected box that generates fake DHCP server offers.

  • Attack Scenario

    • Infected client X: connects to some network N

    • A benign client Y: Requests IP address for N

    • X responds w/ DHCP-offer that supplies rogue DNS server address.

    • Y’s DNS requests can now be translated to arbitrary IP addresses.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Dns robustness l.jpg

DNS Robustness

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Making dns robust l.jpg

Making DNS Robust

  • TSIG (symmetric key crypto)

    • Intended for secure masterslave proxy comm.

    • Issues: Not general, Scalability

  • DNSSEC

    • Stops cache poisoning, but issues of overhead, infrastructure change, key mgmt, etc.

    • Based on PKI, a symmetric key version also exists.

  • Cooperative Lookup

    • Direct requests to responsive clients (CoDNS)

    • Distributed hash table (DHT) structure for DNS (CoDoNS)

    • Cooperative checking between clients (DoX)

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Pk dnssec l.jpg

gov priv.

key

au priv.

key

Root priv.

key

gb

Cert.

gov

Cert.

au

Cert.

PK-DNSSEC

  • Auth. chain starts from root

    • Parent signs child certificates (avoids “lying” about public key)

    • Encrypted exchange also supplies signed public keys

  • F: public key, f: private key

Root

Cert.

root

nz

sg

au

Fgov(query)

gov

edu

DNS

proxy

fgov(resp, Fgb)

Fgb(query)

ips

sa

gb

gb

fgb(resp)

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Codons l.jpg

CoDoNS

  • Organize DNS using DHT (distributed hash table).

    • Enhances availability via distribution and replication

  • Explicit version control to keep all copies current

  • Issues

    • DHT issues (equal capacity nodes)

    • Explicit version control unscalable.

    • Not directed towards poisoning control (but DNSSEC can be used)

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Domain name cross referencing dox l.jpg

root

root

root

root

sg

sg

sg

sg

au

au

au

au

nz

nz

nz

nz

gov

gov

gov

gov

edu

edu

edu

edu

ips

ips

ips

ips

sa

sa

sa

sa

gb

gb

gb

gb

gb

gb

gb

gb

Domain Name Cross-referencing (DoX)

  • Client peer groups

    • Diversity & common interest based

    • Peers agree to cooperate on verification of popular records.

  • Mutual verification

    • Assumes that authoritative server is not poisoned.

Peer2

Peer1

Verify

Peer3

Peer4

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Choosing peers l.jpg

Choosing Peers

  • Give & get

    • Give: A peer must participate in verification even if it is not interested in the address  Overhead

    • Get: Immediate poison detection, high data currency.

  • Selection of peers

    • Topic channel w/ subscription by peers

      • E.g. names under a Google/Yahoo directory

    • Community channel, e.g., peers within the same org

  • Minimizing overhead

    • Verify only popular (perhaps most vulnerable) names

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Dox verification l.jpg

DoX Verification

  • Uses verification cache for efficiency

  • Verification

    • DNS copy (Rd) = verified copy (Rv)  Stop

      Else send (Ro,Rn) = (Rv,Rd) to all other peers

    • At least m peers agree  Stop, else obtain authoritative copy Ra & if Ra != Rd, poison detected.

  • Agreement procedure

    • Involves local copy Rv& remotely received (Ro,Rn)

    • If Rv=Rn  agree, else peer gets auth. copy Ra

    • Several cases, e.g., if Rv=Ro, Ra=Rn  agree

      • Verified copy was obsolete, got correct one now  Forced removal of obsolete copy

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Handling multiple ips per name l.jpg

Handling Multiple IPs per name

  • DNS directed load distribution

    • Easily handled with set comparison

  • Multiple Views

    • Used to differentiate inside/outside clients

    • All peers should belong to same view (statically or trial & error).

  • Content Distribution Networks (CDNs)

    • Same name translates to different IP addresses in different regions

    • Need a flowset based IP address comparison

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Results normal dns l.jpg

Results – Normal DNS

  • Poison spreads in the cache

  • More queries are affected

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Results dox l.jpg

Results – DoX

  • Poison removed immediately

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Dox vs dnssec l.jpg

DoX vs. DNSSEC

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Dns delegation l.jpg

DNS Delegation

  • Delegation

    • Allows portions of name space to be managed separately

    • Often used for geographical diversity & redundancy

  • How it works

    • sales.dell.com delegated to domain pc1.com

    • Client needs to follow DNS tree to resolve pc1.com (unless there is a glue record)

root

.com

dell.com

pc2.com

pc1.com

support.dell.com

sales.dell.com

parts.sales.dell.com

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Delegation related failures l.jpg

Delegation Related Failures

  • Delegation Characteristics

    • Any domain server may delegate its subspace further

    • Delegation done manually w/o any global visibility.

  • Delegation related problems

    • Potential for long delegation chains & long resolution delays

    • Lame delegation: Child fails to inform parent of IP address change

    • Cyclic dependencies

    • Greatly amplified opportunities for compromise, poisoning & delegation to rogue servers.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Exploiting delegation l.jpg

Exploiting Delegation

  • How do we exploit delegation to increase DNS resilience?

    • Diversity: physical, geographical and organizational

    • Careful selection of nodes to delegate to

    • Active monitoring of delegation related anomalies.

  • Delegation Sentry

    • Overlay based monitoring of delegations

      • Each DNS server monitored by a selected set of peers

    • Need to establish policies for selecting delegate

      • Based on reputation, availability, capacity, location, domain, chain length, etc.

      • Mechanisms to defeat compromised servers.

    • Automated checking of problems such as lame delegation, cyclic dependencies, etc.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Conclusions l.jpg

Conclusions

  • DNS has numerous vulnerabilities & easy to attack

    • Several proposed solutions, none entirely satisfactory

    • Large deployed base resists significant overhaul

  • Securing DNS remains a challenge

    • Combine the best of DNSSEC, CoDoNS & DoX.

    • Automated detection and correction of delegation problems.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


That s all folks questions l.jpg

That’s all folks!Questions?

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Bgp references l.jpg

BGP References

  • A.L. Barabasi and R. Albert, “Emergence of Scaling in Random Networks,” Science, pp. 509–512, Oct. 1999.

  • A. Basu, C.L. Ong, et. al. “Route oscillations in IBGP with route reflection”, In Proc. ACM SIGCOMM (Pittsburgh, PA, Aug. 2002).

  • A. Bremler-Barr, Y. Afek, and S. Schwarz, “Improved BGP convergence via ghost flushing,” in Proc. IEEE INFOCOM 2003, vol. 2, San Francisco, CA, Mar 2003, pp. 927–937.

  • S. Deshpande and B. Sikdar,“On the Impact of Route Processing and MRAI Timers on BGP Convergence Times,” in Proc. GLOBECOM 2004, Vol. 2, pp 1147- 1151.

  • L. Gao, T.G. Griffin, J. Rexford, “Inherently safe backup routing with BGP”, In Proc. IEEE INFOCOM (Anchorage, AK, Apr. 2001)

  • T.G. Griffin and B.J. Premore, “An experimental analysis of BGP convergence time,” in Proc. ICNP 2001, Riverside, California, Nov 2001, pp. 53–61.

  • T.G. Griffin, F.B. Shepherd, G. Wilfong, “The stable paths problem and inter-domain routing”, IEEE/ACM Transactions on Networking 10, 1 (2002), 232–243.

  • F. Hao, S. Kamat, and P. V. Koppol, "On metrics for evaluating BGP routing convergence," Bell Labora- tories Tech. Rep., 2003.

  • C. Labovitz, G. R. Malan, and F. Jahanian, “Internet Routing Instability,” IEEE/ACM Transactions on Networking, vol. 6, no. 5, pp. 515–528, Oct. 1998.

  • C. Labovitz, Ahuja, et al., “Delayed internet routing convergence,” in Proc. ACM SIGCOMM 2000, Stockholm, Sweden, Aug 2000, pp. 175–187.

  • C. Labovitz, A. Ahuja, et al., “The Impact of Internet Policy and Topology on Delayed Routing Convergence,” in Proc. IEEE INFOCOM 2001, vol. 1, Anchorage, Alaska, Apr 2001, pp. 537–546.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Bgp references86 l.jpg

BGP References

  • A. Lakhina, J.W. Byers, et al., “On the Geographic Location of Internet Resources,” IEEE Journal on Selected Areas in Communications, vol. 21 , no. 6, pp. 934–948, Aug. 2003.

  • R. Mahajan, D. Wetherall, T. Anderson, “Understanding BGP misconfiguration”, In Proc. ACM SIGCOMM, (Pittsburgh, PA, Aug. 2002), pp. 3–17.

  • A. Medina, A. Lakhina, et al., “Brite: Universal topology generation from a user’s perspective,” in Proc. MASCOTS 2001, Cincinnati, Ohio, Aug 2001, pp. 346-353.

  • N. Feamster & H. Balakrishnan, “Detecting BGP faults with static analysis”, Proc. of NSDI 2005

  • D. Obradovic, “Real-time Model and Convergence Time of BGP,” in Proc. IEEE INFOCOM 2002, vol. 2, New York, June 2002, pp. 893–901.

  • D. Pei, et al., "A study of packet delivery perfor- mance during routing convergence," in Proc. DSN 2003, San Francisco, CA, June 2003, pp. 183-192.

  • Dan Pei, B. Zhang, et al., “An analysis of convergence delay in path vector routing protocols,” Computer Networks, vol. 30, no. 3, Feb. 2006, pp. 398–421.

  • D. Pei, X. Zhao, et al., “Improving BGP convergence through consistency assertions,” in Proc. IEEE INFOCOM 2002, vol. 2, New York, NY, June 23–27, 2002, pp. 902–911.

  • Y. Rekhter, T. Li, and S. Hares, “Border Gateway Protocol 4,” RFC 4271, Jan. 2006.

  • J. Rexford, J. Wang, et al., “BGP routing stability of popular destinations,” in Proc. Internet Measurement Workshop 2002, Marseille, France, Nov. 6–8, 2002, pp. 197–202.

  • A. Sahoo, K. Kant, and P. Mohapatra, “Characterization of BGP recovery under Large-scale Failures,” in Proc. ICC 2006, Istanbul, Turkey, June 11–15, 2006.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Bgp references87 l.jpg

BGP References

  • A. Sahoo, K. Kant, and P. Mohapatra, “Improving BGP Convergence Delay for Large Scale Failures,” in Proc. DSN 2006, June 25-28, 2006, Philadelphia, Pennsylvania, pp. 323-332.

  • A. Sahoo, K. Kant, and P. Mohapatra, "Speculative Route Invalidation to Improve BGP Convergence Delay under Large-Scale Failures," in Proc. ICCCN 2006, Arlington, VA, Oct. 2006.

  • A. Sahoo, K. Kant, and P. Mohapatra, “Improving Packet Delivery Performance of BGP During Large-Scale Failures", submitted to Globecom 2007.

  • “SSFNet: Scalable Simulation Framework”. [Online]. Available: http://www.ssfnet.org/

  • W. Sun, Z. M. Mao, K. G. Shin, “Differentiated BGP Update Processing for Improved Routing Convergence,” in Proc. ICNP 2006, Santa Barbara, CA, Nov. 12–15, 2006 , pp. 280–289.

  • H. Tangmunarunkit, J. Doyle, et al, “Does Size Determine Degree in AS Topology?,” ACM SIGCOMM, vol. 31, issue 5, pp. 7–10, Oct. 2001.

  • R. Teixeira, S. Agarwal, and J. Rexford, “BGP routing changes: Merging views from two ISPs,” ACM SIGCOMM, vol. 35, issue 5, pp. 79–82, Oct. 2005.

  • B. Waxman, “Routing of Multipoint Connections,” IEEE Journal on Selected Areas in Communications, vol. 6, no. 9, pp. 1617–1622, Dec. 1988.

  • B. Zhang, R. Liu, et al., “Measuring the internet’s vital statistics: Collecting the internet AS-level topology ,” ACM SIGCOMM, vol. 35, issue 1, pp. 53–61, Jan. 2005.

  • B. Zhang, D. Massey, and L. Zhang, "Destination Reachability and BGP Convergence Time," in Proc. GLOBECOM 2004, vol. 3, Dallas, TX, Nov 3, 2004, 1383-1389.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Dns references l.jpg

DNS References

  • R. Arends, R. Austein, et.al, ``DNS Security Introduction & Requirements,'' RFC 4033, 2005.

  • G. Ateniese & S. Mangard, ``A new approach to DNS security (DNSSEC),'' in Proc. 8th ACM conf on comp & comm system security, 2001.

  • D. Atkins & R. Austein, ``Threat analysis of the domain name system,'‘ http://www.rfc-archive.org/getrfc.php?rfc=3833, August 2004.

  • R. Curtmola, A. D. Sorbo, & G. Ateniese, ``On the performance and analysis of dns security extensions,'' in Proceedings of CANS, 2005.

  • M. Theimer & M. B. Jones, ``Overlook: Scalable name service on an overlay network,'' in Proc. 22nd ICDCS, 2002.

  • K. Park, V. Pai, et.al, ``CoDNS: Improving DNS performance and reliability via cooperative lookups,'' in Proc. 6th Symp on OS design & impl., 2004.

  • V. Pappas, Z. Xu, S. Lu, D. Massey, A. Terzis, and L. Zhang, “Impact of configuration errors on DNS robustness,” SIGCOMM CCR., vol. 34, no. 4, pp. 319–330, 2004.

  • L. Yuan, K. Kant, et. al, ``DoX: A peer-to-peer antidote for DNS cache poisoning attacks,'' in Proc. IEEE ICC, 2006.

  • L. Yuan, K. Kant & P. Mohapatra, ``A proxy view of quality of domain name service,'' in IEEE Infocom 2007.

  • V. Ramasubramanian and E. G. Sirer, “Perils of transitive trust in the Domain Name System,” in Proc. International Measurement Conference, 2005.

  • V. Ramasubramanium & E.G. Sirer, “The design & implementation of next generation name service for internet”, Sigcom 2004.

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Backup l.jpg

Backup

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Quality of dns service qodns l.jpg

Quality of DNS Service(QoDNS)

Availability

  • Measures if DNS can answer the query.

  • Prob of correct referral when record is not cached.

    Accuracy

  • Prob of hitting a stale record in proxy cache

    Poison Propagation

  • Prob(poison at leaf level at t=t | level k poisoned at t=0)

    Latency – Additional time per query

    Overhead – Additional msgs/BW per query

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Computation of metrics l.jpg

Y

TTL

XR

TTL expiration

O

C

MR

Computation of Metrics

  • Modification at authoritative server

    • Copy obsolete but proxy not aware until TTL expires & a new query forces a reload

modification

U

X

hit

miss

hit

hit

hit

miss

  • XR: Residual time of query arrival process

  • MR: Residual time of modification process

  • Y: Inter-miss period = TTL + XR

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Dealing with hierarchy l.jpg

Dealing with Hierarchy

  • A miss at a node  a query at its parent

  • Superposition of miss processes of children  query arrival process of parent

  • Recursively derive the arrival process bottom-up

.gov

Level h-1

.sa

Level h

.ips

.gb

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


Deriving qodns metrics l.jpg

X

Deriving QoDNS Metrics

  • Accuracy:

    • Prob. leaf record is “current”

  • (Un)Availability

    • Prob. BMR is “Obsolete” referral

  • Latency

    • RTT x # referrals

  • Overhead

    • Related to #referrals for current RRs & #tries for obsolete RRs

BMR is Current

.sg

.nz

.au

.gov

.gov

.edu

BMR is Obsolete

Referral

.sa

.ips

.gb

.gb

.xys

.qwe

.abc

.abc

BMR is Obsolete

Record

K. Kant, Surviving Large Scale Internet Outages -- A Tutorial


  • Login