experiences with internet traffic measurement and analysis l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Experiences With Internet Traffic Measurement and Analysis PowerPoint Presentation
Download Presentation
Experiences With Internet Traffic Measurement and Analysis

Loading in 2 Seconds...

play fullscreen
1 / 89

Experiences With Internet Traffic Measurement and Analysis - PowerPoint PPT Presentation


  • 128 Views
  • Uploaded on

Experiences With Internet Traffic Measurement and Analysis. Vern Paxson ICSI Center for Internet Research International Computer Science Institute and Lawrence Berkeley National Laboratory vern@icir.org March 5th, 2004. Outline. The 1990s: How is the Internet used? Growth and diversity

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Experiences With Internet Traffic Measurement and Analysis' - chantrea


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
experiences with internet traffic measurement and analysis

Experiences With Internet Traffic Measurement and Analysis

Vern Paxson

ICSI Center for Internet Research

International Computer Science Institute

and

Lawrence Berkeley National Laboratory

vern@icir.org

March 5th, 2004

outline
Outline
  • The 1990s: How is the Internet used?
    • Growth and diversity
    • Fractal traffic, “heavy tails”
    • End-to-end dynamics
    • Difficulties with measurement & analysis
  • The 2000s: How is the Internet abused?
    • Prevalence of misuse
    • Detecting attacks
    • Worms
the 1990s

The 1990s

How is the Internet Used?

slide4

= 80% growth/year

Data courtesy of Rick Adams

internet growth exponential
Internet Growth: Exponential
  • Growth of 80%/year
  • Sustained for at least ten years …
  • … before the Web even existed.
  • Internet is always changing. You do not have a lot of time to understand it.
characterizing site traffic
Characterizing Site Traffic
  • Methodology: passively record traffic in/out of a site
  • Danzig et al (1992)
    • 3 sites, 24 hrs, all packet headers
  • Paxson (1994)
    • TCP SYN/FIN/RST control packets
      • Gives hosts, sizes, start time, duration, application
      • Large filtering win (≈ 10-100:1 packets, 1000s:1 bytes)
    • 7 month-long traces at Lawrence Berkeley Natl. Laboratory
    • 8 day-long traces from 6 other sites
findings from site studies
Findings from Site Studies
  • Traffic mix (which protocols are used; how many connections/bytes they contribute) varies widely from site to site.
  • Mix also varies at the same site over time.
  • Most connections have much heavier traffic in one direction than the other:
    • Even interactive login sessions (20:1)
findings from site studies con t
Findings from Site Studies, con’t
  • Many random variables associated with connection characteristics (sizes, durations) are best described with log-normal distributions
    • But often these are not particularly good fits
    • And often their parameters vary significantly between datasets
  • The largest connections in bulk transfers are very large
    • Tail behavior is unpredictable
  • Many of these findings differ from assumptions used in 1990s traffic modeling
theory vs measured reality

Theory vs. Measured Reality

Scaling behavior in Internet Traffic

burstiness
Burstiness
  • Long-established framework: Poisson modeling
  • Central idea: network events (packet arrivals, connection arrivals) are well-modeled as independent
  • In simplest form, there’s just a rate parameter, 
  • It then follows that the time between “calls” (events) is exponentially distributed, # of calls ~ Poisson
  • Implications (if assumptions correct):
    • Aggregated traffic will smooth out quickly
    • Correlations are fleeting, bursts are limited
burstiness theory vs measurement
Burstiness: Theory vs. Measurement
  • For Internet traffic, Poisson models have fundamental problem: they greatly underestimate burstiness
  • Consider an arrival process: Xk gives # packets arriving during kth interval of length T.
    • Take 1-hour trace of Internet traffic (1995)
    • Generate (batch) Poisson arrivals with same mean and variance
burstiness over many time scales
Burstiness OverMany Time Scales
  • Real traffic has strong, long-range correlations
  • Power spectrum:
    • Flat for Poisson processes
    • For measured traffic, diverges to  as   0
  • To build Poisson-based models that capture this characteristic takes many parameters
  • But due to great variation in Internet traffic, we are desperate for parsimonious models (few parameters)
describing traffic with fractals
Describing Traffic with Fractals
  • Landmark 1993 paper by Leland et al proposed capturing such characteristics (in Ethernet traffic) using self-similarity, a form of fractal-based modeling:
    • Parameterized by mean, variance, and Hurst parameter
  • Models predict burstiness on all time scales
    • Queueing delays / drop probabilities much higher than predicted by Poisson-based models
heavy tails
Heavy Tails
  • Key prediction from fractal modeling:
    • One way fractal traffic can arise in aggregate is if individual connections have activity periods (durations, sizes) whose distribution has infinite variance.
  • Infinite variance manifests in distribution’s upper tail
  • Consider Pareto distribution, F(x) = (x/a)-
    • If  < 2, then F(x) has infinite variance
    • Can test for Pareto fit by plotting log F(x) vs. log x
      • Straight line = Pareto distribution, slope estimates -
web connection sizes 226 386 observations
Web connection sizes(226,386 observations)

28,000observations

 = 1.3

 Infinite Variance

self similarity heavy tails con t
Self-Similarity & Heavy Tails, con’t
  • We find heavy-tailed sizes in many types of network traffic. Just a few extreme connections dominate the entire volume.
self similarity heavy tails con t25
Self-Similarity & Heavy Tails, con’t
  • We find heavy-tailed sizes in many types of network traffic. Just a few extreme connections dominate the entire volume.
  • Theorems then give us that this traffic aggregates to self-similar behavior.
  • While self-similar models are parsimonious, they are not (alas) “simple”.
  • You can have self-similar correlations for which magnitude of variations is small  still possible to have a statistical multiplexing gain, especially at very high aggregation
  • Smaller time scales behave quite differently.
    • When very highly aggregated, they can appear Poisson!
end to end dynamics
End-to-End Dynamics
  • Ultimately what the user cares about is not what’s happening on a given link, but the concatenation of behaviors along all of the hops in an end-to-end path.
  • Measurement methodology: deploy measurement servers at numerous Internet sites, measure the paths between them
  • Exhibits N2 scaling: as # sites grows, # paths between them grows rapidly.
end to end routing dynamics
End-to-End Routing Dynamics
  • Analysis of 40,000 “traceroute” measurements between 37 sites, 900+ end-to-end paths.
  • Route prevalence:
    • most end-to-end paths through the Internet dominated by a single route.
  • Route persistence:
    • 2/3’s of routes remain unchanged for days/weeks
    • 1/3 of routes change on time scales of seconds to hours
  • Route symmetry:
    • More than half of all routes visited at least one different city in each direction
    • Very important for tracking connection state inside network!
end to end packet dynamics
End-to-End Packet Dynamics
  • Analysis of 20,000 TCP bulk transfers of 100 KB between 36 sites
  • Each traced at both ends using tcpdump
  • Benefits of using TCP:
    • Real-world traffic
    • Can probe fine-grained time scales but using congestion control
  • Drawbacks to using TCP:
    • Endpoint TCP behavior a major analysis headache
    • TCP’s loading of the transfer path also complicates analysis
end to end packet dynamics unusual behavior
End-to-End Packet Dynamics: Unusual Behavior
  • Out-of-order delivery:
    • Not uncommon. 0.6%-2% of all packets.
    • Strongly site-specific.
    • Generally little impact on performance.
  • Replicated packets:
    • Very rare, but does occur (e.g., 1 packet in, 22 out)
  • Corrupted packets (bad checksum):
    • Overall, 1 in 5,000 (!)
    • Stone/Partridge (2000): between 1 in 1,100 and 1 in 32,000
      • Undetected: between 1 in 16 million and 1 in 10 billion
end to end packet dynamics loss
End-to-End Packet Dynamics: Loss
  • Half of all 100 KB transfers experienced no loss
    • 2/3s of paths within U.S.
  • The other half experienced significant loss:
    • Average 4-9%, but with wide variation
  • TCP loss is not well described as independent
  • Losses dominated by a few long-lived outages
  • (Keep in mind: this is 1994-1995!)
  • Subsequent studies:
    • Loss rates have gotten much better
    • Loss episodes well described as independent
    • Same holds for regions of stable delay, throughput
    • Time scales of constancy  minutes or more
there is no such thing as typical
There is No Such Thing as “Typical”
  • Heterogeneity in:
    • Traffic mix
    • Range of network capabilities
      • Bottleneck bandwidth (orders of magnitude)
      • Round-trip time (orders of magnitude)
    • Dynamic range of network conditions
      • Congestion / degree of multiplexing / available bandwidth
      • Proportion of traffic that is adaptive/rigid/attack
  • Immense size & growth
    • Rare events will occur
  • New applications explode on the scene
there is no such thing as typical con t
There is No Such Thing as “Typical”, con’t
  • New applications explode on the scene
    • Not just the Web, but: Mbone, Napster, KaZaA etc., IM
  • Event robust statistics fail.
    • E.g., median size of FTP data transfer at LBL
      • Oct. 1992: 4.5 KB (60,000 samples)
      • Mar. 1993: 2.1 KB
      • Mar. 1998: 10.9 KB
      • Dec. 1998: 5.6 KB
      • Dec. 1999: 10.9 KB
      • Jun. 2000: 62 KB
      • Nov. 2000: 10 KB
  • Danger: if you misassume that something is “typical”, nothing tells you that you are wrong!
the search for invariants
The Search for Invariants
  • In the face of such diversity, identifying things that don’t change has immense utility
  • Some Internet traffic invariants:
    • Daily and weekly patterns
    • Self-similarity on time scales of 100s of msec and above
    • Heavy tails
      • both in activity periods and elsewhere, e.g., topology
    • Poisson user session arrivals
    • Log-normal sizes (excluding tails)
    • Keystrokes have a Pareto distribution
the danger of mental models
The Danger of Mental Models

“Exponential plus a constant offset”

slide40

Not exponential - Pareto!

Heavy tail:  ≈ 1.0

versus the power of modeling to open our eyes
Versus the Power of Modeling toOpen Our Eyes
  • Fowler & Leland, 1991:
    • Traffic ‘spikes’ (which cause actual losses) ride on longer-term ‘ripples’, that in turn ride on still longer-term ‘swells’
versus the power of modeling to open our eyes43
Versus the Power of Modeling toOpen Our Eyes
  • Fowler & Leland, 1991:
    • Traffic ‘spikes’ (which cause actual losses) ride on longer-term ‘ripples’, that in turn ride on still longer-term ‘swells’
  • Lacked vocabulary that came from self-similar modeling (1993)
  • Similarly, 1993 self-similarity paper:
    • We did so without first studying and modeling the behavior of individual Ethernet users (sources)
  • Modeling led to suggestion to investigate heavy tails
measurement soundness
Measurement Soundness
  • How well-founded is a given Internet measurement?
  • We can often use additional information to help calibrate.
  • One source: protocol structure
    • E.g., was a packet dropped by the network …… or by the measurement device?
      • For TCP, can check: did receiver acknowledge it?
        • If Yes, then dropped by measurement device
        • If No, then dropped by network
  • Can also calibrate using additional information
reproducibilty of results or lack thereof
Reproducibilty of Results(or lack thereof)
  • It is rare, though sometimes occurs, that raw measurements are made available to other researchers for further analysis or for confirmation.
  • It is more rare that analysis tools and scripts are made available, particularly in a coherent form that others can actually get to work.
  • It is even rarer that measurement glitches, “outliers,” analysis fudge factors, etc., are detailed.
  • In fact, often researchers cannot reproduce their own results.
towards reproducible results
Towards Reproducible Results
  • Need to ensure a systematic approach to data reduction and analysis
    • I.e., a “paper trail” for how analysis was conducted, particularly when bugs are fixed
  • A methodology to do this:
    • Enforce discipline of using a single (master) script that builds all analysis results from the raw data
    • Maintain all intermediary/reduced forms of the data as explicitly ephemeral
    • Maintain a notebook of what was done and to what effect.
    • Use version control for scripts & notebook.
    • But also really need: ways to visualize what's changed in analysis results after a re-run.
the 2000s

The 2000s

How is the Internet Abused?

magnitude of internet attacks
Magnitude of Internet Attacks
  • As seen at Lawrence Berkeley National Laboratory, on a typical day in 2004:
    • > 70% of Internet connections (20 million out of 28 million) reflect clear attacks.
    • 60 different remote hosts scan one of LBL’s two blocks of 65,536 address in its entirety
    • More than 10,000 remote hosts engage in scanning activity
  • Much of this activity reflects “worms”
  • Much of the rest reflects automated scan-and-exploit tools
how is the internet abused

How is the Internet Abused?

Detecting Network Attacks

design goals for the bro intrusion detection system
Design Goals for the “Bro” Intrusion Detection System
  • Monitor traffic in a very high performance environment
  • Real-time detection and response
  • Separation of mechanism from policy
  • Ready extensibility of both mechanism and policy
  • Resistant to evasion
slide54

How Bro Works

  • Taps GigEther fiber link passively, sends up a copy of all network traffic.

Network

slide55

How Bro Works

Tcpdump

Filter

Filtered Packet

Stream

  • Kernel filters down high-volume stream via standard libpcap packet capture library.

libpcap

Packet Stream

Network

slide56

How Bro Works

Event

Control

Event

Stream

  • “Event engine” distills filtered stream into high-level, policy-neutral events reflecting underlying network activity
    • E.g., connection_attempt, http_reply, user_logged_in

Event Engine

Tcpdump

Filter

Filtered Packet

Stream

libpcap

Packet Stream

Network

slide57

How Bro Works

Policy

Script

Real-time Notification

Record To Disk

  • “Policy script” processes event stream, incorporates:
    • Context from past events
    • Site’s particular policies

Policy Script Interpreter

Event

Control

Event

Stream

Event Engine

Tcpdump

Filter

Filtered Packet

Stream

libpcap

Packet Stream

Network

slide58

How Bro Works

Policy

Script

Real-time Notification

Record To Disk

  • “Policy script” processes event stream, incorporates:
    • Context from past events
    • Site’s particular policies
  • … and takes action:
      • Records to disk
      • Generates alerts via syslog, paging
      • Executes programs as a form of response

Policy Script Interpreter

Event

Control

Event

Stream

Event Engine

Tcpdump

Filter

Filtered Packet

Stream

libpcap

Packet Stream

Network

experiences with bro
Experiences with Bro
  • Exciting research because used operationally (24x7) at several open sites (LBL, UCB, TUM)
  • Key enabler: sites’ threat model
    • Occasional break-ins are tolerable
    • “Jewels” are additionally protected (e.g., firewalls)
  • Significant real-world concern: policy management
  • Dynamic blocking critical to success
    • Currently, 100-200 blocks/day
the problem of evasion
The Problem of Evasion
  • Fundamental problem passively measuring traffic on a link:Network traffic is inherentlyambiguous
  • Generally not a significant issue for traffic characterization
  • But is in the presence of an adversary: Attackers can craft traffic to confuse/fool monitor
the problem of crud
The Problem of “Crud”
  • There are many such ambiguities attackers can leverage.
  • Unfortunately, they occur in benign traffic, too:
    • Legitimate tiny fragments, overlapping fragments
    • Receivers that acknowledge data they did not receive
    • Senders that retransmit different data than originally
  • In a diverse traffic stream, you will see these
  • Approaches for defending against evasion:
    • Traffic “normalizers” that actively remove ambiguities
    • “Mapping” of local hosts to determine their behaviors
    • Active participation by local hosts in intrusion detection
how is the internet abused63

How is the Internet Abused?

The Threat of Internet Worms

what is a worm
What is a Worm?
  • Self-replicating/self-propagating code.
  • Spreads across a network by exploiting flaws in open services.
    • As opposed to viruses, which require user action to quicken/spread.
  • Not new --- Morris Worm, Nov. 1988
    • 6-10% of all Internet hosts infected
  • Many more since, but none on that scale ….

until ….

code red
Code Red
  • Initial version released July 13, 2001.
  • Exploited known bug in Microsoft IIS Web servers.
  • 1st through 20th of each month: spread.20th through end of each month: attack.
  • Payload: web site defacement.
  • Spread: via random scanning of 32-bitIP address space.
  • But: failure to seed random number generator  linear growth.
code red con t
Code Red, con’t
  • Revision released July 19, 2001.
  • Payload: flooding attack onwww.whitehouse.gov.
  • Bug lead to it dying for date ≥ 20th of the month.
  • But: this time random number generator correctly seeded. Bingo!
network telescopes
Network Telescopes
  • Idea: monitor a cross-section of the IP address space to measure network traffic involving random addresses (flooding “backscatter”; worm scanning)
  • LBL’s cross-section: 1/32,768 of Internet.
  • UCSD’s cross-section: 1/256.
spread of code red
Spread of Code Red
  • Network telescopes give lower bound on # infected hosts: 360K.
  • Course of infection fits classic logistic.
  • Note: larger the vulnerable population, faster the worm spreads.
  • That night ( 20th), worm dies …

… except for hosts with inaccurate clocks!

  • It just takes one of these to restart the worm on August 1st …
striving for greater virulence code red 2
Striving for Greater Virulence: Code Red 2
  • Released August 4, 2001.
  • Comment in code: “Code Red 2.”
  • But in fact completely different code base.
  • Payload: a root backdoor, resilient to reboots.
  • Bug: crashes NT, only works on Windows 2000.
  • Localized scanning: prefers nearby addresses.
  • Kills Code Red I.
  • Safety valve: programmed to die Oct 1, 2001.
striving for greater virulence nimda
Striving for Greater Virulence: Nimda
  • Released September 18, 2001.
  • Multi-mode spreading:
    • attack IIS servers via infected clients
    • email itself to address book as a virus
    • copy itself across open network shares
    • modifying Web pages on infected servers w/ client exploit
    • scanning for Code Red II backdoors (!)
  • worms form an ecosystem!
  • Leaped across firewalls.
a lesson in economy
A Lesson in Economy
  • Slammer exploits a connectionless UDP service, rather than connection-oriented TCP.
  • Entire worm fits in a single packet!
  • When scanning, worm can “fire and forget”.
  • Worm infects 75,000+ hosts in 10 minutes (despite broken random number generator).
  • Progress limited by the Internet’s carrying capacity!
blaster
Blaster
  • Released August 11, 2003.
  • Exploits flaw in RPC service ubiquitous across Windows.
  • Payload: attack Microsoft Windows Update.
  • Despite flawed scanning and secondary infection strategy, rapidly propagates to 100K’s of hosts.
  • Actually, bulk of infections are really Nachia, a Blaster counter-worm.
  • Key paradigm shift: firewalls don’t help.
what if spreading were well designed
What if Spreading WereWell-Designed?
  • Observation (Weaver): Much of a worm’s scanning is redundant.
  • Idea: coordinated scanning
    • Construct permutation of address space
    • Each new worm starts at a random point
    • Worm instance that “encounters” another instance re-randomizes.
  • Greatly accelerates worm in later stages.
what if spreading were well designed con t
What if Spreading WereWell-Designed?, con’t
  • Observation (Weaver): Accelerate initial phase using a precomputed hit-list of say 1% vulnerable hosts.
  • At 100 scans/worm/sec, can infect huge population in a few minutes.
  • Observation (Staniford): Compute hit-list of entire vulnerable population, propagate via divide & conquer.
  • At 10 scans/worm/sec, infect in 10s of sec!
defenses
Defenses
  • Detect via honeyfarms: collections of “honeypots” fed by a network telescope.
    • Any outbound connection from honeyfarm = worm.
    • Distill signature from inbound/outbound traffic.
    • If telescope covers N addresses, expect detection when worm has infected 1/N of population.
  • Thwart via scan suppressors: network elements that block traffic from hosts that make failed connection attempts to too many other hosts.
defenses85
Defenses?
  • Observation:worms don’t need to randomly scan
  • Meta-server worm: ask server for hosts to infect. E.g., query Google for “index.html”.
  • Topological worm: fuel spread with local information from infected hosts (web server logs, email address books, config files, SSH “known hosts”)
  • No scanning signature; with rich inter- connection topology, potentially very fast.
defenses86
Defenses??
  • Contagion worm: propagate parasitically along with normally initiated communication.
  • E.g., using 2 exploits - Web browser & Web server - infect any vulnerable servers visited by browser, then any vulnerable browsers that come to those servers.
  • E.g., using 1 KaZaA exploit, glide along immense peer-to-peer network in days/hours.
  • No unusual connection activity at all! :-(
some observations
Some Observations
  • Today’s worms have significant real-world impact:
    • Code Red disrupted routing
    • Slammer disrupted elections, ATMs, airline schedules, operations at an off-line nuclear power plant …
    • Blaster possibly contributed to North American Blackout of Aug. 2003
  • But today’s worms are amateurish
    • Frequent bugs, algorithm/attack botches
    • Unimaginative payloads
next generation worm authors
Next-Generation Worm Authors
  • Potential for major damage with more nasty payloads :-(.
  • Military (“cyberwarfare”)
  • Criminals:
    • Denial-of-service, spamming for hire
    • Access for Sale: A New Class of Worm(Schecter/Smith, ACM CCS WORM 2003)
  • Money on the table  Arms race
summary
Summary
  • Internet measurement is deeply challenging:
    • Immense diversity
    • Internet never ceases to be a moving target
    • Our mental models can betray us: the Internet is full of surprises!
    • Seek invariants
  • Many of the last decade’s measurement questions -- “What are the basic characteristics and properties of Internet traffic?” -- have returned …
  • … but now regarding Internet attacks
  • What on Earth will the next decade hold??