Misuse and Anomaly Detection - PowerPoint PPT Presentation

Misuse and anomaly detection
1 / 21

  • Uploaded on
  • Presentation posted in: General

Misuse and Anomaly Detection. Sampath Kannan Wenke Lee Insup Lee Diana Spears Oleg Sokolsky William Spears Linda Zhao. Network Intrusion Detection Systems (NIDS). Important defense to protect sensitive information and resources on the network.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Misuse and Anomaly Detection

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Misuse and anomaly detection

Misuse and Anomaly Detection

Sampath Kannan Wenke Lee

Insup Lee Diana Spears

Oleg Sokolsky William Spears

Linda Zhao

Network intrusion detection systems nids

Network Intrusion Detection Systems (NIDS)

  • Important defense to protect sensitive information and resources on the network.

  • Usually have the following functionalities.

    • Observe traffic and extract features

    • Pattern match with database of “attack signatures” to detect misuse(intrusion)

    • Observe statistical properties and check against specifications of correct behavior to detect anomalies

Shortcomings of current nids

Shortcomings of Current NIDS

  • New attack strategies arise constantly and attack signature databases become obsolete rapidly.

  • Volume and interleaving of traffic at backbone of network makes complex signature recognition infeasible.

Shortcomings cont d

Shortcomings cont’d

  • Anomaly detection algorithms are primitive. We want more scalable yet moresophisticated techniques.

  • Want to reduce the number of false positives in anomaly detection to make it useful.

Our approach

Our Approach

  • Use Machine Learning, Data Mining, and Case-Based Reasoning techniques to learn new intruder models on the fly.

  • Build a taxonomy of possible anomalies; extract relevant features; use statistical and machine learning techniques to reduce false-alarm rate.

Our approach cont d

Our Approach – Cont’d

  • Apply sophisticated algorithms designed inthe resource-constrained data stream model to NIDS.

  • Integrate all of these modules into a MaC-based system architecture.

Existing infrastructure

Existing Infrastructure

  • Monitoring and Checking (MaC) architecture for run-time monitoring

    • User specified instrumentation of running programs to extract important state changes.(Primitive Event Definition Language (PEDL)).

    • User specified conversion of these low-level events to abstract events relevant to properties (MEDL).

    • Checker for processing abstract event streamto monitor correctness.

Existing infrastructure cont d

Existing Infrastructure – Cont’d

  • An experimental test-bed to test performance of Intrusion and Anomaly Detection Systems.

    • Enhancement of a similar set-up from MIT Lincoln Labs from the 90’s.

    • Models hacker profiles and taxonomy of attacks and generates “realistic” normal and attack traffic.

    • Metrics for evaluating potency of attacks.

Using mac for nids

Using MaC for NIDS

  • Need multiple Primitive Event Definition Languages (PEDLs) to model different algorithmic techniques for extracting abstract events.

  • Need dynamically changeable properties as machine learning approaches discover new attack signatures.

  • Need integration module that combines the results of various modules.

Inferring mixtures of markov chains

Inferring Mixtures of Markov Chains

A theoretical result ...

Batu, Guha, Kannan

An example

An example

  • Network traffic log … each party behaves like a Markov Chain

  • Some parties are malicious

  • Can you tease out the malicious chains from a single common log?

Another example browsing habits

Another Example: Browsing habits

  • You read sports and cartoons. You’re equally likely to read both. You do not remember what you read last.

  • You’d expect a “random” sequence


Suppose there are two

Suppose there are two

  • I like health, entertainment, and fashion

  • I always read entertainment first, health next and fashion last

  • The sequence would be


Two readers one log file

Two readers, one log file

  • If there is one log file…

  • Assume there is no correlation between us


Is there enough information to tell that there are two people browsing?

What are they browsing? How are they browsing?

Clues in stream

Clues in stream?

  • Yes! (under model assumptions).

  • H,E, F have special relationship.

  • They cannot belong to different (uncorrelated) people.

  • Not clear about S and C ... Could be 3 uncorrelated persons.


Markov chains as stochastic sources

Markov Chains as Stochastic Sources




Output sequence:

1 4 7 7 1 2 5 7 ...



















Markov chains on s e c h f







Markov chains on S,E,C,H,F

Modeled by …







Problem statement informal

Problem Statement (informal)

  • Two or more probabilistic processes

  • We are observing interleaved behavior

  • We do not know which state belongs to which process – cold start.

The problem

The Problem

... 1 3 2 5 1 4


...2 6 1 3 2 7 5 3 1 4 1


... 2 6 7 3 1

Observe ...2 6 1 3 2 7 5 3 1 4 1...

Infer: MC1, MC2, & mixing parameters

Misuse and anomaly detection

  • For our problem we assume:

  • Stream is polynomially long in the number of states of each Markov chain (need perhaps long stream).

    • C : maximum cover time

    • Q : upper bound on the denominator of any probability

  • Nonzero probabilities are bounded away from 0.

  • Space available is some small polynomial in #states.

  • Under these assumptions, we can identify individual chains if their state spaces are disjoint.

Research directions

Research Directions

  • Many exciting directions

  • Our research team has expertise in network security, machine learning, AI, real-time systems, and algorithm design

  • We expect interesting synergies between these strengths.

  • Login