information fusion for cyber situation awareness l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
INFORMATION FUSION FOR CYBER SITUATION AWARENESS PowerPoint Presentation
Download Presentation
INFORMATION FUSION FOR CYBER SITUATION AWARENESS

Loading in 2 Seconds...

play fullscreen
1 / 18

INFORMATION FUSION FOR CYBER SITUATION AWARENESS - PowerPoint PPT Presentation


  • 775 Views
  • Uploaded on

INFORMATION FUSION FOR CYBER SITUATION AWARENESS. George Tadda Fusion Technology Branch Information Directorate Air Force Research Laboratory E-mail: george.tadda@rl.af.mil Phone: 315-330-3957. Outline. Introduction Motivation Situation Awareness Reference Model Metrics

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'INFORMATION FUSION FOR CYBER SITUATION AWARENESS' - andrew


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
information fusion for cyber situation awareness

INFORMATION FUSION FOR CYBER SITUATION AWARENESS

George Tadda

Fusion Technology Branch

Information Directorate

Air Force Research Laboratory

E-mail: george.tadda@rl.af.mil

Phone: 315-330-3957

outline
Outline
  • Introduction
  • Motivation
  • Situation Awareness Reference Model
  • Metrics
  • Application of Lessons Learned
work in situation awareness sa
Work in Situation Awareness (SA)
  • Used reference models to demonstrate/build prototype systems for:
      • Cyber (Defense & Security (D&S) ’05, SIMA ‘05)
      • Tactical (ISIF ’02)
      • Global (ISIF ’04)
      • Maritime
      • and Many Others
  • Developed Metrics (D&S ’04) to Evaluate Level 2 Systems and applied them to Cyber (D&S ’05)
    • After much discussion we questioned the difference between tracking objects and situations and whether the majority of the metrics are just another way to measure integrity of tracks
  • Additional Activities:
    • Jean Roy, under The Technical Cooperative Program, presented a

definition of situational analysis and included in "Concepts,

Models, and Tools for Information Fusion“

    • Snidaro, M. Belluz, G. Foresti, “Domain knowledge for

security applications”, ISIF’07 defined types of events (simple,

spatial, and transitive)

    • Dale Lambert, formalizing situation awareness through mathematics

07-210

motivation reality of today s environment
Motivation(Reality of Today’s Environment)

Moving Objects 80/sec

1000’s of Objects

Data

  • The Analyst/Operator
  • Drowning in data and Inundated with “dots” on map or messages. INFORMATION STARVED
  • INCOMPLETE, CONFLICTING DATA
  • SA is Highly Operator Dependent and 100% Mental Process
  • - Stress
  • - Fatigue
  • - Experience
  • LIMITED BY INDIVIDUAL’S ABILITIES

Class B Address Space

26,000 Alerts/day

Data

3 – 4 Petabytes/day

(E-mail, Published Pages, etc)

Data

Today WE Have…

Objects

Tactical

Alerts

Cyber

Events

Global

…and MORE

07-291

motivation
Motivation

Knowledge Of Units

Today WE Have…

Anticipation

Most Likely/

Worst Case eCOA

Sensemaking

What is…

A Measure of Success

SPATIAL

(Obj Types/No.)

Data/Information Ratio (DIR)

(Examples)

Plausible Futures

(Intent, Opportunity,

Capability)

Units

Objects

Tactical

Pre Iran/Kuwait

Conflict

Data

Information

Objects*: 16,203

No. Units: 42

DIR: 386

Knowledge Of Atcks

TRENDS

(Network, Host)

Plausible Futures

(Intent, Opportunity,

Capability)

*No noise/clutter

ALERTS

Attacks

Alerts

Cyber

SKAION Datasets

(3s8, 3s26, 3s28, 3s29)

Data

Information

Events Attacks DIR

20,131 107 188

19.531 66 296

8,681 62 140

31,513 155 203

Knowledge Of Sits

TRENDS

(Economic, Military)

Plausible Futures

(Intent, Opportunity,

Capability)

Situations

Events

Global

EVENTS

Unit A

xx

xx

xx

xx

xx

Data

Information

Information

Information

Information

Information

Assessment

Assessment

Assessment

Assessment

…and MORE

(STEP 1: From Data -> Complex Relations/Situation(s))

(STEP 2: From Complex Relations/Situation(s) -> Anticipation)

07-291

slide6

Goal Focused

Data Focused

BALANCE

Sharing the Stage

(From A Model Perspective)

SITUATION AWARENESS

FUSION - TACTICAL

  • Most popular is the Joint Director’s of Laboratory (JDL) Model (Sensor-based)
  • Functional Model
  • 5 Levels (Level 0, 1, 2, 3, 4)
  • Published By Llinas, Hall, White (1992)
  • Most work concentrated on Level 0/1/4 (Dots on Map)
  • Little definition of Level 2/3 (What do they mean?)
  • Bottom-up, Data Driven
  • Receiving Much Attention Today from the Cognitive Community
  • Mental Model
  • 3 Levels: Perception, Comprehension, Projection
  • Developed by: M. Endsley (1995)
  • Extended by McGuinness and Foy for Resolution
  • Top Down, Goal Driven

07-291

slide7

Assessment

Situation

Situation Awareness Reference Model

(Combining The “Best” Of Both Worlds)

  • Based on JDL & Endsley’s Models
  • - Plus Initial Data Requirement
  • - Textual Inputs (Info Exploit)
  • Define Problem/Goal – Top Down
  • - What/Where/Who…
  • Processing Flow ( )
  • - Projection – The Alert(s)
  • - Comprehension
  • -- Model Analysis
  • - Perception
  • -- Data Collection
  • -- Parsing/Extraction
  • -- Data Cleansing
  • - JDL: Level 0/1
  • Process Refinement ( )
  • - Missing Data
  • - Additional Data
  • - Input for Sensor Mgmt
  • Off-Line Processing ( )
  • - Knowledge Discovery

Level 0/1

Data

Parsing

Sources

Sources

Collection

Extraction

Perception

Data Cleansing

Data

Requirements

Evidence

Model

Knowledge

Knowledge

Additional

Analysis

Discovery

Discovery

Info

Tools

Tools

Tools

Comprehension

Matches/

Partial

Matches

Target

*Missed

Questions

Models

ANTICIPATION

Potential New

Relationships

Tools

The “Problem”

*Based on Model Unfolding

07-291

slide8

Client/Host

Configurations

Mission

Model Matching

Algorithm

Equip Fail

Attack A

Attack N

Potential

Attacks

Assessment

Intrusion

Attempt

Recon

Situation

Privilege

Escalation

A Priori Knowledge

Situation Awareness Reference Model

(Applied to Cyber SA)

Open Source

Host IDS

Snort

Dragon

Network Stats

Web Logs

Sys Logs

Level 0/1

Post Proc

Data

Parsing

Data

Parsing

Collection

Perception

Extraction

Sources

Sources

Collection

Extraction

Perception

Data Cleansing

Data Cleansing

Data

Requirements

Evidence

(Alerts)

Evidence

Model

Knowledge

Knowledge

Additional

Analysis

Discovery

Discovery

Info

Comprehension

Tools

Tools

Tools

Comprehension

Matches/

Partial

Matches

Target

*Missed

Questions

Models

ANTICIPATION

Potential New

Relationships

Tools

The “Goal”

Anticipation

07-291

situation awareness reference model applied to cyber domain
Situation Awareness Reference Model(Applied to Cyber Domain)

Model Template Y

A Priori

Knowledge

Client/Host

Configurations

Snort

Model Template X

Recon

Intrusion

Attempt

Dragon

Privilege

Escalation

Multi-Stage

Attack

Goal

Potential

Attacks

Host IDS

Attack A

The Network

Potential to

Advance to

Next Stage

Evidence

Model Matching

Algorithm

Attack B

Sys Logs

Equip Failure

Post Proc

List Based

On Risk

Anticipation

Web Logs

Impact

Assessment

Network Stats

Business

Model

Perception

Comprehension

Open

Source

TBD

07-291

lexicon background
Lexicon(Background)
  • Evidence
    • IDS Alerts (i.e., Snort, Dragon)
    • System Logs
    • Service Logs (i.e., Apache, IIS)
    • Network Flow Data
  • Track – collection of all evidence available against one or more targets originating from one or more attackers
  • Situation – set of tracks at a snapshot in time
  • Situation Awareness of a Network – analyst’s mental model of the situation
  • True Positive* – successful attack
  • False Positive* – incorrectly identified attack
  • Non-relevant Positive* – correctly identified attack that fails or is incomplete (i.e., try to exploit a ‘blocked’ vulnerability)

*Valeur et al, “A Comprehensive Approach to Intrusion Detection Alert Correlation, IEEE Transactions on Dependable and Secure Computing, Jul-Sep 04

06-081

metrics overview
Metrics Overview
  • Confidence – measures the ability of the system to correctly identify the track(s)
    • Recall: Percentage of tracks detected in relation to the “total known”
    • Precision: Percentage of correct tracks detected in relation to number of detections
    • Fragmentation: Percentage of tracks reported as multiple tracks that should have been reported as a single track
    • Mis-Association: Percentage of tracks that are neither correct nor a fragment in relation to the number of detections
  • Purity – characterizes the quality of the detections
    • Mis-Assignment Rate: Percent of evidence incorrectly assigned to a given track
    • Evidence Recall: Percentage of evidence detected in relation to the “total known” evidence
  • Cost Utility – a single weighted measure of the system in identifying “important or key” tracks with respect to a concept of cost
  • Timeliness – measures the ability of the system to respond within time requirements of a particular domain

06-081

slide12

Cost Utility(Weighted Cost and Attack Score)

∑Weighted Values for Results

Weighted Cost =

∑Weighted Values for Ground Truth

[No. Attacks in Results][No. Results] – [[Sum of Positions of Attacks in Results] – [Geometric Sum ([No Attacks in Results] -1)]]

Attack Score =

[No. Attacks in GT][No. Results]

Given:

100 pts ATTACK

5 pts Background Scan

5 pts Background Attack

-50 pts False Positive

Proposed Attacks

Ground Truth

R0 Background Scan

R1 UNASSIGNED

R2 Attack

R3 Background Scan

R4 Background Scan

R5 Background Attack

5

- 50

100

5

5

5

70

GT0 Background Scan

GT1 Background Attack

GT2 Background Scan

GT3 Attack

GT4 Background Scan

5

5

5

100

5

120

NOTE: Sorted Based on Score

Weighted Cost = 70/120 = .5833

Attack Score = [(1)(6) – (2-(1-1))]/(1)(6) = .6667

06-081

the infrastructure
The Infrastructure

Skaion Dataset

Processing Results

Results

UsingAFRL Schema

Viewing Ground Truth

Cyber Fusion System

AFRL Results

Analyzer Tools

AFRL Ground Truth Correlation

Ground Truth

Assignment Matrix

.csv

.html

List of Potential

Attacks

REPORTS

Alerts correlated to selected Attack Track

Filter by score

Play Buttons

Metric Report

(Confidence, Purity, Cost)

06-081

work has raised many questions resulting in few answers
Work has Raised Many Questions … Resulting in Few Answers
  • Where do groups, events, activities fit in?
    • Can we not track a group, an activity (Why only Objects?)
    • Is a group or activity only a complex object?
  • What is a Situation? Is there more than one? Is it Context-based?
  • Where does Knowledge Discovery exist? Forensics?
  • What is Situation Assessment?
  • Is Threat Assessment only of the future – what about current threat?
  • What about forecasting or projecting the “future” state?

No one model answers ALL of these questions and

even addresses them!

07-210

so then what
…so Then What
  • Treating Situation as a composite of activities and tracking activities as complex objects allows for a “cleaner” distinction between fusion levels
      • Situation(s)-> Activity(s) -> Group(s)/Entity(s) -> Event(s): These are ALL OBJECTS THAT CAN BE TRACKED
      • Object Assessment has really been performing Tracking & Identification – LET’S TRACK ALL TYPES OF OBJECTS
  • Knowledge Discovery and a priori knowledge necessary and integral to building “complex” objects (e.g., Groups, Activities)
      • Updating knowledge/relationships (models) is continuous and part of refinement process
  • Define Situation Assessment based on Jean Roy’s Definition for Situational Analysis:
      • Behavior Analysis – Activity Level Analysis
      • Intent Analysis – Salience Analysis
      • Capacity/Capability Analysis – Impact Analysis
      • Threat Analysis

07-210

slide16
…and
  • Use Time to distinguish between JDL Level 2 and 3 as does Endsley’s comprehension and projection
        • Same analysis is done for both levels only difference is time
        • Thus JDL Level 2 is assessment of “current situation and JDL Level 3 is the assessment of the current situation projected forward in time.
  • Process Refinement involves not only sensor movement/collection (sensor management) BUT fusion algorithm management (which algorithms and which parameters to use) and model management from ALL processes. Possible sources to refinement include:

L1: Prediction where object is moving/next event

L2: Missing data, increase certainty of current assessments

L3: Forecasted actions/placement to pre-position sensors

07-210

slide17

Revised Situational Awareness Reference Model (Based on Previous Suggestions)

Level 1: Object Tracking

and Identification

Level 3: Assessing the

Forecasted Situation(s)

Level 2: Assessing the

Current Situation(s)

*Based on JDL, Endsley’s, and Jean Roy’s work

07-210

wrap up
Wrap Up
  • We proposed a revised Reference Model that includes many of the lessons learned to date
  • Plans are to continue to apply this revised model to Air, Cyber and Space Situation Awareness – UNIVERSAL SITUATION AWARENESS
        • …with emphasis on current and forecasted situation assessment

07-210