Report on Intrusion Detection and Data Fusion

1. Report on Intrusion Detection and Data Fusion By Ganesh Godavari

2. Outline of the talk • Intrusion Detection • Data fusion • Motivation • Traditional models

3. Intrusion Detection & Data Fusion • Intrusion Detection System • Protect availability and provide confidentiality and integrity of critical information infrastructures • Data Fusion : task of data processing aiming at making decisions on the basis of distributed data sources specifying an object • Data sources • Different physical nature • Electromagnetic signals, sensor data… • Different accuracy • Reliability?

4. Motivation & challenges • Threat analysis • Known & unknown Pattern templates, traffic analysis, statistical-anomaly detection and state based detection • Provide Reliability • Reduce false alarms, increase user confidence

5. Characteristics of IDS based on Waltz model • Detection performance • Detection characteristics like false alarm rate, detection probabilities and ranges for an intrusion characteristic • Spatial/temporal resolution • Ability to distinguish between two or more intrusions in space and time • Spatial coverage • Span of coverage or field of view of the sensor • Detection and Tracking modes • Mode of operation of the sensor i.e. staring or scanning; single or multiple target tracking • Target Revisit Rate • Rate at which an intrusion is revisited by the sensor to perform measurements • Measurement Accuracy • Statistical probability that the measurement or observation is accurate • Measurement dimensionality • Number or measure of variables between target categories

6. Contd.. • Hard Vs. Soft Data Reporting • Status of the sensor reports • can a decision be made without correlation or does the sensor require confirmation • Detection/Tracking Reporting • Characteristic of the sensor to report individual events or maintain a time-sequence of the events or events

7. Hierarchy of IDS Data Fusion Inferences Threat Analysis Situation Assessment Behavior of Intruder Identity of Intruder Rate of Intrusion Existence of intrusion High Medium Low Level of Inference Types of Inference

8. Data fusion and OODA model • Decision support systems and data fusion system need to be tightly coupled • Decision support system must • Observe • Collection of data from sensors, network sniffers, system log files • Orient • Data mining concepts of learning unknown characteristics. • Decide • Refinement of knowledge into threat knowledge and determination of appropriate counter measures • Act • Automated and human responses to threat/vulnerability

9. OODA mapping • Three levels of abstraction • Data • Measurement and observations • Information • Data placed in context, indexed and organized. • Knowledge/intelligence • Information explained and understood

10. Intrusion Detection Data Fusion Situational knowledge used for Analyzing objects and groups against existing Intrusion detection templates to provide assessment This ID model is based on deductive process used to detect previously known patterns in many sources of data Data is correlated in time Data is assigned weighted Metrics based on relative importance Alignment to a common frame of reference Calibration and filtering Observation identifiers, time of observation, and description Correlation between level 3 threat assessment and security Policy and objectives determine the implications of current Situation base. The whole process is refined via level 4 resource Management based on situational awareness

11. notes • Situational data is collected from sniffers and other ID sensors with primitive observation identifiers, time of observation and descriptions. This raw data requires calibration or filtering known as level0 refinement. All the three measurements must be aligned to a common frame of reference. This alignment is known as level1 object refinement. Here data is correlated in time and data is assigned weighted metrics based in relative importance. Observation may be associated and paired and placed in context in an information base. Situation refinement provides situational knowledge and awareness. Situational knowledge is used to analyze objects and aggregated groups against existing intrusion detection templates to provide assessment of the current situation and suggest or identify future threat attacks. Correlation between level3 threat assessment and security policy and objectives determine the implications of the current situation base. The entire process is refined via level 4 resource management based on situational awareness.

12. Technical terms !! • Data mining/knowledge discovery : search for hidden patterns based on previously undetected intrusions to help develop new detection templates • Data fusion Vs data mining • Inference method and temporal perspective

13. Intrusion detection data mining

14. notes Raw data from relevant network management and intrusion detection systems are collected and indexed in the data warehouse. Major Technical issue is how to reconcile the raw data from many different formats and inconsistent data definitions.

15. Process involved in intrusion detection data mining • Data cleansing • check to insure the collected data is in correct ranges and limits • evaluate overall consistency of the data • ensure hierarchical relationship exists • Data selection and transformation • Initial sets that will be used for data mining are selected • Data mining • Performed on selected data sets in either manual or automated modes

16. Data mining operations characterized by waltz • Clustering • Data is segmented into subsets that share common properties • Association • Analysis of both the cause and effect and structure relationship between data sets • Statistical Analysis • Determine the likelihood of characteristics and association in selected data sets • Rule Abduction • Development of IF-THEN-ELSE rules that describe associations, structures and test rules • Link or tree abduction • Performed to discover relationships between data sets and interesting connecting pattern properties • Deviation Analysis • Locate and analyze deviations from normal statistical behavior • Neural Abduction • Process of training artificial neural networks to match data, extract node weights and structure (similar to abducted rule sets)

17. Intrusion detection data mining contd.. • Discovery modeling • Information is mined into new ID knowledge • Development of refined models to predict future events based on historical data • Visualization • human process of pattern recognition

18. Questions ?

19. References • Intrusion detection systems and multi sensor data fusion: creating cyber situational Awareness by Tim Bass Communications of the ACM (2000)