Host-based Intrusion Detection (Work in Progress)

1. Host-based Intrusion Detection(Work in Progress) The SOS Group School of Computing and Information Sciences Florida International University

2. Talk outline • Background on intrusion detection • Current Approaches • Challenges in Intrusion Detection • Holistic Intrusion Detection • Philosophy • Design & Implementation • Open Questions

3. Intrusions • Typically occur as sequence of operations • Exploit vulnerabilities in software • Operations affect various subsystems (e.g., memory, network, or filesystem) • A seemingly normal operation can be part of a intrusive operation sequence • Intrusions result in system compromise • Unauthorized access to system resources • Exploitation domains: • User • Root

4. Intrusion Detection • Detection generation • Current and/or Past intrusions • Detection point • “At entry” : Vulnerability exploited (Legal entry?) • “Post entry”: Ongoing misuse, unusual user or process behavior • Detection scope • User / Root domains • Detection automation • Human initiated, Human aided, Fully automated, etc. • Detection accuracy • Metrics: false+/- rates

5. Current Approaches • Subsystem-centric • E.g., Snort [Roe99], Bro [Pax99], Tripwire [KS94], Stackguard [CMH+98], etc. • Handicapped by local view of system state • Detection inaccuracy • Incorrect inferences about system status • Increased false+ and false- • E.g. False+: /bin/ls is modified => Rootkit • E.g. False- : rhupdate contacts unlisted server • Detection scope and automation • Too broad scope encompassing user and root domains • Human initiated [KC05] • Detection technique • Vulnerability-specific entry point triggers • Post-intrusion anomaly detection

6. Challenges in Intrusion Detection • Global state monitoring • Subsystem activity correlation • Generic activity signatures • Infinite good-set ? • User-specific behavior modeling • Framework for combining malicious and anomalous activity detection • Detection accuracy • Detection efficiency

7. Approach Philosophies • Holistic System View • Division of responsibility with subsystem sensors • Low-overhead correlation and signature detection • Ignore User, Protect ROOT! • Monitor everything • Detect root domain compromise only • Two-Level Detection • Perimeter detector to monitor entry-points • Internal detector to monitor system misuse

8. I. Holistic System View Process Global System State Memory Filesystem Network

9. Subsystem Events • Network activity • E.g., Connection established, Connection bandwidth exceeds threshold, Connection closed, etc. • Filesystem activity • E.g., File is opened, Permissions/Owner changed, Symbolic link created, Write/Read performed, etc. • Process activity • E.g., Process created, Executable loaded, Process defunct, Process terminated, EUID changed, etc. • Memory activity • E.g., Memory allocated/deallocated, Protected memory overwritten, etc.

10. Detection Errors • False - • /etc/passwd modifed (by non-root process) • rhupdate contacts unlisted server for updates • TOCTTOU exploit • False + • rhupdate modifies ls • Unusual network activity (backup process)

11. Holistic View • Subsystem Sensors: Monitor low-level events occurring within each subsystem • Filter and aggregate into high-level semantic events • Escalate high-level events to the central analyzer • Sensor policies can be tuned by central analyzer • Allow integration of third-party subsystem profilers with standard interfaces • Central Analyzer: Sink for high-level events • Integrates and correlates them • Tracks global system state • Uses both good and malicious signatures for intrusion detection

12. Holistic Detection Architecture Process Sensor Central Analyzer Memory Filesystem Sensor Sensor Sensor Network

13. Terminology • Event: Any significant occurrence • Low-level • High-level • Activity: Set of ordered sequence of events • Signature: Generic activity (with wildcards) • Signatures database: Collection of signatures • Malicious • Good

14. High-level Events • High-level event: [start:end] [subsystem] [pid] [event-type] <sub-type> <attr-list> • Examples: • 21000:21019 P 666 Start backup SYSTEM • 38450:38900 P 25 Start rhupdate SYSTEM • 39453:39454 P 46 Defunct 46 httpd SERVER • 40039:41001 F 25 Access /bin/ls W • 45233:52001 F 562 Delete /var/log/messages • 54883:54889 N 25 ConnEst TCP 128.111.58.143 • 65900:66005 N 666 LdSpike TCP 152.41.20.1 • 70001:70010 M 46 Write RA_STACK 0xF04986FF 0xF04986AF • 89000:89001 M 46 Write CODE_SEG 0xFF98754A 551

15. Activity • Complex co-operating events from multiple subsystems • Can be simplified into (one or more) event sequences • E.g. False+ 38450:38900 P 25 Start rhupdate SYSTEM 40039:41001 F 25 Access /bin/ls W 45091:45092 P 25 Stop

16. Signatures • Generic activity pattern • Malicious signatures database • Infer malicious activity • Increase detection accuracy (Reduce False-) • Good signatures database • Infer non-malicious activity • Increase detection accuracy (Reduce False+) • Example

17. II. Ignore User, Protect ROOT! • Intrusions can effect two domains • User • Root • Approach: Ignore user, protect root! • Too many application-specific vulnerabilities • New applications introduce new entry-points • Users make mistakes: cannot be controlled! • No uniform user behavior model (also application dependent) • User compromise does not affect other users or system • OS must implement fairness and it does (mostly) • Significantly reduced signature setand detection overhead

18. III. Two-Level Detection • Internal operation of the Central Analyzer • Detector • Input: Global event log • Perimeter Detector: Monitor entry points into system (i.e. vulnerabilities) for compromise • Bad signature DB entries • Vulnerability-generic class signatures • What about? • Unknown vulnerabilities • Entry through legitimate entry point (careless admin?) • Internal Detector: Monitor for misuse/anomalies • Good signature DB entries • Potentially infinite • Only register: Apparently bad, but Actually OK.

19. Perimeter Detector: Vulnerability-generic bad signatures Feedback (backtracking) Internal Detector: Misuse-specific good signatures Legend: Known vulnerability Unknown vulnerability Two-Level Detection Framework

20. Perimeter Detector • Entry points are system vulnerabilities • Class 1: Improper protection • Incorrect initial assignment (incorrect permissions, config, etc.) • Improper isolation of implementation detail (bypass OS) • Improper change (TOCTTOU) • Improper naming (symbolic link ambiguity) • Improper allocation/de-allocation (old data reused) • Class 2: Improper validation • E.g., memory overflows, format string, type clashes • Class 3: Improper synchronization • Improper indivisibility (atomicity, cache consistency) • Improper sequencing (writing during reading, etc.) • Class 4: Improper choice of operand or operation • E.g., Not resetting GID allowing read of kernel memory

21. Perimeter Detector • Example • Vulnerability: TOCTTOU • Bad signature (p:pid, f:filename): F p Access f F !p Delete f F !p SymLink * f F p {Rd,Mdfy} f

22. Internal Detector • Monitor system resource usage • Need only monitor root activity • Good signature: Apparently BAD, but OK! • Policy framework for capturing Good Root behavior • Protected resource, Authorized set (users, executables) • Must be administrator tunable • Password protected console interface • Trigger Events • Access to protected resource • Check good DB for eliminating false + • E.g. Good Signature P X Start UpdateClass:A(Y) SYSTEM F X Access ProtectedClass:Y W (Trigger Event) P X Stop

23. Open Questions (?) • Signature generation • Source: public signature db, sysadmin (?) • Automation (from specific activity) • Generality • Monitoring overhead • Analysis overhead • Intrusion response • Automatic vulnerability removal (?)

24. Faculty Dr. Raju Rangaswami Dr. Geoff Smith Dr. Masoud Sadjadi Dr. Tao Li Students Krista Merrill Joseph Marrero Igor Hernandez Mandy Barsilai Ruben Duque The SOS Group @ FIU

25. Thank You!

26. Back-up Slides