1 / 25

An Out-of-the-Box Approach to High Assurance Computer System Monitoring and Integrity Protection

Cyber Defense Conference, Rome, NY, May 12-14, 2008. An Out-of-the-Box Approach to High Assurance Computer System Monitoring and Integrity Protection. Xuxian Jiang. Dongyan Xu. Assistant Professor Dept. of Computer Science George Mason University. Associate Professor

chanda-olsen
Download Presentation

An Out-of-the-Box Approach to High Assurance Computer System Monitoring and Integrity Protection

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Cyber Defense Conference, Rome, NY, May 12-14, 2008 An Out-of-the-Box Approach to High Assurance Computer System Monitoring and Integrity Protection Xuxian Jiang Dongyan Xu Assistant Professor Dept. of Computer Science George Mason University Associate Professor CERIAS and Dept. of Computer Science Purdue University

  2. Outline • Motivation • “Out-of-the-box” for high assurance • New VMM component: OBSERV • New capabilities enabled • High assurance system monitoring • Stealth malware detection • External run of COTS anti-virus software • OS integrity protection against kernel rootkits • Planned work • Summary

  3. Motivation • Malware remains a top concern in cyber defense • Malware: viruses, worms, rootkits, spyware, bots…

  4. Motivation • Rootkit attack trend Viruses, worms, bots, … 700% growth 400% growth Q1 of 2005 Source: McAfee Avert Lab Report (April 2006)

  5. Why Going “Out-of-the-Box”? • State-of-the-art: Running high-assurance modules (e.g., anti-virus systems) inside the monitored system • Advantage: They can see everything (e.g., files, processes…) • Disadvantage: They cannot see anything! IE Firefox VirusScan … OS Kernel

  6. VirusScan IE Firefox … OS Kernel Why Going “Out-of-the-Box”? • Fundamental flaw in current practice • Malware and malware defense running in the same system space at the same privileged level • No clear winner in this “arms race” • Solution: Going “out-of-the-box” ? Virtual Machine Monitor (VMM)

  7. VirusScan The “Semantic-Gap” Challenge Semantic Gap Guest OS Virtual Machine Monitor (e.g., VMware, Xen) • What we get: • Low-level states • Memory pages, disk blocks… • Low-level events • Privileged instructions, • Interrupts, I/O… • What we want: • High-level semantic states • Files, processes… • high-level semantic events • System calls, context switches…

  8. Our Solution: OBSERV • OBSERV: “Out-of-the-Box” with SEmantically Reconstructed View • A new component missing in current VMMs IE Firefox … OS Kernel OBSERV Virtual Machine Monitor (VMM)

  9. In-the-box View OBSERV View IE Firefox … OS Kernel OBSERV Virtual Machine Monitor (VMM) New Capabilities Capability I: High-assurance system logging Diff Capability II: Malware detection by view comparison Capability III: External run of COTS anti-virus software Capability IV: OS kernel integrity protection

  10. OBSERV: Bridging the Semantic Gap • Step 1: Procuring low-level VM states and events • Disk blocks, memory pages, registers… • Traps, interrupts… • Step 2: Reconstructing high-level semantic view • Files, directories, processes, and kernel modules… • System calls, context switches… VM Introspection Guest View Casting

  11. Step 1: VM Introspection VM disk image VM physical memory VM hardware state (e.g., registers) VM-related low-level events (e.g., interrupts) VMware Academic Program

  12. Step 2: Guest View Casting Semantic Gap Guest OS OBSERV Virtual Machine Monitor (VMM) Key observation: The guest OS provides all semantic “templates” of data structures and functions to reconstruct VM’s semantic view

  13. Guest View Casting Device drivers, file system drivers VM disk image Memory translation, task_struct, mm_struct VM physical memory Syscalls, context switches, .... VM-related low-level events (e.g., interrupts) Event semantics VM hardware state (e.g., registers) CR3, MSR_SYSENTER_CS, MSR_SYSENTER_EIP/ESP Event-specific arguments…

  14. Guest View Casting on Memory State Process List Process Memory Layout

  15. IE Firefox … OS Kernel OBSERV Virtual Machine Monitor (VMM) OBSERV Capability I Capability I: High-assurance system logging Demo X. Jiang, X. Wang, "'Out-of-the-Box' Monitoring of VM-Based High-Interaction Honeypots", International Symposium on Recent Advances in Intrusion Detection (RAID 2007)

  16. In-the-box View OBSERV View IE Firefox … OS Kernel OBSERV Virtual Machine Monitor (VMM) OBSERV Capabilities II and III Diff Capability II: Stealth malware detection by view comparison Capability III: External run of COTS anti-virus software X. Jiang, X. Wang, D. Xu, "Stealthy Malware Detection Through VMM-Based 'Out-of-the-Box' Semantic View Reconstruction", ACM Conference on Computer and Communications Security (CCS 2007)

  17. View Comparison for Malware Detection Experiment setup Both guest OS and host OS run Windows XP (SP2) VMM: VMware Server 1.0.1 Running Symantec AntiVirus twice Inside Outside Hacker Defender NTRootkit

  18. Internal Scanning Result Diff External Scanning Result

  19. OBSERV Capability IV: OS Kernel Integrity Protection • High-assurance OS kernel • No malicious kernel code • No kernel rootkit attacks • Two main tasks: • Tracking run-time kernel code layout • Enforcing the following properties • Only loading authenticated kernel code • Only executing authenticated kernel code R. Riley, X. Jiang, D. Xu, "Guest-Transparent Prevention of Kernel Rootkits with VMM-based Memory Shadowing", CERIAS Technical Report TR2001-146, Purdue University, 2008

  20. NICKLE: “No Instruction Creeping into Kernel Level Executed” • Step 1: Create two memory spaces • Standard memory • Shadow memory • Step 2: Authenticate and copy kernel code to shadow memory • Step 3: Memory access dispatch • Kernel code fetch -> shadow memory • All other accesses -> standard memory Guest OS VMM OBSERV NICKLE Kernel Code Kernel Code Standard memory Shadow memory

  21. Demonstration of Effectiveness Successfully preventing 23 real-world kernel rootkits!

  22. Planned Work • Porting OBSERV to hardware • FPGA, multicore, PCI card… • Research problems • Software/hardware function division • Hardware primitives/policies for high assurance • Formal verification of OBSERV capabilities • Performance optimization

  23. Summary • OBSERV enables “out-of-the-box” malware defense paradigm, bringing high assurance to • System logging and monitoring • Malware detection and prevention • OS kernel (against kernel rootkits) • We are looking for • Applications in Cyber Defense activities • Collaboration/deployment/funding opportunities

  24. Process Coloring: An Information Flow-Preserving Approach to Malware Investigation Eugene Spafford, Dongyan Xu, Ryan Riley Department of Computer Science and Center for Education and Research in Information Assurance and Security (CERIAS) Purdue University Xuxian Jiang Department of Computer Science George Mason University A related project funded by IARPA through AFRL Part of NICIAR Program

  25. Thank you! For more information: xjiang@gmu.edu, dxu@cs.purdue.edu http://www.cs.gmu.edu/~xjiang http://friends.cs.purdue.edu

More Related