Intrusion Detection via Static Analysis

1. Intrusion Detection via Static Analysis David Wagner Drew Dean

2. Motivation • New security problems are discovered every day • Majority of security problems are buffer overflows (foreign code is executed) and not logic errors (native application code continues to execute) • Try to solve this by monitoring applications for any abnormal behavior

3. Framework • Assumption: A compromised application cannot cause much harm unless it interacts with the underlying operating system, and those interactions may be readily monitored. • In most of the cases, the only way to interact with an OS is via system calls. • Solution: Monitor application’s system call trace for any unexpected interaction with an OS.

4. Trivial Model • Create the set of system calls that the application can ever make • If a system call outside of the allowed set is executed, terminate the application • Pluses: simple, easy to implement, efficient • Minuses: Fails to detect many attacks (i.e. ones that use only system calls from the allowed set), too coarse-grained (certain system calls can cause a lot of damage, i.e. open() )

5. Callgraph Model • Improves the trivial model by reintroducing the ordering of the system calls • Represent the system call trace as a non-deterministic finite automaton (NDFA) • Monitor the application by simulating the operation of the NDFA on the observed system call trace • Pluses: more precise than the trivial model, does not introduce any false positive alarms • Minuses: harder to implement, not efficient, includes impossible paths due to function call treatment and presents certain risks due to non-determinism of the model

6. Abstract Stack Model • Improves the callgraph model by eliminating impossible paths (by characterizing more precisely the set of possible syscall traces) • Represent the system call trace as a non-deterministic pushdown automaton (NDPDA) • Monitor the application by simulating the operation of the NDPDA and comparing the application call stack with a list of all valid stacks • Pluses: eliminates impossible paths • Minuses: much harder to monitor the application efficiently

7. Digraph Model • Combines some of the advantages of the callgraph model in a simpler formulation • Model consists of a list of possible k-sequences of consecutive system calls (k=2 for simplicity) • Monitor the application by checking the executed system calls vs. a precomputed list of the allowed k-sequences • Pluses: much more efficient than the callgraph or abstract stack models • Minuses: less precise than the callgraph or abstract stack models

8. Implementation Issues • Non-standard control • Function pointers • Signals • Setjmp() • Other modeling challenges • Libraries • Dynamic linking • Threads

9. Optimizations • Irrelevant systems calls • Not monitoring harmless but frequently executed system calls such as brk() can greatly improve the performance • System call arguments • Monitoring the arguments at runtime improves both precision and performance

10. Evaluation • Performance • Precise callgraph and abstract stack models introduce too much overhead • Mimicry attacks • Require high precision models to detect (poor performance)

11. Unaddressed issues • Applying static analysis to binaries when source code is not available • Explaining how runtime monitoring agent works and how it can be used to improve the security and the model precision (run as a separate SMT thread with 0 overhead?)

12. Questions • What are the ramifications of the callgraph model non-determinism? • What are the risks of having k=2 in the k-sequences model?