Formalizing Sensitivity in Program Models for Intrusion Detection

1. Formalizing Sensitivityin Program Modelsfor Intrusion Detection Jonathon T. Giffin Somesh Jha Barton P. Miller University of Wisconsin {giffin,jha,bart}@cs.wisc.edu Wenke Lee Georgia Instituteof Technology wenke@cc.gatech.edu Henry Hanping Feng Yong Huang University of Massachusetts {hfeng,yhuang}@ecs.umass.edu

2. Important Ideas • Formalizing program models facilitates understanding & comparison. • Exposing additional program state improves monitoring speed & model accuracy. • VPStatic model: reads program’s call stack • Dyck model: instruments binary code Henry Feng & Jonathon Giffin

3. Model-Based Intrusion Detection Build model of correct program behavior Model: automaton specifying all valid system call sequences Runtime monitor ensures execution does not violate model User Process Operating System Henry Feng & Jonathon Giffin

4. Model-Based Intrusion Detection Model must be fast to operate Model must accurately represent program Context-sensitive models restrictimpossible paths User Process Operating System Henry Feng & Jonathon Giffin

5. Code Example char *filename; pid_t[2] pid; int prepare (int index) { char buf[20]; pid[index] = getpid(); strcpy(buf, filename); return open(buf, O_RDWR); } getpid open Henry Feng & Jonathon Giffin

6. Code Example void action (void) { uid_t uid = getuid(); int handle; if (uid != 0) { handle = prepare(1); read(handle, …); } else { handle = prepare(0); write(handle, …); } close(handle); } getuid prepare prepare read write close Henry Feng & Jonathon Giffin

7. NFA Model getuid getpid prepare prepare open read write close Function action Function prepare Henry Feng & Jonathon Giffin

8. NFA Model getuid getpid open read write close Function action Function prepare Henry Feng & Jonathon Giffin

9. Impossible Path Exploit void action (void) { uid_t uid = getuid(); int handle; if (uid != 0) { handle = prepare(1); read(handle, …); } else { handle = prepare(0); write(handle, …); } close(handle); } getuid prepare prepare read write close Henry Feng & Jonathon Giffin

10. PDA Model push X getuid getpid push Y pop X open pop Y read write close Function action Function prepare Henry Feng & Jonathon Giffin

11. PDA Problems X • Impossible paths still exist • Non-determinism indicates missing execution information • PDA run-time state explosion • ε-edge identifiers maintained on a stack • Stack non-determinism is expensive • post* algorithm: cubic in automaton size push X getuid getpid push Y Henry Feng & Jonathon Giffin

12. Determinize PDA • Expand the input alphabet by exposing the stack operations and the target state of the transition • fa,p,z indicates consume input a, push z on the stack, and transition to state p. • ga,p,zfor pop operations. • ea,pfor operations with no stack activity. • Result in a Deterministic PDA (or DPDA). • Exposing only stack operations we get PDA with deterministic stack operations (or sDPDA). Henry Feng & Jonathon Giffin

13. NFA State non-determinism is cheap. State non-determinism unlink unlink Henry Feng & Jonathon Giffin

14. Non-Deterministic PDA Stack non-determinism push X push Y Stack non-determinism is expensive. State non-determinism unlink unlink Henry Feng & Jonathon Giffin

15. Deterministic PDA (DPDA) Stack non-determinism push X push Y VPStatic Model • Model exposes stack operations & target states • Possible exponential increase in model size? State non-determinism unlink unlink Henry Feng & Jonathon Giffin

16. Stack-Deterministic PDA (sDPDA) Stack non-determinism push X push Y Dyck Model • Model exposes stack operations • No increase in model size? State non-determinism unlink unlink Henry Feng & Jonathon Giffin

17. Input Symbol Processing Complexity • n is state count • m is transition count • k is PDA input alphabet size • r is PDA stack alphabet size Henry Feng & Jonathon Giffin

18. VPStatic • A variant of VtPath model. • DPDA. • Generated by static analysis of binary. • Use Addr(state) to expose states. Henry Feng & Jonathon Giffin

19. Determinizing via Observation • Extract return addresses from call stack into virtual stack list for each system call. • Generate a bunch of input symbols for each system call. Henry Feng & Jonathon Giffin

20. Determinizing via Observation char *filename; pid_t[2] pid; int prepare (int index) { char buf[20]; pid[index] = getpid(); strcpy(buf, filename); return open(buf, O_RDWR); } … … Addr(C0) Addr(C1) Addr(C1) getpid open e(open,Addr(Sopen)) e(none,Exit(prepare)) g(none,Addr(C1),Addr(C1)) e(none,Addr(C0)) f(none,Entry(prepare),Addr(C0)) e(open,Addr(Sopen)) Henry Feng & Jonathon Giffin

21. Dyck Model [X push X getuid getpid [Y push Y pop X open ]X ]Y pop Y read write close Henry Feng & Jonathon Giffin

22. Dyck Model getuid [X getpid open ]X read close getuid [Y getpid open ]Y write close • Matching brackets are alphabet symbols • Expose stack operations to runtime monitor • Language of bracket symbols is a Dyck language • Rewrite binary to generate bracket symbols Henry Feng & Jonathon Giffin

23. Determinizing via Binary Rewriting Insert code to generate bracket symbols around function call sites Notify monitor of stack activity Determinizes stack operations void action (void) { uid_t uid = getuid(); int handle; if (uid != 0) { precall(X); handle = prepare(1); postcall(X); read(handle, …); } else { precall(Y); handle = prepare(0); postcall(Y); write(handle, …); } close(handle); } Henry Feng & Jonathon Giffin

24. Dyck Model Dyck model stack-determinizes PDA Stack non-determinism Stack deterministism push X push Y [X [Y Only one valid stack configuration push X push Y Henry Feng & Jonathon Giffin

25. Test Programs Henry Feng & Jonathon Giffin

26. Henry Feng & Jonathon Giffin

27. Henry Feng & Jonathon Giffin

28. Questions? • Henry Hanping Feng, Yong HuangUniversity of Massachusetts—Amherst {hfeng,yhuang}@ecs.umass.edu • Jonathon T. Giffin, Somesh Jha, Barton P. MillerUniversity of Wisconsin—Madison{giffin,jha,miller}@cs.wisc.edu • Wenke LeeGeorgia Institute of Technologywenke@cc.gatech.edu Henry Feng & Jonathon Giffin