Effective and Efficient Malware Detection at the End Host

1. USENIX Security Symposium ‘09 Effective and Efficient Malware Detection at the End Host Clemens Kolbitsch, Paolo MilaniComparetti @ TU Vienna Christopher Kruegel @ UCSB EnginKirda @ Institute Eurecom Xiaoyong Zhou, XiaoFeng Wang @ Indiana Univ. at Bloominton

2. Outline • Motivation • System Overview • System Details • Evaluation • Limitation • Conclution

3. Effectiveness & Efficiency Motivation

4. Motivation • Efficiency • Binary signature based detection • Network-based detection • Effectiveness • Behavior-based detection • Detection based on malware's behavior • Behavior is hard to obfuscate • Behavior is hard to randomize • Behavior is often stable across various malware version

5. Motivation • This Paper proposes… • A behavior-based solution with Efficiency • For end hosts

6. Modeling Behaviors and Making detection efficient System Overview

7. System Overview • Malware behaviors • Manifest on system (i.e., survive reboot) • (Over-) write system executables, dlls, files • Create registry entries • Register as Windows (startup) service • Conceal from being detected • Restart under some stealthy name (e.g., svchost.exe) • Inject into legitimate processes • Replicate • Send emails • Copy to Samba shares, USB drives, etc. • Scan and exploit services on LAN or WAN

8. System Overview • Detection based on execution characteristics • Execute malware in full system emulator (Anubis) • Monitor interaction with the operating system • Perform detailed taint analysis • Generate detection graphs • Describe sequence of required system calls leading to security relevant system activity • Include dependencies to related, previous calls (using taint dependencies) • Detect described behavior on end host • Log system call activity of unknown executable • Match against behavior graph

9. System Overview • Example: Agent (trojan) • As part of its system manifestation, it • Reads content from binary image • Decrypts binary content • Proprietary decryption routine • Simple, XOR based algorithm • Stores binary in system file (C:\Windows\system32\drivers\ip6fw.sys) • Later, restarts IPv6 firewall • Turns itself into a system service

10. System Overview

11. Generate Behavior Graphs, Match Behavior Graphs System Details

12. System Details • Behavior graphs • Directed acyclic graph • Node: system calls • Edges: dependencies • Dependencies • Handle dependencies • Direct value propagation • System provided identifiers • Must be constant

13. System Details • Data dependencies • Arbitrary data (& control) dependency between system calls • Might modify values between system calls

14. System Details • Generate behavior graphs • Analyze executable in Anubis sandbox • Obtain instruction level log • Obtain program flow log • Obtain memory access log • Generate precise taint propagation trees • Data/control dependencies • Instructions that access/generate tainted data • Link system calls consuming data with all taint generating calls (sources)

15. System Details • Generate behavior graphs (cont.) • Scan logs for security relevant behavior • Provided with a list of interesting system calls • Extract propagation formulas

16. System Details • Match behavior graphs • Active(inactive) node • Simple(complex) function • Security-relevant system calls or the Buttom • Confirmed(deactivate all)

17. System Details

18. System Details

19. Effectiveness, Efficiency evaluation

20. Evaluation • Effectiveness

21. Evaluation

22. Evaluation • False Positive • IE, Firefox, Thunderbird, putty, notepad • 0

23. Evalution • Efficiency

24. Limitation & conclusion

25. Limitation • Evading signature generation • Detect the virtual environment • Delays, time-triggered behavior • Modifying the algorithm behavior

26. Conclusion • Behavior can be detected • Behavior detection is fast enough for end hosts • Approach intrinsically robust against polymorphism and metamorphism • To some extent, behavior graphs are usable across malware variants