OmniUmpack: Fast, Generic, and Safe Unpacking of Malware

1. OmniUmpack: Fast, Generic, and Safe Unpacking of Malware Authors: Lerenzo Martignoni, Mihai Christodorescu and Somesh Jha Computer Security Applications Conference 2007 (ACSAC’07) Presented by LIU Limin

2. Agenda • Introduction • Design • Implementation • Evaluation • Conclusion

3. Agenda • Introduction • Design • Implementation • Evaluation • Conclusion

4. Packed malware • Term packed refers to malware whose payload is either compressed or encrypted. • E.g. polymorphic malware • An unpacking routine that restores the original code from memory and then executes it during runtime. • Self-decompression or self-decryption • Several layers of unpacking

5. Why malware packed? • Anti-analysis • Reverse engineering • Experts can make the malware hard to analyze • Anti-virus tools evasion • Existing packers are modified to evade anti-virus tools at the rate of 10-15 per month. • 80% new malware are packed with various packers. • 50% new malware samples are repacked versions of existing malware. How to effectively recover the real malicious code?

6. Traditional unpacking approaches • Algorithmic unpacking • Using specific unpacking routines to recover the original program, one routine per packing algorithms. • ClamAV • Limited to a fixed set of known packers.

7. Traditional unpacking approaches (Cont.) • Generic unpacking • Emulation/tracing of the execution until the unpacking routine terminates. • PolyUnpack ([ACSAC 06]), Renovo ([Worm 07]) • Limitations • Unpacking is slow because of instruction-level tracing. • Effectiveness depends on the fidelity of the emulation environment. • The detection of the termination of the unpacking routine is undecidable.

8. Goals of OmniUnpack • Generic unpacking with low-overhead by tracking memory access at the page level. • Precise unpacking by running the program on native OS. • An in-memory malware detection strategy independent of packing and self-modification.

9. Agenda • Introduction • Design • Implementation • Evaluation • Conclusion

10. OmniUnpack: simple strategy • Tracking all memory writes and the program counter. • The execution of a previously written memory locations indicates the end of an unpacking stage. • All written-then-executed (or written and about to be executed) memory locations should then be analyzed by a malware detector. Starting from this strategy, designing an unpacking algorithm that achieves low overhead yet does not compromise the security of the system.

11. Execution traces for a packed executable • Ideally • The detection of the termination of the unpacking routine is undecidable. • Coarse-grainedmemory access tracking • usinghardware mechanism at page level. • Written-then-executed pages are indicative of unpacking but not indicative of the end of unpacking. • Increasing the chance to detect spurious unpacking stages (up to hundreds of thousands stages) . • The overhead introduced by invoking the malware detector every time a written memory page is executed is prohibitive.

12. OmniUnpack-enhanced detection • Heuristic: An unpacking stage is completed when the execution of previous written page is followed by a dangerous system call. • Choice of dangerous system calls • A dangerous system call is a system call whose execution can leave the system in an unsafe state.

13. Continuous monitoring of the execution • OmniUnpack implements a continuous monitoring approach, where the execution is observed in its entirety. • Multiple unpacking stages • Approximation of the end of an unpacking stage • A solution for the undecidability of the problem of detecting the real end of the unpacking stage. • The low overhead of OmniUnpack allows for continuous monitoring in an end-user environment.

14. Algorithm

15. Example • Execution Trace : <x(0), w(1), s1, w(2), x(1), s2, x(2), s3, …> • x(0): W = Ø, WX = Ø • w(1): W= {1}, WX = Ø • s1 (NtOpenFile): W = {1}, WX = Ø • w(2): W = {1,2}, WX= Ø • x(1): W = {1,2}, WX= {1} • s2 (NtOpenKey): W = {1,2}, WX= {1} • x(2): W = {1,2), WX = {1,2} • s3 (NtDeleteFile): The system call is dangerous, malware detector is invoked to scan all the memory pages in W. s1 is not dangerous and WX is empty. s2 is not dangerous.

16. Agenda • Introduction • Design • Implementation • Evaluation • Conclusion

17. Architecture • Kernel driver • Tracking memory access • Detecting when the dangerous system call occurs • Triggering malware detection • User-space Component • Malware detector responsible for on-demand scanning of memory of the suspicious program.

18. Monitoring memory accesses • The W⊕X policy is enforced on the memory pages of the suspicious program. • A memory page can be either writable or executable. • Page-fault exceptions are trapped by OmniUnpack. • When an unpacking routine writes the unpacked code to memory, the destination page is marked writable but not executable. • When an unpacking stage ends, the access to this page for execution will cause a protection exception because of the lack of execution permission. • Intercept and process such exceptions. • Non-executable pages can be emulated via software mechanism with a minimal overhead. • PaX PAGEEXEC

19. Monitoring system calls • Intercepting the system calls of interest (dangerous ones) through interposition on the system-call dispatch table. • If newly generated code is present in the program memory, the code is executed, and the execution is followed by a dangerous system call, then consider the unpacking stage concluded and trigger malware detection. • At the end of scanning, write permissions are revoked from all scanned pages.

20. User-space malware detector • Any malware detection strategy can be used to scan the code generated during the previous stage. • OmniUnpack only provides a detector with data to analyze and then authorizes, or denies, the execution of such data according to the response of the detector. • A bug in the malware detector will not compromise the stability of the whole system since the detector is running in user-space.

21. Agenda • Introduction • Design • Implementation • Evaluation • Conclusion

22. Experimental evaluation • Successfully monitored all the analyzed samples and detected the end of each unpacking stage. • Handled 80% of packed malware, while ClamAV handled only about 15%. • Armadillo and CExe unpack the original payload on disk and then execute this executable. • N/A: programs that did not execute correctly after packing, with or without OmniUnpack. • IMPL: a limitation of the current OmniUnpack implementation.

23. Time to unpack • OmniUnpack was significantly faster than PolyUnpack. (e.g., with UPX OmniUnpack was 20.56 times faster than PolyUnpack.) • In 42% of the cases PolyUnpack timeout (300 secs). • OmniUnpack was within one order of magnitude of ClamAV (e.g., with UPX OmniUnpack was only 5 times slower). • In 75% cases, ClamAV failed to unpack.

24. Overhead for benign programs • About 6% average overhead on non-packed applications. • About 11% average overhead on packed applications (with UPX).

25. Agenda • Introduction • Design • Implementation • Evaluation • Conclusion

26. Conclusion • A new unpacking technique that addresses the shortcomings of previous unpackers. • Generic unpacker • Able to deal with known and unknown unpacking algorithms. • resilient to anti-debugging, anti-VM and anti-emulation techniques. • Integrated with any operating system and any malware detector without trading off efficiency or unpacking capability. • Suitable for use on end-user environment. • Minimizing the overhead by using hardware mechanism and working directly in the OS. • Safe in the presence of multiple unpacking stages. • Improved detection rate. • Any malware detector can delegate to OmniUnpack the recovery of the packed code.

27. Limitations • Imprecision of page-level tracking. • The memory monitoring at page level caused the undecidability of the specific addresses which are really accessed on the page. • The assumption that memory accesses potentially affect all addresses on a page will cause the identification of spurious unpacking stages. • Incomplete extracted instructions. • The content of written-then-executed page is modified before invoking a dangerous system call.

28. Thank you! Questions?

29. PolyUnpack ([ACSAC 06]) • Performing static analysis over malware instance to acquire its static code model. • The statically derived model and malware instance are fed into the dynamic analysis component. • The execution of malware paused after each instruction and its context is compared with the model. • When the first instruction of a sequence not found in the static model is detected, representations of that unknown instructions sequences are written out and the malware’s execution is halted.

30. Renovo ([Worm 07]) • Heuristic: the original program code and data should be present in memory to be executed, and also the instruction point should jump to the OEP of the restored program code which has been written in memory at runtime. • A technique dynamically extract the hidden original code and the OEP from the packed executable by examining whether the current instruction has been generated at run-time, after the program binary was loaded. • Instruction level.