TxBox : Building Secure, Efficient Sandboxes with System Transactions

1. Suman Jana The University of Texas at Austin Donald E. Porter Stony Brook University VitalyShmatikov The University of Texas at Austin IEEE Security & Privacy 2011 TxBox: Building Secure, Efficient Sandboxes with System Transactions

2. Outline • Introduction • Building a Better Sandbox • Related Work • System Transactions • Design and Implementation • Evaluation • Limitations • Conclusions

3. Introduction • ACID: atomicity, consistency, isolation, and durability [wiki] • TxBox: • Speculative execution of untrusted application • Uncircumventable enforcement of system-call policies • Automatic recovery from the effects of malicious execution

4. Introduction • TxBox components: • Grouping • Rollback • Access summary • Access check • Based on TxOS[link] • Linux with support for system transactions • Prevent race conditions

5. Building a Better Sandbox • TOCTTOU example Public file 1. Link to public file Softlink Private file 2. Change to private file System-call monitor must know all possible system-call sequences that lead to a violation and the monitor must check every call ! Difficult and Slow !

6. Building a Better Sandbox • TxBox is immune to TOCTTOU attacks • Sandboxed processes run inside separate transactions • Whitelist / blacklist access-control policies

7. Building a Better Sandbox • Drawback of conventional sandbox • Violation can be detected only after multiple system calls, the damage may already have been done • TxBox: • Transaction is aborted, and the system automatically reverts

8. Building a Better Sandbox • Parallelism (Multi-core processors) • Conventional sandbox or anti-virus: • Sandboxed program makes a system call, its execution must be paused • Split-personality malware • TxBox: • Sandboxed program run with close-to-native performance while performing checks in parallel • Only a single copy is executed

9. Related Work • Speck, 2008 • Multi-core processor • Concurrently perform checks on an instrumented copy on another core • Cannot detect “split-personality” malware • Use Pin [link] • Pin can be detected by malicious program • Suffer TOCTTOU attacks

10. Related Work

11. Related Work • Software Transactional Memory (STM) [wiki] • Lock-based synchronization • STM-based systems • Enforce application-specific security policies • Cannot span system calls

12. Related Work • Capsicum, 2010 • Capability-based sandboxing system • Help benign application to increase their trustworthiness • Native Client, 2009 • Require the code to be recompiled to a restricted subset of the x86 ISA and also confines it using segmentation

13. System Transactions • ACID properties for sequences of updates to system resources • If the transaction aborted before it finish, all intermediate updates are rolled back • All accesses to the system within a transaction are kept isolated and invisible to the rest of the system until the transaction commits

14. System Transactions • TxOS: • sys_xbegin – start transaction • sys_xend – end transaction • Maintain a transaction workset • Kernel objects (inodes, etc.) • Each entry in workset contains a pointer to the stable object, a pointer to the shadow copy • information: • read-only / read-write • Methods: commit / abort / lock / unlock / release

15. Design and Implementation

16. Design and Implementation • TxBox policy • BLACKLIST • Forbidden behavior • WHITELIST • Required behavior • DENY • Lists of system calls should be denied

17. Design and Implementation • BLACKLIST and WHITELIST primitives • Regular expressions • System-call names and arguments, or objects • Objects • (I) inodes • Inode number • Mode: read or write • (S) sockets • Type: INET or UNIX • Attributes: IP address (for INET socket) or names (for UNIX datagram sockets)

18. Design and Implementation • Example • (e)* means “match expression e any number of times” • Any deviation from a model-based whitelist policy means that the sandboxed program is no longer executing the original code (code injection attacks)

19. Design and Implementation

20. Design and Implementation • DENY primitives • DENY primitives is violated • Block calls and continue execution • Run a sandboxed program without letting it perform certain operations

21. Design and Implementation • Policy enforcement • Character device driver • Send policy and pid to monitor • Only communicate with root

22. Design and Implementation • System-call interposer • TxBox call wrappers • Symbolic link issue • rename calls • Enforcer • Force sandboxed process to run in transaction mode

23. Design and Implementation • Policy decision engine • Invoked when sandboxed process attempts to perform a critical call

24. Design and Implementation • Implementation • Based on Dazuko[wiki], by Avira • File Access Control • Can also be based on LSM • xbegin -> beginTransaction kernel function • beginTransaction checks if the current call is xbegin • Forced_transaction flag

25. Design and Implementation • Handling external I/O by sandboxed process • Bidirectional communication cannot be buffered until the end of a transaction • Conventional system-call monitors • Call-by-call • TxBox • If DENY primitive exists • return error • Check WHITELIST and BLACKLIST • Inspects the trace of the process • Terminate and roll back

26. Evaluation • Performance Test • TxBox with Linux 2.6.22.6 • Linux with Dazuko module • Quad-core Intel X5355 2.66GHz • 4GB RAM

27. Evaluation • Micro-benchmarks • Policy • BLACKLIST WREGEX *I:1234* Worst system call for TxBox

28. Evaluation • Application benchmarks

29. Evaluation • Application benchmarks • PostMark: a file system benchmark • Due to the fact that the transaction commit groups all writes and presents them to the I/O scheduler all at once, thus improving disk arm scheduling

30. Evaluation • Scalability • Opens 100 existing files • Intel Core Due 2.00 GHz with 2GB RAM

31. Evaluation Original TxBox fd = open(“foo”, ..) read(fd, ..) sockfd = socket( .. ) sendto = (sockfd, .. ) close(sockfd) close(fd) … TX BEGIN fd = open(“foo”, ..) read(fd, ..) sockfd = socket( .. ) CHECK TX POLICY VIOLATION NONTRANSACTIONAL sendto(sockfd, ..) close(sockfd) close(fd) … CHECK TX POLICY VIOLATION TX END

32. Evaluation • Non-transactional system call • sendto / reccvfrom / connect / send / recv /ioctl / read (from a socket) / write (to a socket) • IO-intensive applications • wget • System time increases by 30%-40%

33. Evaluation • Overhead for trusted applications • A single non-transactional system call • 44% overhead • Compile a non-transactional Linux kernel • Less than 2% overhead

34. Evaluation • Creating files in a protected directory • make install for vim • BLACKLIST WREGEX *I:164564* • 164564 -> /usr/local/bin/ • Correctly roll back all effects on the system

35. Evaluation • Malicious MIME handler in a browser • Tarhandler with lynx browser for MIME type ‘application/x-tar’ • Tarhandler reads file from /home/secret and writes into /tmp/foo • BLACKLIST WREGEX *I:183145* • 183145 -> /home/secret

36. Evaluation • Multimedia converter • ffmpeg: a popular codec • Create output files in the /home/user1/ • BLACKLIST WREGEX *I:181064* • 181064 -> /home/user1

37. Evaluation • Javascript engine • Google V8 benchmark • SpiderMonkeyJavascript engine • Less than 5%

38. Evaluation • On-access anti-virus scanning and parallelization of security checks • ClamAV • Small file -> the overhead of transactional mechanism dominates

39. Evaluation • With 4 threads, TxBox is 2.3 times better than Linux+ Dazuko

40. Limitations • Kernel-based security monitor is vulnerable to kernel atttacks • Transactional semantics may change the behavior of sandboxed processes • Transactional conflict • Auto-retry • Transactional state is not shared • Some anti-virus may need to observe the execution of the program

41. Limitations • Colluding malware may evade security policies • No sandboxing mechanism can reliably prevent this • Processes tha generate very large worksets are killed • DoS Attack

42. Conclusions • Use system transactions • Cannot be circumvented by TOCTTOU • Performance improvement (Multi-core)