Towards Application Security On Untrusted Operating Systems

1. Towards Application Security On Untrusted Operating Systems Authors: Dan R. K. Ports & Tal Garfinkel Course: Design of Secure Operating System Presented By: SaiUdayKiran Ravi

2. Outline • Introduction • Isolation Architectures • Implications of a Malicious OS • Attacks and Mitigations • File system • Inter-process Communication • Process Management • Time and Randomness • I/O and Trusted paths • Identity Management • Conclusion

3. Introduction • Commodity Operating Systems provides rich functionalities which leads to the system compromise. • Solutions such as NGSCB, Proxos and XOM have been proposed to enhance the assurance of the systems. But it was poorly understood about the OS services if they turn malicious. • Proposed solution: Overshadow, a virtualization based system that protects applications. • Goal of Overshadow: To maintain the secrecy and integrity of an application’s data even if the OS is completely compromised.

4. Isolation Architectures • Traditional commodity operating systems are monolithic programs that directly manage all hardware, so any OS exploit results in total system compromise. • Architectures such as NGSCB, Proxy and XOM have the basic property: Some or all of the application(s) run in a separate protection domain from the OS.

5. Isolation Architectures II • Overshadow: protects applications running inside a VM from the operating system in that VM. Mechanism behind Overshadow: • Controls memory mappings to present different views of memory pages depending on the execution context. • When an application accesses its own memory, access proceeds as normal. • If the OS or another application accesses it, the VMM transparently encrypts the memory page and saves a secure hash for integrity verification.

6. Isolation Architectures III • Shim (User level code) is added to each application at load time to support new execution environment without requiring any modification to the application. Trusted components are indicated with a shaded background. OS kernel and all other applications are untrusted. Each application is identified by a SID. (Security ID) Fig: Overshadow Architecture

7. Implications of a Malicious OS • The protection architectures guarantee that applications that do not interact explicitly with the OS execute correctly, Why ? - the integrity of code and data is protected, and control flow proceeds as normal. • However, any non-trivial program makes system calls, and this presents an opportunity for a malicious OS to influence the program’s data and control flow by manipulating system call results.

8. Implications of a Malicious OS II • Characterizing the security properties that are possible when the operating system is malicious is challenging. • Even if the OS behaves correctly, a buggy application might expose its own sensitive data (e.g. via a buffer overflow). • Problem must be ensuring that applications continue to run normally (or fail-stop) even if the OS behaves maliciously, rather than protecting application data.

9. Approaches to Ensuring Security • For programs to run normally, we must guarantee that the action performed and value returned by each system call conforms to the application’s model of how system calls behave. • Modeling all behaviors of every system call would be equivalent to re-implementing the entire OS ( nearly ).

10. Approaches to Ensuring Security II • To develop a new specification consisting only of safety properties, i.e. a model of normal OS semantics that pertains only to security, not to availability. • By weakening the specification to only provide safety properties, it becomes easier to hold the OS to its contract. • Providing weaker semantics does mean that it is impossible to guarantee correctness of arbitrary unmodified applications.

11. Approaches to Ensuring Security III • Three fundamental methods for ensuring that system calls are executed correctly: • Disallow (use of system call in security critical code, use it at own risk) • Emulate ( the system call in trusted code; re-implementing system call increases the size of TCB ) • Verify (results of system call after OS execution are verified; preferred approach) • Proposal: To implement emulation and verification using overshadows existing system call interposition mechanism, which redirects control to the shim, a trusted component, whenever an application makes a system call.

12. Attacks and Mitigations - File Systems Program code and its (potentially sensitive) data are stored on the file system. • File Contents Attack: If unprotected, a malicious OS could directly read an application’s secret data as soon as it is written to disk, tamper with application binaries, launch a replay attack by reverting file to earlier version that contains buffer overflow. Solution: For defense, Overshadow maintains protection metadata for each file which is protected by MAC and freshness counter.

13. File Systems II • File Metadata File metadata needs to be protected, including file names, sizes, and modification times. Attack: A malicious OS could perform a pathname lookup incorrectly. Even a system that protects file contents may be subverted if the OS redirects a request for a protected file to a different but still valid protected file. Solution: Using a trusted, protected daemon to maintain a secure namespace, mapping a file’s pathname to the associated protection metadata

14. Inter Process Communication (IPC) • Secured IPC is necessary. • Protecting application processes as well building block for other secure components. • Attacks: • Insecure IPC channels provided by OS . • Reason:- Malicious OS. • Consequences:-Spy, tamper, drop, delay, reorder or spoof messages.

15. Inter Process Communication (IPC) II • Direct Attack • Accessing credit card details. • Indirect Attack • Pointing on Metadata. • Example: \bin\cat to get the files details. • Solution I • Implementing trusted layer by setting message queue in VMM. • Processes could then enqueue messages or check for pending messages via secure hypercall.

16. Inter Process Communication (IPC) III • Problem:- impractical for applications to poll for messages. • This either requires waking up each process regularly, or tolerating a high message latency. Solution II • Use of Guest OS to provide asynchronous notifications • After sending though VMM, sender also sends the receiving process a signal. • Problem: only for small data transfers and need to keep copy of data in and out of VMM.

17. Inter Process Communication (IPC) IV Solution III • Shared memory regions for most of the communication, using VMM-assisted communication only for bootstrapping the secure channel. • A protected process wishing to communicate with another process in the same compartment would create a shared memory region (e.g. using mmap), and populate it with a pair of message queue. • Using Overshadows protection mechanism for memory-mapped file contents, the OS cannot read or modify the contents of the shared memory region. Problem • As OS manages namespaces so an attempt of attack on different region corresponding to a different IPC channel.

18. Inter Process Communication (IPC) V • Problem Solving of namespace can be done by using random nonce in the memory region and communicate. • Problem:- no timely ordering of messages when the OS could delay or terminate a process. • Available option of adding an acknowledgement but the OS will stop receiving the process after acknowledging. • Finally, implementation of own acknowledgement protocol can make the situation better.

19. Process Management • OS manages processes, process identities, which applications rely on for directing signals and IPC messages. Attacks: • when a new process is started OS might interfere with program execution contexts and control flow during normal operation. • Signal delivery : OS might try to redirect signals, process return values, or other information to the wrong process • Might attempt to change a process’s ID

20. Process Management II Solution: • Overshadow interposes on clone and fork system calls to set up the new thread’s initial state. This includes cloning the memory integrity hashes and thread context (including the instruction pointer), thereby ensuring that the new thread can only be started in the same state as its parent. • To ensure that signals are delivered to the correct entry point, Overshadow also maintains its own protected table of the application’s signal handlers. • Problems related to the OS managing process identity by using an independent process identity in conjunction with the secure IPC mechanism.

21. Time and Randomness Attacks: • System clock: Malicious OS could speed up or slow down the clock. It might also cause the clock to move backwards, an unexpected situation that could expose bugs in application code. • Randomness: The standard system of sources is not suitable for use in cryptographic applications.

22. Time and Randomness II Solutions: • To implement a trusted clock and source of secure randomness in the VMM. This requires additional trusted components to the system. • It is challenging to keep time perfectly synchronized between the guest OS’s clock and the VMM’s even with a correctly functioning OS.

23. I/O & Trusted Paths Attacks: • An application’s input and output paths to the external world go through the operating system, including display output and user input. • Network I/O also depends on the operating system, but many applications treat the network as an untrusted entity.

24. I/O & Trusted Paths II Solution(s): • Overshadows memory protection can ensure that only the application and the virtual graphics card can access the server’s frame buffer in unencrypted form.

25. Identity Management Attack(s): • The OS is responsible for managing a number of types of identities. These OS-managed identities are frequently used in authentication • A malicious OS could cause a connection from an attacker to appear to be coming from a trusted local user or host.

26. Identity Management II Solutions: • Applications should not rely on these identities for authentication or other security criticalpurposes. • Sometimes it is possible to securely authenticate a local connection simply by verifying that both endpoints are in the same secure compartment.

27. Conclusions • Systems looking to enhance OS assurance typically focus on core isolation mechanisms such as memory and CPU protection, but this is not sufficient to build a secure system and attention must be onto how applications are affected by malicious OS behavior. • The trusted part can be made considerably smaller than the un-trusted part. • Proposed extensions to Overshadow that allow it to tolerate more complex malicious behavior from the OS, based on the principle of verifying system call results.

28. Acknowledgements • I would like to thank my professor Michael Rothstein . • I am grateful to AvinashBolla, Poorna Kumar Raju, KarthikAvss without whose help this presentation would not have been possible on time.

29. Thank you