IMPROVING THE RELIABILITY OF COMMODITY OPERATING SYSTEMS

1. IMPROVING THE RELIABILITY OF COMMODITY OPERATING SYSTEMS Michael M. Swift Brian N. Bershad Henry M. Levy University of Washington

2. Key Idea • Kernel extensions are a major source of system failures • Nooks isolates extensions within lightweight protection domains inside the kernel address space • Requires little or no changes to extension and kernel code • Still prevents most system failures

3. INTRODUCTION • Want to allow existing kernel extensions to execute safelyin commodity kernels because • Computer reliability remains anunsolved issue • Kernel extensions are increasingly popular in Windows and Linux • Account for most system failures (85% for Windows XP)

4. Why? • Programmers writing device drivers are often less experienced than kernel programmers • Number of variants prevent us from testing them completely

5. The Nooks approach • Works with commodity operating systems • Requires little or no changes to extension and kernel code • Prevents most but not all system failures • Lets kernel extensions reside in the kernel address space • Solution is practical, backward-compatible and efficient

6. PREVIOUS WORK (I) • Capability-based architectures, ring and segment architectures: • Enable construction and isolation of privileged subsystems. • Slow and do not address recovery issues

7. PREVIOUS WORK (II) • Microkernels: • Put extensions into separate address spaces • Slow and do not address recovery issues • Database recovery techniques: • Atomic transactions • Work well for file system • Often awkward and slow

8. PREVIOUS WORK (III) • Type-safe programming languages: • Must rewrite the kernel • Static analysis of extensions: • Can detect some errors

9. PREVIOUS WORK (III) • Virtual machines: • Can reduce the amount of code that can crash whole machine • Fails if extension executes inside the virtual machine monitor VM VM VM VM Monitor Hardware

10. Required modifications Approach Hardware OS Extensions Capabilities yes yes yes Microkernels no yes yes Languages no yes yes New Driver Arch. no yes yes Transactions no no yes Virtual Machines no no no Static Analysis no no no Nooks no no no

11. NOOKS • Design for fault-resistance, not fault-tolerance: • System must prevent and recover from mostextension mistakes • Occupy middle ground between unprotected (Linux, Windows) and safe (SPIN, Java VM) • Design for mistakes, not abuse: • Exclude malicious behavior

12. Nooks goals • Isolation: • Isolate kernel from extension failures • Detect extension failures before they corrupt kernel • Automatic recovery from extension failures • Backward compatibility with existing systems and extensions

13. OS Kernel Isolation Interposition Object Tracking Recovery Kernel extensions Nooks functions • New system reliability layer:the Nooks Isolation Manager (NIM)

14. Nooks isolation mechanisms • Every extension executes within its own lightweight kernel protection domain • Communication between kernel and extensions must go through new kernel service, the Extension Procedure Call (XPC) • Similar to an RPC between two processes located on the same machine (Lightweight RPC)

15. Lightweight protection domains • Provide protection by having extensions execute with a different page table giving them • Read/write access to their own pages • Read only access to other kernel pages

16. Lightweight protection domains • Lightweight solution because extensions execute in kernel mode • A malicious extension could switch back to the kernel’s page table • Another could misuse DMA

17. Nooks interposition mechanisms • Ensure that • All extension-to-kernel and kernel-to-extension control flow goes through Nooks XPC mechanism • All data transfers between kernel and extension go through Nooks object tracking code • All interfaces are done through wrappers, similar to the stubs of an RPC package

18. Nooks object tracking functions • Maintain a list of kernel data structures accessed by each extension • Control all modifications to these structures • Provide object information for cleanup if object fails

19. Nooks object tracking functions • Extensions cannot directly modify kernel data structures • Object tracking code will: • Copy kernel data structures into extension address space • Copy them back after changes have been applied • Perform checks whenever possible

20. Nooks recovery functions • Detect and recover from various extension faults: • When an extension improperly invokes a kernel function • When processor raises an exception

21. Nooks recovery functions • Recovery helped by Nooks isolation mechanisms: • All access to kernel structures are done through wrappers • Nooks can successfully release extension-held kernel structures

22. IMPLEMENTATION • Inside Linux 2.4.18 kernel on Intel x86 architecture • Linux kernel provides • over 700 functions callable by extensions • over 650 extension-entry functions callable by the kernel • Most interactions between kernel and extensions go through function calls

23. Interposition Interposition Extension Extension Nooks Linux Kernel Nooks Isolation Manager Extensions are wrapped by Nooks wrapper stubs

24. Details • 22,000 lines of code • Linux kernel has 2.4 million lines • Makes no use of Intel x86 protection rings • Extensions execute at same protection level as kernel

25. Isolation • Two parts: • memory management:to implement lightweight protection domains • extension procedure call (XPC)

26. Memory management • All components share the same kernel address space • Each extension executes in a separate lightweight protection domain • Extension has: • Read-only access to kernel • Read-write access to its own domain

27. Lightweight protection domain • Each lightweight protection domain has • A synchronized copy of the kernel page table • Its own heap, a pool of stacks and other private data structures • Changing protection domains requires a change of page tables • Results in a TLB flush! • No protection against DMA misuse by extensions

28. Interposition (I) • Done by wrapper stubs all executing in kernel protection domain: • Before the call when kernel calls an extension • After the call when an extension calls the kernel • Wrappers: • Check parameters for validity • Implement call by value and result

29. Passing by value and result Variable i is increased after caller receivesserver’s reply Caller: … i = 0; abc(&i); … i = 0 abc(int *k){ (*k)++;} i = 1 i

30. Interposition (II) • Writing wrappers is not an easy task: • Requires knowing how parameters are used • Significant amount of wrapper sharing among extensions • Especially when extensions implement the same functionality

31. Object tracking • Records kernel objects and types manipulated by extensions

32. Recovery • Mostly through undoing

33. Limitations • Cannot prevent extensions from executing privileged instructions that would corrupt the kernel • Cannot prevent infinite loops inside an extension • Prevented by Linux semantics to do a thorough check of the parameters passed to the operating system • Current implementation of recovery assumes that extensions can be killed and restarted

34. Transparency • “Neither the extension not the kernel is aware of the Nooks layer” • In reality, must sometimes modify a few lines of extension code

35. Reliability • Tested eight extensions • Two sound card drivers • Four Ethernet drivers • A Win95 compatible file system (VFAT) • An in-kernel Web server • Injected 400 faults • 317 resulted in extension failures

36. Test results • Nooks eliminated 99% of the crashes observed with native Linux (313 out of 317) • Nooks is slower • VFAT benchmark spent 165 s in the kernel instead of 29.5 s for native Linux • Web server could only serve about 6,000 pages/s instead of 15,000 pages/s for native Linux Ouch!

37. PERFORMANCE • Ten percent performance penalty for sound and Ethernet drivers • Sixty percent performance penalty for in-kernel web server • (15,000 - 6,000)/15,000

38. Recovery errors • Big problems with VFAT extension • 90% of attempted recoveries resulted in on-disk corruption • VFAT extension cannot be safely killed and restarted We should expect that!

39. CONCLUSIONS • Nooks approach focuses on achievingbackward compatibility • Cannot provide complete isolation and fault-tolerance • Can still achieve “an extremely high level of operating system reliability” • Performance loss varies between 0 and 60%