Xen and the Art of Virtualization

1. Xen and the Art of Virtualization By Paul Barham, Boris Dragovic, Stevan Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield. Presented by Diana Carroll

2. Virtual Machines • One hardware system with memory, processors, I/O devices. • Multiple execution environments that each map to an identical representation of the physical system. • An OS running on a virtual machine is not aware that it is sharing the machine. • Virtual machines must be isolated from each other even though they share the same hardware. • The execution of one can’t stall or corrupt the others. • The performance overhead needs to be acceptably small. • The virtual machines must share the hardware as equally as possible. • A Virtual Machine Monitor is needed to accomplish this. Xen and the Art of Virtualization Diana Carroll

3. Virtual Machine Monitors • Also known as a hypervisor. • Provides an interface for multiple virtual machines to coexist together. • Can run multiple operating systems on a single computer. • Provides stability, since even if one OS crashes, the rest of the machine remains functional. • Can eliminate the need for multiple machines dedicated to different operating systems. • Provides isolation between operating system instances and multiplexes physical resources across the running virtual machines. • Much like an OS does with processes. Xen and the Art of Virtualization Diana Carroll

4. Xen • Xen is a Virtual Machine Monitor (VMM). • Allows users to dynamically instantiate an operating system. • Hosts operating systems such as Linux and Windows. • Some source code modifications are necessary. • In the paper, XenoLinux was complete, Windows XP and NetBSD still in progress. • Now, NetBSD, Linux, FreeBSD, Plan9, and NetWare are complete. WindowsXP port was successful, but licensing prohibitions prevent it from being released. (1) • Multiple operating systems can run simultaneously and perform different tasks. • Is completely software based and requires no special hardware support. • Full virtualization, in which the virtual hardware is identical to the underlying physical hardware, is virtually impossible on the x86 architecture. • Xen provides a similar, but not quite identical view of the hardware. Xen and the Art of Virtualization Diana Carroll

5. Xen Design Principles • Support unmodified application binaries. • Necessary to ensure that it is useful for users. • Support fully functional, multi-application operating systems a guests. • Use paravirtualization to provide high performance and good resource isolation. • The guest operating system has to be modified to run on the Virtual Machine Monitor. • Specifically, the guest OS can no longer execute in ring 0, because that ring is now occupied by the VMM. • The guest OS has to be modified to run outside of ring 0. • Sometimes more correct behavior and better performance are achieved when the resource virtualization is not completely hidden. Xen and the Art of Virtualization Diana Carroll

6. Xen versus Disco • Disco uses true virtualization (almost) • True virtualization does not require any modification of the guest OS. • The virtual machine is indistinguishable from the real hardware. • Xen uses paravirtualization • The guest OS has to be modified, or ported, onto the Xen hypervisor. • Xen virtual machines resemble the real hardware but do not attempt to be an exact match. • When appropriate, the guest OS makes calls to the hypervisor rather than to the hardware. • e.g. For memory management and I/O. • Solves the problem of architectures like the x86 that do not support true virtualization. • The TLB is hardware managed rather than software managed. Xen and the Art of Virtualization Diana Carroll

7. The Virtual Machine Interface • A paravirtualized version of the X86 interface. • In this case, the x86 architecture is a worst-case environment. • Divided into memory management, CPU, and I/O. • Guest operating systems execute within domains. • A domain is a running virtual machine. Xen and the Art of Virtualization Diana Carroll

8. Memory Management • Guest OS’s are responsible for allocating and managing the hardware page tables. • Minimal involvement from Xen is required to ensure safety and isolation. • Necessary since x86 does not have a software-managed TLB, which could be efficiently virtualized. • Xen exists in a 64MB section at the top of each address space. • This avoids the TLB being flushed each time the execution path enters or leaves the hypervisor. • Guest OS allocates and initializes a new page table from its own memory and then registers it with Xen. • All subsequent updates must be validated by Xen. • Updates can be batched to improve efficiency. • Segment descriptors are also validated. They must have lower privelege than Xen, and cannot allow access to the Xen-reserved portion of the address space. Xen and the Art of Virtualization Diana Carroll

9. Virtualizing the CPU • Applications run at different privilege levels. • Typically, in x86, an OS runs at ring 0, as the most privileged entity in the system. • Applications usually run at ring 3. • With a virtualized CPU, the OS no longer runs at ring 0. • This privilege level is now reserved for the VMM. • The guest OS must be modified to run at a lower privilege level. • Since most OS implementations do not use rings 1 and 2, the guest OS can be ported to ring 1. • This prevents the guest OS from executing privileged hypervisor code, but keeps it safely isolated from applications that are still running in ring 3. Xen and the Art of Virtualization Diana Carroll

10. CPU virtualization continued • Privileged instructions are required to be validated and executed within Xen. • e.g. Installing a new page table or yielding the processor. • Attempts to execute a privileged instruction fails since only Xen operates at the highest privilege level. • Exceptions are managed using a table of exception handlers. • Page fault handler is the only one that has to be modified to read from an extended stack frame instead of a register. • System calls allow each guest OS to register a ‘fast’ exception handler, since it is not necessary for it to run in ring 0. • All exception handlers are validated by Xen. • Checked to ensure that the handler code does not specify execution in ring 0. Xen and the Art of Virtualization Diana Carroll

11. I/O • Xen uses a set of device abstractions instead of emulating existing hardware devices. • I/O data is transferred to and from each domain via Xen. • “Shared memory, asynchronous buffer-descriptor rings” are used to pass I/O buffer information vertically through the system. • Asynchronous notifications of I/O events are made to a domain. • Made by updating a bitmap of pending event types, and possible calling an event handler as specified by its OS. Xen and the Art of Virtualization Diana Carroll

12. Requires less than 2% of the total lines of code to be modified. Porting an OS to Xen The User Software runs on the Guest OS without requiring modification. Xen and the Art of Virtualization Diana Carroll

13. Separating Policy from Mechanism • The hypervisor only provides basic control operations. • Authorized domains can export these operations through a control interface. • An initial domain, Domain0, is created at boot time and can access the control interface. • It can then use the control interface to create and manage additional domains. • Responsible for building the domain and initial structures to support each guest OS. • Can be specialized to handle the varying requirements of different OSes. • The control interface also supports virtual I/O devices. • Virtual Network Interfaces (VIF) and Virtual Block Devices (VBD). • Additional administrative tools may be added to Domain0 in the future. Xen and the Art of Virtualization Diana Carroll

14. Control Transfer • Hypercalls are made from a domain to Xen. • A synchronous software trap into the hypervisor. • e.g. to request a set of page-table updates or other privileged operation. • Control is returned to the calling domain when the call is completed. • Notifications from Xen to a domain are made using an asynchronous event mechanism. • Replaces the delivery mechanism for device interrupts. • Allows lightweight notification of events. • Similar to Unix signals. Xen and the Art of Virtualization Diana Carroll

15. I/O descriptor rings are a circular queue of descriptors that hold producer/consumer pointer pairs. Descriptors are allocated by a domain, but accessible from within Xen. Access to the ring is controlled by two pairs of pointers. Domains produce requests and advance the request producer pointer. Xen removes requests and advances the request consumer pointer. Xen produces responses and advances the response producer pointer. Domains remove responses and advance the response consumer pointer. Device Transfer • The virtual memory manager is an extra protection domain between guest OS and I/O device. • Data needs to be transferred from I/O device to OS with as little overhead as possible. Xen and the Art of Virtualization Diana Carroll

16. Virtualization of System Components • CPU scheduling is done using the Borrowed Virtual Time algorithm. • Thread execution is monitored in terms of virtual time. • The scheduler selects the thread with the earliest effective virtual time. • A thread can borrow virtual time by warping back to appear earlier and gain dispatch priority. • But it then goes to the end of the line after execution. • Protects against low-latency threads using excessive processing cycles. • CPU resources are allocated dynamically, no need to predict processing requirements in advance. • Guest OSes are given three ways of interpreting time. • Virtual time only advances while the domain is executing. • Real time is the time in nanoseconds since the machine boot (can be locked to an external time source). • Wall-clock time is real-time + offset. Xen and the Art of Virtualization Diana Carroll

17. Components Continued: Virtual Address Translation • The x86 architecture uses hardware page tables, which makes memory virtualization more difficult. • Xen only deals with page table updates. • Guest OS page tables are registered directly with the MMU. • Guest OSes have read-only access. • No need to use shadow page tables. • A guest OS passes Xen its page table updates using a hypercall. • Requests are validated, and then applied. • A type and reference count are kept for each machine page frame, and are used to validate updates. • Frames that have already been validated are marked so they do not have to be revalidated. • Hypercall requests can be batched to improve performance. Xen and the Art of Virtualization Diana Carroll

18. Physical Memory and Disk • Each domain receives an initial reservation of memory. • Memory is statically divided between domains. • A domain may claim additional memory pages up to its reservation limit. • A domain may also release pages back to Xen. • A balloon driver passes memory pages from Xen to the guest OS’s page allocator. • Mapping from physical to hardware addresses is left to the OS. • Xen provides a shared translation array that is readable by all domains. Updates are validated by Xen first. • Only Domain0 has direct access to all physical drives. All other domains access a virtual block device (VBD) • Domain0 manages the virtual block devices, using the I/O ring queuing mechanism to control access. • A VBD is composed of a list of extents with associated ownership and access control information. • To a guest OS, the VBD behavior is very similar to that of a SCSI disk. • Xen keeps the translation table, and my reorder requests or process them in batches. Xen and the Art of Virtualization Diana Carroll

19. Performance Five implementations compared in total. • Compared 3 VMM’s • Vmware workstation 3.2 • User-Mode Linux (runs the Linux OS in user-mode on a Linux host) • Xen with XenoLinux port • Also Native Linux • All used Redhat 2.0 with the Linux 2.4.21 kernel, i686 architecture, ext3 file system. • All used Dell 2650 dual processor 2.4GHz systems, 2GB RAM, gigabit Ethernet, and 146GB SCSI drive. Hyperthreading disabled. • Also tested the ESX server, which replaces the guest OS with a dedicated kernel on VMware, but unable to report the results (EULA restrictions). Xen and the Art of Virtualization Diana Carroll

20. Performance Results • Cluster 1: SPEC CPU suite. • Computationally intensive application, very little I/O and OS interaction. • Cluster 2: Time taken to build a default configuration of the Linux 2.4.21 kernel with gcc v2.96. • Cluster 3: Open Source Database Benchmark suite in default configuration. • Information retrieval shown in tuples per second. • Cluster 4: Open Source Database Benchmark suite in default configuration. • Online Transaction Processing workloads shown in tuples per second. • Cluster 5: dbench program emulating load placed on a file server. • Cluster 6: SPEC Web99 is a web server benchmark. Xen and the Art of Virtualization Diana Carroll

21. Operating System Benchmarks Measured using the lmbench program, version 3.0-a3 • L-UP is native Linux uniprocessor. • L-SMP is native Linux multiprocessor. • Xen is running XenoLinux, their port of the Linux OS. • VMW is VMware. • UML is user-mode Linux. Xen and the Art of Virtualization Diana Carroll

22. Further Performance Measures Multiple instances of PostreSQL in separate domains • OSDB-IR = Open Source Database Benchmark Information Retrieval. • OSDB-OLTP = Open Source Database Benchmark On-line Transaction Processing. • Performance Isolation • They couldn’t find another OS-based implementation of performance isolation to compare it with. • They tested Xen using 4 domains running with equal resource allocations. • 2 domains running previously-measured workloads. • 2 domains running disruptive processes (e.g. disk bandwidth hog, fork bomb, memory grabber). • The impact of the disruptive processes was only a 2-4% decrease in performance . • Same processes effectively shut down a native Linux system. Xen and the Art of Virtualization Diana Carroll

23. Scalability • Xen’s target was to scale to 100 domains. • They were able to configure a guest OS for server functionality, running a memory of only 4MB with swap. • When an incoming request was received, it could request more memory from Xen. • Compared to native Linux, they found a tradeoff situation. • Long time slices gives the highest throughput, but less responsiveness. Xen running with 50ms time slices had similar throughput to Linux. • Short time slices lowered throughput but improved responsiveness. • With 128 domains running, Xen still provided a response time of 5.4ms. • 5ms time slices resulted in 7.5% lower throughput. Xen and the Art of Virtualization Diana Carroll

24. Conclusion • Xen is a software based Virtual Machine Monitor (hypervisor). • Allows multiple OSes to be hosted simultaneously on the same machine. • Requires the OS to be modified (ported) in order to run on the VMM. • Provides the protection of performance isolation between domains. • Xen today… • Open-source project published under the GPL. • Currently on version 3.0. • NetBSD, Linux (several distros, including SuSE, Fedora, RHEL, Mandrake), FreeBSD, Plan9, and NetWare are complete. WindowsXP port was successful, but licensing prohibitions prevent it from being released. • Hardware support for virtualization • Intel is releasing a new line of processors that support virtualization. • 2 forms of CPU operation. • In addition to levels 0-3, there is also a root level where the VMM can run. • Guest OSes still can run at level 0, so porting is no longer required. • Virtual Machine Control Structure (VMCS) manages VM entries and exits. Xen and the Art of Virtualization Diana Carroll

25. References • University of Cambridge Xen page • http://www.cl.cam.ac.uk/Research/SRG/netos/xen/ • Wikipedia entry for Xen • http://en.wikipedia.org/wiki/Xen_%28virtual_machine_monitor%29 • Intel Virtualization Technology, by Rich Uhlig, Gil Neiger, Dio Rodgers, Amy Santoni, Fernando Martins, Andrew Anderson, Steven Bennett, Alain Kagi, Felix Leung, and Larry Smith. • Published in Computer magazine, May 2005 (Vol. 38, No. 5) ISSN: 0018-9162 • Borrowed-Virtual-Time (BVT) scheduling: Supporting Latency-sensitive Threads in a General-purpose Scheduler • Kenneth J. Duda and David R. Cheriton Xen and the Art of Virtualization Diana Carroll