1 / 22

John R. Lange and Kevin Pedretti Trammell Hudson , Peter Dinda , Zheng Cui , Lei Xia ,

Palacios and Kitten: New High Performance Operating Systems For Scalable Virtualized and Native Supercomputing. John R. Lange and Kevin Pedretti Trammell Hudson , Peter Dinda , Zheng Cui , Lei Xia , Patrick Bridges , Andy Gocke , Steven Jaconette ,

astra-nolan
Download Presentation

John R. Lange and Kevin Pedretti Trammell Hudson , Peter Dinda , Zheng Cui , Lei Xia ,

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Palacios and Kitten:New High Performance Operating Systems ForScalable Virtualized and Native Supercomputing John R. Lange andKevin Pedretti Trammell Hudson, Peter Dinda, Zheng Cui, Lei Xia, Patrick Bridges, Andy Gocke, Steven Jaconette, Mike Levenhagen and Ron Brightwell Northwestern University Sandia National Labs University of New Mexico

  2. Summary • Palacios • First VMM for scalable HPC • Open Source and available • Kitten • First open source Lightweight Kernel for High Performance Computing (HPC) • Open Source and available • Proved HPC virtualization is effective at scale • Performance within 5% of native • Largest scale study of virtualization

  3. What is a virtual machine? • Run an OS as an application • Run multiple OS environments on a single machine • Start, stop, pause • Can easily move entire OS environments Page Tables CPU state Hardware Application Application Application Guest Application Guest OS Guest OS Guest OS OS Page Tables CPU state Hardware HostOS/VMM VMM Emulate Hardware Hardware Hardware

  4. What are VMMs currently used for? • Server Consolidation • Fault tolerance • Legacy application support • Debugging • Isolation • Virtual appliances • Failover and disaster recovery • Market size • 2007: $5.5 billion • 2011: $11.7 billion $7.58 Billion $16.70 Billion

  5. High Performance Computing (HPC) • Large scale simulations to solve Big Problems

  6. Virtualization in HPC • Fault tolerance • RedStorm MTBI target: 50 hours • RedStorm Min TTR: 30 minutes – 1 hour • Broader usage • Allow applications to select best OS • Only if it doesn’t degrade performance… • Tightly coupled parallel applications • Very large scale A.B. Nagarajan, F. Mueller, C. Engelmann, and S.L. Scott Proactive Fault Tolerance for HPC with Xen Virtualization ICS 2007

  7. Palacios VMM • OS-independent embeddable virtual machine monitor • Developed at Northwestern and University of New Mexico • Open source and freely available • Downloaded over 1000 times as of July 2009 • Users: • Kitten: Lightweight supercomputing OS from Sandia National Labs • MINIX 3 • Modified Linux versions • Successfully used on supercomputers, clusters (Infiniband and Ethernet), and servers • http://www.v3vee.org/palacios

  8. Palacios as an HPC VMM • Minimalist interface • Suitable for an LWK • Compile and runtime configurability • Create a VMM tailored to specific environments • Low noise • Contiguous memory pre-allocation • Passthrough resources and resource partitioning

  9. Lightweight Kernel Timeline 1991 –Sandia/UNM OS (SUNMOS), nCube-2 1991 – Linux 0.02 1993 –SUNMOS ported to Intel Paragon (1800 nodes)‏ 1993 –SUNMOS experience used to design Puma First implementation of Portals communication architecture 1994 – Linux 1.0 1995 –Puma ported to ASCI Red (4700 nodes)‏ Renamed Cougar, productized by Intel 1997 –Stripped down Linux used on Cplant (2000 nodes)‏ Difficult to port Puma to COTS Alpha server Included Portals API 2002 –Cougar ported to ASC Red Storm (13000 nodes)‏ Renamed Catamount, productized by Cray Host and NIC-based Portals implementations 2004 – IBM develops LWK (CNK) for BG/L/P (106000 nodes)‏ 2005 – IBM & ETI develop LWK (C64) for Cyclops64 (160 cores/die)‏

  10. Kitten: An Open Source LWK http://software.sandia.gov/trac/kitten • Better match for user expectations • Provides mostly Linux-compatible user environment • Including threading • Supports unmodified compiler toolchains and ELF executables • Better match vendor expectations • Modern code-base with familiar Linux-like organization • Drop-in compatible with Linux • Infiniband support • End-goal is deployment on future capability system

  11. Complexity • Scalable HPC performance requires minimal overhead Xen: 580k lines (50k – 80k core) KVM: 50k-60k lines + Kernel dependencies (??) + User level devices (180k)

  12. HPC Performance Evaluation • Virtualization is very useful for HPC, but… Only if it doesn’t hurt performance • Virtualized RedStorm with Palacios • Evaluated with Sandia’s system evaluation benchmarks 17th fastest supercomputer Cray XT3 38208 cores ~3500 sq ft 2.5 MegaWatts $90 million

  13. Virtualized performance(Catamount) Within 5% Scalable HPCCG: conjugant gradient solver

  14. Comparison of Operating Systems Shadow Paging Compute Node Linux Catamount HPCCG: conjugant gradient solver

  15. Comparison of Operating Systems Compute Node Linux Catamount CTH: multi-material, large deformation, strong shockwave simulation

  16. Large Scale Study • Evaluation on full RedStorm system • 12 hours of dedicated system time on full machine • Largest virtualization performance scaling study to date • Measured performance at exponentially increasing scales • Up to 4096 nodes • Publicity • New York Times • Slashdot • HPCWire • Communications of the ACM • PC World

  17. Scalability at Large Scale(Catamount) Within 3% Scalable CTH: multi-material, large deformation, strong shockwave simulation

  18. Commodity Systems • Kitten and Palacios fully support commodity systems • Infiniband clusters • Ethernet servers • Generic PC hardware • Palacios embeddable in many OSes • Kitten • MINIX 3 • Linux • GeekOS

  19. Infiniband on Commodity Linux (Linux guest on IB cluster) 2 node Infiniband Ping Pong bandwidth measurement

  20. Summary • Virtualization can scale • Near native performance for optimized VMM/guest (within 5%) • VMM needs to know about guest internals • Should modify behavior for each guest environment • Example: Paging method to use depends on guest • Black Box inference is not desirable in HPC environment • Unacceptable performance overhead • Convergence time • Mistakes have large consequences • Need guest cooperation • Guest and VMM relationship should be symbiotic • Paper forthcoming (4096 scaling results and techniques)

  21. Future Work • Continue exploring virtualization in HPC • NU, UNM and SNL collaboration • Granted 5 million hours on Jaguar • Current fastest supercomputer in the world Oak Ridge National Labs Cray XT5 224,256 cores 4352 sq. ft 6.95 MegaWatts $104 million

  22. Conclusion • Palacios and Kitten • Two open source tools for HPC • Proved virtualization of HPC systems can scale • Contributions Welcome!! • http://www.v3vee.org • http://software.sandia.gov/trac/kitten

More Related