Palacios and Kitten:
1 / 22

Palacios and Kitten: New High Performance Operating Systems For Scalable Virtualized and Native Supercomputing - PowerPoint PPT Presentation

  • Uploaded on

Palacios and Kitten: New High Performance Operating Systems For Scalable Virtualized and Native Supercomputing. John R. Lange and Kevin Pedretti Trammell Hudson , Peter Dinda , Zheng Cui , Lei Xia , Patrick Bridges , Andy Gocke , Steven Jaconette ,

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Palacios and Kitten: New High Performance Operating Systems For Scalable Virtualized and Native Supercomputing' - Sharon_Dale

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Slide1 l.jpg

Palacios and Kitten:New High Performance Operating Systems ForScalable Virtualized and Native Supercomputing

John R. Lange andKevin Pedretti

Trammell Hudson, Peter Dinda,

Zheng Cui, Lei Xia,

Patrick Bridges, Andy Gocke,

Steven Jaconette,

Mike Levenhagen and Ron Brightwell

Northwestern University

Sandia National Labs

University of New Mexico

Summary l.jpg

  • Palacios

    • First VMM for scalable HPC

    • Open Source and available

  • Kitten

    • First open source Lightweight Kernel for High Performance Computing (HPC)

    • Open Source and available

  • Proved HPC virtualization is effective at scale

    • Performance within 5% of native

    • Largest scale study of virtualization

What is a virtual machine l.jpg
What is a virtual machine?

  • Run an OS as an application

    • Run multiple OS environments on a single machine

    • Start, stop, pause

    • Can easily move entire OS environments

Page Tables

CPU state







Guest OS

Guest OS

Guest OS


Page Tables

CPU state








What are vmms currently used for l.jpg
What are VMMs currently used for?

  • Server Consolidation

  • Fault tolerance

  • Legacy application support

  • Debugging

  • Isolation

  • Virtual appliances

  • Failover and disaster recovery

  • Market size

    • 2007: $5.5 billion

    • 2011: $11.7 billion

$7.58 Billion

$16.70 Billion

High performance computing hpc l.jpg
High Performance Computing (HPC)

  • Large scale simulations to solve Big Problems

Virtualization in hpc l.jpg
Virtualization in HPC

  • Fault tolerance

    • RedStorm MTBI target: 50 hours

    • RedStorm Min TTR: 30 minutes – 1 hour

  • Broader usage

    • Allow applications to select best OS

  • Only if it doesn’t degrade performance…

    • Tightly coupled parallel applications

    • Very large scale

A.B. Nagarajan, F. Mueller, C. Engelmann, and S.L. Scott

Proactive Fault Tolerance for HPC with Xen Virtualization

ICS 2007

Palacios vmm l.jpg
Palacios VMM

  • OS-independent embeddable virtual machine monitor

  • Developed at Northwestern and University of New Mexico

  • Open source and freely available

    • Downloaded over 1000 times as of July 2009

  • Users:

    • Kitten: Lightweight supercomputing OS from Sandia National Labs

    • MINIX 3

    • Modified Linux versions

  • Successfully used on supercomputers, clusters (Infiniband and Ethernet), and servers


Palacios as an hpc vmm l.jpg
Palacios as an HPC VMM

  • Minimalist interface

    • Suitable for an LWK

  • Compile and runtime configurability

    • Create a VMM tailored to specific environments

  • Low noise

  • Contiguous memory pre-allocation

  • Passthrough resources and resource partitioning

Lightweight kernel timeline l.jpg
Lightweight Kernel Timeline

1991 –Sandia/UNM OS (SUNMOS), nCube-2

1991 – Linux 0.02

1993 –SUNMOS ported to Intel Paragon (1800 nodes)‏

1993 –SUNMOS experience used to design Puma

First implementation of Portals communication architecture

1994 – Linux 1.0

1995 –Puma ported to ASCI Red (4700 nodes)‏

Renamed Cougar, productized by Intel

1997 –Stripped down Linux used on Cplant (2000 nodes)‏

Difficult to port Puma to COTS Alpha server

Included Portals API

2002 –Cougar ported to ASC Red Storm (13000 nodes)‏

Renamed Catamount, productized by Cray

Host and NIC-based Portals implementations

2004 – IBM develops LWK (CNK) for BG/L/P (106000 nodes)‏

2005 – IBM & ETI develop LWK (C64) for Cyclops64 (160 cores/die)‏

Kitten an open source lwk l.jpg
Kitten: An Open Source LWK

  • Better match for user expectations

    • Provides mostly Linux-compatible user environment

      • Including threading

    • Supports unmodified compiler toolchains and ELF executables

  • Better match vendor expectations

    • Modern code-base with familiar Linux-like organization

      • Drop-in compatible with Linux

    • Infiniband support

  • End-goal is deployment on future capability system

Complexity l.jpg

  • Scalable HPC performance requires minimal overhead

Xen: 580k lines (50k – 80k core)

KVM: 50k-60k lines

+ Kernel dependencies (??)

+ User level devices (180k)

Hpc performance evaluation l.jpg
HPC Performance Evaluation

  • Virtualization is very useful for HPC, but…

    Only if it doesn’t hurt performance

  • Virtualized RedStorm with Palacios

    • Evaluated with Sandia’s system evaluation benchmarks

17th fastest supercomputer

Cray XT3

38208 cores

~3500 sq ft

2.5 MegaWatts

$90 million

Virtualized performance catamount l.jpg
Virtualized performance(Catamount)

Within 5%


HPCCG: conjugant gradient solver

Comparison of operating systems l.jpg
Comparison of Operating Systems

Shadow Paging

Compute Node Linux


HPCCG: conjugant gradient solver

Comparison of operating systems15 l.jpg
Comparison of Operating Systems

Compute Node Linux


CTH: multi-material, large deformation, strong shockwave simulation

Large scale study l.jpg
Large Scale Study

  • Evaluation on full RedStorm system

    • 12 hours of dedicated system time on full machine

    • Largest virtualization performance scaling study to date

  • Measured performance at exponentially increasing scales

    • Up to 4096 nodes

  • Publicity

    • New York Times

    • Slashdot

    • HPCWire

    • Communications of the ACM

    • PC World

Scalability at large scale catamount l.jpg
Scalability at Large Scale(Catamount)

Within 3%


CTH: multi-material, large deformation, strong shockwave simulation

Commodity systems l.jpg
Commodity Systems

  • Kitten and Palacios fully support commodity systems

    • Infiniband clusters

    • Ethernet servers

    • Generic PC hardware

  • Palacios embeddable in many OSes

    • Kitten

    • MINIX 3

    • Linux

    • GeekOS

Infiniband on commodity linux l.jpg
Infiniband on Commodity Linux

(Linux guest on IB cluster)

2 node Infiniband Ping Pong bandwidth measurement

Summary20 l.jpg

  • Virtualization can scale

    • Near native performance for optimized VMM/guest (within 5%)

  • VMM needs to know about guest internals

    • Should modify behavior for each guest environment

    • Example: Paging method to use depends on guest

  • Black Box inference is not desirable in HPC environment

    • Unacceptable performance overhead

    • Convergence time

    • Mistakes have large consequences

  • Need guest cooperation

    • Guest and VMM relationship should be symbiotic

    • Paper forthcoming (4096 scaling results and techniques)

Future work l.jpg
Future Work

  • Continue exploring virtualization in HPC

    • NU, UNM and SNL collaboration

    • Granted 5 million hours on Jaguar

      • Current fastest supercomputer in the world

Oak Ridge National Labs

Cray XT5

224,256 cores

4352 sq. ft

6.95 MegaWatts

$104 million

Conclusion l.jpg

  • Palacios and Kitten

    • Two open source tools for HPC

    • Proved virtualization of HPC systems can scale

  • Contributions Welcome!!