1 / 29

Cloud Computing with Nimbus

Cloud Computing with Nimbus. April 2009 Kate Keahey (keahey@mcs.anl.gov) University of Chicago Argonne National Laboratory. Cloud Computing. SaaS Software-as-a-Service. elasticity computing on demand. PaaS Platform-as-a-Service. capital expense operational expense. IaaS

edmondm
Download Presentation

Cloud Computing with Nimbus

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Cloud Computing with Nimbus April 2009 Kate Keahey (keahey@mcs.anl.gov) University of Chicago Argonne National Laboratory

  2. Cloud Computing SaaS Software-as-a-Service elasticity computing on demand PaaS Platform-as-a-Service capital expense operational expense IaaS Infrastructure-as-a-Service

  3. Cloud Computing for Science • Environment • Resource control

  4. “Workspaces” • Dynamically provisioned environments • Environment control • Resource control • Implementations • Via leasing hardware platforms: reimaging, configuration management, dynamic accounts… • Via virtualization: VM deployment Isolation

  5. A Brief History of Nimbus First STAR production run on EC2 Xen released EC2 goes online Nimbus Cloud comes online 2003 2006 2009 Research on agreement-based services First WSRF Workspace Service release Support for EC2 interfaces EC2 gateway available Context Broker release

  6. Nimbus Goals • Allow providers to build clouds • Private clouds (privacy, expense considerations) • Workspace Service: open source EC2 implementation • Allow users to use cloud computing • Do whatever it takes to enable scientists to use IaaS • Context Broker: turnkey virtual clusters • IaaS Gateway: interoperability • Allow developers to experiment with Nimbus • For research or usability/performance improvements • Community extensions and contributions

  7. The Workspace Service Pool node Pool node Pool node VWS Service Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node

  8. The Workspace Service The workspace service publishes information about each workspace Pool node Pool node Pool node VWS Service Pool node Pool node Pool node Users can find out information about their workspace (e.g. what IP the workspace was bound to) Pool node Pool node Pool node Pool node Pool node Pool node Users can interact directly with their workspaces the same way the would with a physical machine.

  9. Workspace Service: Interfaces and Clients • Web Services based • Web Service Resource Framework (WSRF) • WS + state management (WS-Notification) • Elastic Computing Cloud (EC2) • Compatible with EC2 clients • Supported: ec2-describe-images, ec2-run-instances, ec2-describe-instances, ec2-terminate-instances, ec2-reboot-instances, ec2-add-keypair, ec2-delete-keypair • Unsupported: availability zones, security groups, elastic IP assignment, REST

  10. Workspace Service: Security • GSI authentication and authorization • PKI-based • VOMS, Shibboleth (via GridShib), custom PDPs • Secure access to VMs • EC2 key generation or accessed from .ssh • Validating images and image data • Extensions from Vienna University of Technology • Paper: Descher et al., Retaining Data Control in Infrastructure Clouds, ARES (the International Dependability Conference), 2009.

  11. Workspace Service: Networking • Network configuration • External: public IPs or private IPs (via VPN) • Internal: private network via a local cluster network • Each VM can specify multiple NICs mixing private and public networks (WSRF only) • E.g., cluster worker nodes on a private network, headnode on both public and private network

  12. Workspace Components workspace resource manager WSRF workspace service workspace control EC2 workspace pilot workspace client OpenNebula Project • See papers at: http://workspace.globus.org/papers/index.html • “Simple Leases with Workspace Pilot” (EuroPar08) • “Combining Batch Execution and Leasing Using • Virtual Machines” (HPDC08),

  13. Cloud Capabilities storage service workspace resource manager WSRF workspace control workspace service workspace pilot EC2 cloud client workspace client

  14. The IaaS Gateway storage service workspace resource manager WSRF workspace control workspace service workspace pilot EC2 IaaS gateway EC2 potentially other providers cloud client workspace client

  15. Cloud Computing Ecosystem Appliance Providers Marketplaces, commercial providers, Virtual Organizations Appliance management software Deployment Orchestrator User Environments VMM/DataCenter/IaaS User Environments VMM/DataCenter/IaaS

  16. Turnkey Virtual Clusters IP1 HK1 IP2 HK2 IP3 HK3 • Turnkey, tightly-coupled cluster • Shared trust/security context • Shared configuration/context information IP1 HK1 IP1 HK1 IP1 HK1 MPI IP2 HK2 IP2 HK2 IP2 HK2 IP3 HK3 IP3 HK3 IP3 HK3 Context Broker

  17. Context Broker Goals • Can work with every appliance • Appliance schema, can be implemented in terms of many configuration systems • Can work with every cloud provider • Simple and minimal conditions on generic context delivery • Can work across multiple cloud providers, in a distributed environment

  18. Context Broker Status • Releases • In alpha since 08/07, first release 06/08, update 01/09 • Used to contextualize cluster composed of 100s of virtual nodes for multiple production apps • Contextualized images on workspace marketplace • Working with rPath to make contextualization easier for the user • Discussing OVF extensions Paper: Keahey&Freeman, Contextualization: Providing One-Click Virtual Clusters, eScience 2008

  19. End of Nimbus Tour storage service workspace resource manager WSRF workspace control workspace service EC2 workspace pilot context broker IaaS gateway EC2 potentially other providers context client cloud client workspace client

  20. Science Clouds • Goals • Enable scientific projects to experiment with IaaS clouds • Evolve software in response to the needs of scientific projects • A laboratory for exploration of cloud interoperability issues • Participants • University of Chicago (since 03/08, 16 nodes), University of Florida (05/08, 16-32 nodes, access via VPN), Masaryk University, Brno, Czech Republic (08/08), Wispy @ Purdue (09/08) • In progress: Grid5K, Vrije, others • Using EC2 for large runs • Simple governance model, access given to any scientific project • http://workspace.globus.org/clouds

  21. Who Runs on Nimbus at UC? 100+ DNs projects ranging across Science, CS, education, build&test…

  22. STAR • STAR: a nuclear physics experiment studies fundamental properties of nuclear matter • Computations require complex and consistently configured environments • Requirements • A virtual OSG STAR cluster: OSG headnode (gridmapfiles, host certificates, NFS, Torque), worker nodes: SL4 + STAR • From Science Clouds to EC2 runs • One-click virtual cluster deployment: Context Broker • Producing just-in-time results for Quark Matter conference: http://www.isgtw.org/?pid=1001735 • Work by Jerome Lauret, Doug Olson, Leve Hajdu, Lidia Didenko at BNL

  23. Alice HEP Experiment at CERN • Collaboration with CERNVM project • HPCwire article

  24. Sky Computing U of Florida U of Chicago ViNE router ViNE router ViNE router Purdue

  25. Sky Computing U of Chicago U of Florida • Papers: • “Sky Computing”, by K. Keahey, A. Matsunaga, M. Tsugawa, J. Fortes. Submitted to IEEE Internet Computing. • “CloudBLAST: Combining MapReduce and Virtualization on Distributed Resources for Bioinformatics Applications” by A. Matsunaga, M. Tsugawa and J. Fortes. eScience 2008. Hadoop cloud Purdue

  26. IaaS Clouds vs Grids • Grid computing • Assumption: site retains control over resources • Remote interfaces to local site mechanisms • Tradeoff: difficult to provide the right environments and control but easy to deploy • Cloud computing • Assumption: a user gets a “lease” on a remote resource that it gets to control • Enabled by virtauliaztion (Xen) • Tradeoff: eanbles a larger class of applications but hard to deploy • Raises issues: e.g., site licenses? Configuration support? • Towards “sky computing” • I can now trust a remote resource: I configured it myself • Cloud computing + virtual networks • Local distributed environment

  27. Friends and Family • Committers: Kate Keahey & Tim Freeman (ANL/UC), Ian Gable (UVIC) • A lot of help from the community, see: http://workspace.globus.org/people.html • Collaborations: • Cumulus: S3 implementation (Globus team) • EBS: IU project • Appliance management: rPath, Bcfg2 project, CohesiveFT • Virtual network overlays: University of Florida • Security (research): Vienna University of Technology

  28. Open Source IaaS Implementations • OpenNebula • Open source datacenter implementation • University of Madrid, I. Llorente & team, 03/2008 • Eucalyptus • Open source implementation of EC2 • UCSB, R. Wolski & team, 06/2008 • Cloud-enabled Nimrod-G • Open source implementation of EC2 • Monash University, MeSsAGE Lab, 01/2009 • Industry efforts • openQRM, Enomalism

  29. Parting Thoughts • Science-driven cloud computing • Importance of open source • Drive requirements into the infrastructure, customize • Drive the development of standards • Cloud computing for the user • Combine with what we have (grid computing) • Explore new potential • Future directions • Creating the ecosystem, working out the issues, e.g. licensing, appliance support • Interoperability and standards • Service Levels

More Related