Exogeni tools and capabilities for developers
This presentation is the property of its rightful owner.
Sponsored Links
1 / 21

ExoGENI : tools and capabilities for developers PowerPoint PPT Presentation


  • 60 Views
  • Uploaded on
  • Presentation posted in: General

ExoGENI : tools and capabilities for developers. Ilya Baldin, Paul Ruth, Anirban Mandal , Jonathan Mills, Claris Castillo, Chris Heermann RENCI , UNC-CH Jeff Chase, Victor Orlikowski Duke University. What’s new for GEC20?. Image snapshot script Easy way to capture the ExoGENI image

Download Presentation

ExoGENI : tools and capabilities for developers

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Exogeni tools and capabilities for developers

ExoGENI: tools and capabilities for developers

Ilya Baldin, Paul Ruth, AnirbanMandal, Jonathan Mills, Claris Castillo, Chris Heermann

RENCI, UNC-CH

Jeff Chase, Victor Orlikowski

Duke University


What s new for gec20

What’s new for GEC20?

  • Image snapshot script

    • Easy way to capture the ExoGENI image

    • Tested and ‘GPO approved’

    • Replaces ‘playpen’

  • Speaks-for in controllers

  • New libraries for building tools


Plans for gec21

Plans for GEC21

  • Release ndlLib

    • Port Flukes to use ndlLib

    • Develop other applications with it

  • Image capture

    • Integrate with NEuca

    • Make it work for bare-metal nodes

    • Obsolete playpen – it was a temporary awkward solution.


More on what s next

More on what’s next

  • Part of upcoming ORCA 5.0

    • More storage slivering for

      • Cisco racks NetApp

      • New IBM racks and Dell Racks with server-based storage

      • With I/O isolation

      • Add support to attach to bare-metal nodes

    • Virtual wire support for OVS

      • To improve network performance isolation for VMs

    • Hybrid network support for switches

      • Ethernet VLANs for those who care about performance

      • OpenFlow for those who actually care about it

    • Fully dynamic slices

      • Add/remove links/nodes at runtime

    • Inter-slice stitching


Overview

Overview

  • Details on image snapshot utility

  • Image portability with InstaGENI

  • Turning tools into libraries

  • ExoGENI Storage slivering

  • ExoGENI Stitching


Exogeni image background

ExoGENI Image Background

  • Images defined by XML metadata file

    • Image (AMI) - mandatory

    • Kernel (AKI) - optional

    • Ramdisk (ARI) - optional

  • Hosted on HTTP server

    • Permanent server at your institution

    • Soon will work with DropBox and Google Drive

  • Slice requests require

    • URL to metadata file – goes into the ‘name’ in RSpec

    • Hash of metadata file – goes into the ‘vesrion’ in RSpec


Exogeni image background metadata

ExoGENI Image Background: Metadata

<images>

<image>

<type>ZFILESYSTEM</type>

<signature>9efee8fa9c6705455c87a42477bd4ebd04d39c81</signature>

<url>http://geni-images.renci.org/images/standard/centos/centos6.3-v1.0.10.tgz</url>

</image>

<image>

<type>KERNEL</type>

<signature>5e76d034e27cb861275fc3d1bd9d97c3905ef5a2</signature>

<url>http://geni-images.renci.org/images/standard/centos/kernels/vmlinuz-2.6.32-431.5.1.el6.x86_64</url>

</image>

<image>`

<type>RAMDISK</type>

<signature>c999527cc0d8739a3d3f38d48f03fa6c73151cca</signature>

<url>http://geni-images.renci.org/images/standard/centos/kernels/initramfs-2.6.32-431.5.1.el6.x86_64.img</url>

</image>

</images>


Exogeni image snapshot

ExoGENI Image Snapshot

  • Snapshot of running ExoGENI VM

    • Creates: image, kernel, ramdisk, metadata

  • High level steps

    • Create/modify a VM – update OS, install needed packages

    • Run the script, make sure you have enough disk space

      • Use ExoGENI storage slivering to help here (next topic)

    • Copy the new image files to an http server

    • Insert metadata URL and hash into a request

More details on the ExoBlog:

http://www.exogeni.net/2014/06/creating-a-custom-image-from-an-existing-virtual-machine/


Image portability with ig

Image portability with IG

  • Porting from EG to IG is a complicated and error-prone process

  • Porting from IG to EG has not yet been achieved

  • Ultimately image snapshot utility should work for IG nodes to capture image that works on EG


Exogeni tools and capabilities for developers

  • Log into your VM

  • Get even more coffee

  • Submit RSpec

  • Your RSpec “disk_image” is the “Metadata URL”

  • Create a new Rspec

  • Boot a VM

    • Click: <Your Image Name>

    • Click: Experimentation -> List ImageIDs

    • Toggle “Global” to “Yes”

    • Get Coffee / Wait for email from Emulab declaring victory

    • Click “Submit”

      • Set types to only “pcvm”

      • Set “EC2 [email protected]: [email protected]

      • Set “Filename”:  should be preset with project dir and “Description Name”.  Example: “/proj/GENISciShakedown/images/Centos6.3.ndz”

      • Set “OS Version”: Version of the image??? I’m not sure what this is for.

      • Set “Operating System”:   Linux

      • Set “Description”: choose a description

      • Set "Descriptor Name”:  choose a name

    • Click: “Import an Amazon EC2 Instance Image”

    • Click: Experimentation -> List ImageIDs

    • Log into www.emulab.net

  • Perform ec2 import

    • cat key to >> .ssh/authorized_keys”

    • Copy emulab keys into ExoGENI instance

    • Make user the “EC2 [email protected]" is in the shudders file (i.e. [email protected] )

    • yum install ruby zip unzip sudo e2fsprogs rsync

  • setup for ec2 import 

    • edit /usr/local/etc/emulab/rc/rc.mounts and change this line:

    • Create a file called /etc/emulab/genvmtype and stick in one line that says “xen”

    • Allow sudo to run without a tty.  Add the following to the sudoers file:    "Defaults    !requiretty” 

    • Add  /users

    • Fix resov.conf and hostname

  • Install client side tools

  • Install pubsub

  • yum install perldhclientopensshmysql bison byacc flex boost gitmysql-server mysql-develmysql-embedded-develmysql-proxy-devellibtoolperl-Time-HiRes yum install python-develhttpd man httpd-devel dos2unix swig swig-devel boost-develpcap-devellibpcap-develmysql-devel db4-devel db4 libdb-develcompat-dbtcsh

  • install Emulabtools

  • Modify the VM as you want.

    • cleanup anything the post boot script did

    • cleanup hostname

    • cleanup sudoers

    • remove users if desired

    • remove public keys (from at least root)

    • chkconfigneucaoff

    • /etc/init.d/neucastop

  • Clean up VM

  • Boot a Centos VM on ExoGENI.


  • Turning tools into libraries

    Turning tools into libraries

    • ExoGENI has a number of tools, that are now being converted into libraries for developing more tools

    • Existing and newly-developed libraries

      • Written in Java

      • orca.ndl– NDL parsing and generation

        • Used by ORCA, Flukes, NDL converter

      • orca.ndl-conversion.core – Conversions between NDL and other formats (GENI RSpec, PG Rspec, graph representations)

      • ndlLib – easy application-level slice management abstractions– in progress

        • Integration with pubsub for status notifications

        • Slice visualization, graph layouts

    • Available from snapshot Maven repository

      • http://ci-dev.renci.org/nexus/content/repositories/geni-orca-snapshot/


    Registries

    Registries

    • Built on CouchDB

    • REST APIs

    • Registries for

      • Images

      • Stitchports

      • Actors


    Storage slivering

    Storage Slivering

    • Overview of architecture and implementation


    Storage architecture

    Storage Architecture


    Slices with storage

    Slices with storage


    Lun creation time estimates

    Lun Creation Time Estimates


    Exogeni stitching

    ExoGENI stitching

    • Unlike GENI stitching, ExoGENI stitching logic is

      • Integrated into ORCA `controller’ that speaks GENI AM API

        • Makes it easy to place requests for complex inter-domain topologies

      • Integrated with ORCA resource brokering/ticketing system

        • Provides reliable single-shot provisioning that (except when OSCARS on AL2S or ESnet decides to fall over)

    • We considered pulling it out of the controller and making a toolkit

      • Still in consideration, no immediate plans though


    Reasons to use exogeni native stitching

    Reasons to use ExoGENI native stitching

    • ExoGENI stitching

      • Any XO rack to any other XO rack transparently to the user

        • AL2S, ION, ESnet, BEN

      • Works with various corner cases

        • Internal label translations by transit providers

          • E.g. NICTA

        • Inability of AL2S to connect racks that share the same AL2S port

          • E.g. FIU to UFL

      • Multipoint VLAN support – create large Layer 2 domains

      • Storage stitching support – create large slivers of iSCSI storage

      • Stitch ports – connect slices to the outside world


    Ways to use it

    Ways to use it

    • GENI Tools (omni, Flack, GENI Portal):

      • Works with other rack types

      • Limited topology embedding support

    • ExoGENI native tools:

      • Expose full range of capabilities

        • Multi-point

        • Storage slivering and stitching

        • Slice modifications (elastic slices)

      • Use regular GENI credentials

      • Work only with ExoGENI racks

    • More information:

      • http://tinyurl.com/l5qsbwy

      • http://www.exogeni.net


    Thank you

    Thank you!

    • http://www.exogeni.net

    • http://wiki.exogeni.net


  • Login