exogeni tools and capabilities for developers
Download
Skip this Video
Download Presentation
ExoGENI : tools and capabilities for developers

Loading in 2 Seconds...

play fullscreen
1 / 21

ExoGENI : tools and capabilities for developers - PowerPoint PPT Presentation


  • 90 Views
  • Uploaded on

ExoGENI : tools and capabilities for developers. Ilya Baldin, Paul Ruth, Anirban Mandal , Jonathan Mills, Claris Castillo, Chris Heermann RENCI , UNC-CH Jeff Chase, Victor Orlikowski Duke University. What’s new for GEC20?. Image snapshot script Easy way to capture the ExoGENI image

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'ExoGENI : tools and capabilities for developers' - erv


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
exogeni tools and capabilities for developers

ExoGENI: tools and capabilities for developers

Ilya Baldin, Paul Ruth, AnirbanMandal, Jonathan Mills, Claris Castillo, Chris Heermann

RENCI, UNC-CH

Jeff Chase, Victor Orlikowski

Duke University

what s new for gec20
What’s new for GEC20?
  • Image snapshot script
    • Easy way to capture the ExoGENI image
    • Tested and ‘GPO approved’
    • Replaces ‘playpen’
  • Speaks-for in controllers
  • New libraries for building tools
plans for gec21
Plans for GEC21
  • Release ndlLib
    • Port Flukes to use ndlLib
    • Develop other applications with it
  • Image capture
    • Integrate with NEuca
    • Make it work for bare-metal nodes
    • Obsolete playpen – it was a temporary awkward solution.
more on what s next
More on what’s next
  • Part of upcoming ORCA 5.0
    • More storage slivering for
      • Cisco racks NetApp
      • New IBM racks and Dell Racks with server-based storage
      • With I/O isolation
      • Add support to attach to bare-metal nodes
    • Virtual wire support for OVS
      • To improve network performance isolation for VMs
    • Hybrid network support for switches
      • Ethernet VLANs for those who care about performance
      • OpenFlow for those who actually care about it
    • Fully dynamic slices
      • Add/remove links/nodes at runtime
    • Inter-slice stitching
overview
Overview
  • Details on image snapshot utility
  • Image portability with InstaGENI
  • Turning tools into libraries
  • ExoGENI Storage slivering
  • ExoGENI Stitching
exogeni image background
ExoGENI Image Background
  • Images defined by XML metadata file
    • Image (AMI) - mandatory
    • Kernel (AKI) - optional
    • Ramdisk (ARI) - optional
  • Hosted on HTTP server
    • Permanent server at your institution
    • Soon will work with DropBox and Google Drive
  • Slice requests require
    • URL to metadata file – goes into the ‘name’ in RSpec
    • Hash of metadata file – goes into the ‘vesrion’ in RSpec
exogeni image background metadata
ExoGENI Image Background: Metadata

ZFILESYSTEM

9efee8fa9c6705455c87a42477bd4ebd04d39c81

http://geni-images.renci.org/images/standard/centos/centos6.3-v1.0.10.tgz

KERNEL

5e76d034e27cb861275fc3d1bd9d97c3905ef5a2

http://geni-images.renci.org/images/standard/centos/kernels/vmlinuz-2.6.32-431.5.1.el6.x86_64

`

RAMDISK

c999527cc0d8739a3d3f38d48f03fa6c73151cca

http://geni-images.renci.org/images/standard/centos/kernels/initramfs-2.6.32-431.5.1.el6.x86_64.img

exogeni image snapshot
ExoGENI Image Snapshot
  • Snapshot of running ExoGENI VM
    • Creates: image, kernel, ramdisk, metadata
  • High level steps
    • Create/modify a VM – update OS, install needed packages
    • Run the script, make sure you have enough disk space
      • Use ExoGENI storage slivering to help here (next topic)
    • Copy the new image files to an http server
    • Insert metadata URL and hash into a request

More details on the ExoBlog:

http://www.exogeni.net/2014/06/creating-a-custom-image-from-an-existing-virtual-machine/

image portability with ig
Image portability with IG
  • Porting from EG to IG is a complicated and error-prone process
  • Porting from IG to EG has not yet been achieved
  • Ultimately image snapshot utility should work for IG nodes to capture image that works on EG
slide10
Log into your VM
    • Get even more coffee
    • Submit RSpec
    • Your RSpec “disk_image” is the “Metadata URL”
    • Create a new Rspec
  • Boot a VM
    • Click:
    • Click: Experimentation -> List ImageIDs
    • Toggle “Global” to “Yes”
    • Get Coffee / Wait for email from Emulab declaring victory
    • Click “Submit”
      • Set types to only “pcvm”
      • Set “EC2 [email protected]”:  i.e. [email protected]
      • Set “Filename”:  should be preset with project dir and “Description Name”.  Example: “/proj/GENISciShakedown/images/Centos6.3.ndz”
      • Set “OS Version”: Version of the image??? I’m not sure what this is for.
      • Set “Operating System”:   Linux
      • Set “Description”: choose a description
      • Set "Descriptor Name”:  choose a name
    • Click: “Import an Amazon EC2 Instance Image”
    • Click: Experimentation -> List ImageIDs
    • Log into www.emulab.net
  • Perform ec2 import
      • cat key to >> .ssh/authorized_keys”
      • Copy emulab keys into ExoGENI instance
      • Make user the “EC2 [email protected]" is in the shudders file (i.e.  [email protected] )
      • yum install ruby zip unzip sudo e2fsprogs rsync
    • setup for ec2 import 
      • edit /usr/local/etc/emulab/rc/rc.mounts and change this line:
      • Create a file called /etc/emulab/genvmtype and stick in one line that says “xen”
      • Allow sudo to run without a tty.  Add the following to the sudoers file:    "Defaults    !requiretty” 
      • Add  /users
      • Fix resov.conf and hostname
    • Install client side tools
    • Install pubsub
    • yum install perldhclientopensshmysql bison byacc flex boost gitmysql-server mysql-develmysql-embedded-develmysql-proxy-devellibtoolperl-Time-HiRes yum install python-develhttpd man httpd-devel dos2unix swig swig-devel boost-develpcap-devellibpcap-develmysql-devel db4-devel db4 libdb-develcompat-dbtcsh
  • install Emulabtools
  • Modify the VM as you want.
    • cleanup anything the post boot script did
    • cleanup hostname
    • cleanup sudoers
    • remove users if desired
    • remove public keys (from at least root)
    • chkconfigneucaoff
    • /etc/init.d/neucastop
  • Clean up VM
  • Boot a Centos VM on ExoGENI.
turning tools into libraries
Turning tools into libraries
  • ExoGENI has a number of tools, that are now being converted into libraries for developing more tools
  • Existing and newly-developed libraries
    • Written in Java
    • orca.ndl– NDL parsing and generation
      • Used by ORCA, Flukes, NDL converter
    • orca.ndl-conversion.core – Conversions between NDL and other formats (GENI RSpec, PG Rspec, graph representations)
    • ndlLib – easy application-level slice management abstractions– in progress
      • Integration with pubsub for status notifications
      • Slice visualization, graph layouts
  • Available from snapshot Maven repository
    • http://ci-dev.renci.org/nexus/content/repositories/geni-orca-snapshot/
registries
Registries
  • Built on CouchDB
  • REST APIs
  • Registries for
    • Images
    • Stitchports
    • Actors
storage slivering
Storage Slivering
  • Overview of architecture and implementation
exogeni stitching
ExoGENI stitching
  • Unlike GENI stitching, ExoGENI stitching logic is
    • Integrated into ORCA `controller’ that speaks GENI AM API
      • Makes it easy to place requests for complex inter-domain topologies
    • Integrated with ORCA resource brokering/ticketing system
      • Provides reliable single-shot provisioning that (except when OSCARS on AL2S or ESnet decides to fall over)
  • We considered pulling it out of the controller and making a toolkit
      • Still in consideration, no immediate plans though
reasons to use exogeni native stitching
Reasons to use ExoGENI native stitching
  • ExoGENI stitching
    • Any XO rack to any other XO rack transparently to the user
      • AL2S, ION, ESnet, BEN
    • Works with various corner cases
      • Internal label translations by transit providers
        • E.g. NICTA
      • Inability of AL2S to connect racks that share the same AL2S port
        • E.g. FIU to UFL
    • Multipoint VLAN support – create large Layer 2 domains
    • Storage stitching support – create large slivers of iSCSI storage
    • Stitch ports – connect slices to the outside world
ways to use it
Ways to use it
  • GENI Tools (omni, Flack, GENI Portal):
    • Works with other rack types
    • Limited topology embedding support
  • ExoGENI native tools:
    • Expose full range of capabilities
      • Multi-point
      • Storage slivering and stitching
      • Slice modifications (elastic slices)
    • Use regular GENI credentials
    • Work only with ExoGENI racks
  • More information:
    • http://tinyurl.com/l5qsbwy
    • http://www.exogeni.net
thank you
Thank you!
  • http://www.exogeni.net
  • http://wiki.exogeni.net
ad