1 / 47

GEC 12 DiCloud Tutorial

GEC 12 DiCloud Tutorial. GEC12 DiCloud Tutorial. University of Massachusetts Amherst November 2 nd , 2011. Project Summary. DiCloud enables GENI experiments to incorporate cloud computing resources Focuses on Amazon web services EC2 (VMs), EBS (block storage), S3 (object storage)

kenna
Download Presentation

GEC 12 DiCloud Tutorial

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GEC 12 DiCloud Tutorial GEC12 DiCloud Tutorial University of Massachusetts Amherst November 2nd, 2011

  2. Project Summary • DiCloud enables GENI experiments to incorporate cloud computing resources • Focuses on Amazon web services • EC2 (VMs), EBS (block storage), S3 (object storage) • How are cloud resources different from other GENI resources? • They cost money! • Requires active Amazon account and credit card • Require authentication with Amazon

  3. Project Summary • DiCloud tracks per GENI-user Amazon costs • Amazon charges can be complicated • Pay per VM time used • Pay for data transfer in and out • Pay for IOs to block devices • No current facilities for tracking per user costs • Enables controlled access to cloud resources • Stand-alone: setup your own DiCloud instance to track your own costs • GENI CF-mode: place “underneath” GENI CF, where AM pays the bills, but gives GENI users access

  4. Elastic Compute Cloud (EC2) • Rent VMs from Amazon “on-demand” • http://wwww.amazon.com/ec2

  5. Elastic Compute Cloud (EC2) • Rent VMs from Amazon “on-demand” • Pay per hour based on VM performance • Pay for data transfer in/out

  6. Elastic Block Store (EBS) • Rent block storage from Amazon • Pay per GB-month (currently $0.10) • Also pay per 1 million IO requests (currently $0.10)

  7. Simple Storage Service (S3) • Rent object storage from Amazon • Pay per GB-month • Pay for data transfer in/out

  8. Tutorial Overview • Setup DiCloud server for stand-alone mode • Access DiCloud through console • Learn how to create VMs, block devices, storage objects • Learn how to check our balance and set quotas • Access DiCloud through web portal • Learn how to create users and edit quotas • Learn how to give users access to cloud resources • Learn how to console actions through web portal • Time permitting • Use your Amazon resources in a simple GENI experiment using Gush

  9. Tutorial Step 1: Let’s get started! • Trac page with both setup and tutorial instructions • http://groups.geni.net/geni/wiki/DICLOUD/GEC12tutorial • Setup instructionsdone for you • Tutorial instructions step-by-step in slides • Download GEC12tutorial.txt file with commands • Pre-requisites • Download/start your Virtual Box image • Login as user: geni • Open a Firefox web browser • Open 3 terminal windows and go to ~/Tutorials/DiCloud • $ cd ~/Tutorials/DiCloud

  10. Tutorial Step 1a: Common Errors • Be careful cutting and pasting • Doesn’t work from Trac! • Sometimes even txt file inserts hidden characters • Some lines require edits; don’t cut and paste the newline • Make terminal windows big • Sometimes terminal gets corrupted when pasting a line that runs to the next line (especially when backspacing) • On the website • Remember to hit down before hitting enter if the browser auto-fills in a text box (from its history)

  11. Tutorial Step 2: Getting your Keys • We’ve created temporary keys for you to access Amazon using a UMass account • Run the following commands: • $ cd ~/Tutorials/DiCloud • $ ./key_install.shgeni.cs.umass.edu • Verify Success: geni@geni-tutorial:~/Tutorials/DiCloud$ ls -la ec2/keys/ total 40 drwxr-xr-x 2 genigeni 4096 2011-10-24 11:30 . drwxr-xr-x 6 genigeni 4096 2011-10-17 09:21 .. -rw------- 1 genigeni 120 2011-10-24 11:26 credentials.csv -rw-r--r-- 1 genigeni 91 2011-10-24 11:29 geni_aws_credentials.txt -rw-r--r-- 1 genigeni 1375 2011-10-24 11:26 geni_cert.pem -rw-r--r-- 1 www-data www-data 1375 2011-10-24 11:30 geni_cert_web.pem -rw------- 1 genigeni 1675 2011-10-24 11:26 geni-keypair.pem -rw-r--r-- 1 genigeni 400 2011-10-24 11:26 geni-keypair.pub -rw-r--r-- 1 genigeni 887 2011-10-24 11:26 geni_pk.pem -rw-r--r-- 1 www-data www-data 887 2011-10-24 11:30 geni_pk_web.pem

  12. Tutorial Step 3: DiCloud Server • Run following command to start DiCloud server: • $ cd ~/Tutorials/DiCloud • $ source ./environment • $ cd dicloud/bin • $ ./dicloud_server.sh • Verify Success 11:20:31,376 INFO root Reading properties from dicloud.properties 11:20:31,466 INFO root Starting RMI registry on port 1099 11:20:31,474 INFO root Registering DiCloudserver 11:20:33,253 INFO root Registering AWS accoutingservice 11:20:33,260 INFO root Registering EC2 handler 11:20:33,275 INFO root Registering S3 handler 11:20:33,289 INFO root Registering EBS handler 11:20:33,302 INFO root Waiting for console connections • Move terminal to background but keep DiCloud server running

  13. Tutorial Step 3: DiCloud Server • Switch to another terminal • $ cd ~/Tutorials/DiCloud • $ source ./environment • $ cd dicloud/bin • $ cat ec2pricing.properties cloudwatch=-0.015 network.in.per.gb=-.10 network.out.per.gb=-.15 s3.storage.per.gb=-.15 s3.put=-.00001 s3.get=-.000001 ebs.storage.per.gb=-.10 ebs.io.per.million=-.10 t1.micro,us-west-1=-.01 m1.small,us-west-1=-.042 m1.large,us-west-1=-.16 m1.xlarge,us-west-1=-.325 m2.xlarge,us-west-1=-.23 m2.2xlarge,us-west-1=-.571 ……..

  14. Tutorial Step 4a: DiCloudConsole • Now open a new terminal and start console: • $ cd ~/Tutorials/DiCloud • $ source ./environment • $ ./dicloud_console.sh • Verify Success geni@geni-tutorial:~/Tutorials/DiCloud$ ./dicloud_console.sh dicloud- not connected> • Now we are in the DiCloud console • Type “help” to see a list of the possible console commands

  15. Tutorial Step 4b: DiCloudConsole • Now connect to server and start creating resources • dicloud - not connected> connect localhost 1099 Connecting to //localhost:1099/dicloud/server dicloud- //localhost:1099> • dicloud- //localhost:1099> get balance Current balance is: $0.0 • dicloud - //localhost:1099> add credit 100 Current balance is: $0.0 New balance is: $100.0 • dicloud- //localhost:1099> get balance Current balance is: $100.0 • Next we are going to create an S3 storage object, an EC2 VM, and an EBS storage volume

  16. Tutorial Step 4c: S3 objects <rand> must be random (S3 namespace globally shared) • Create S3 storage object • dicloud - //localhost:1099> s3 create gec12-tutorial-<rand> us • dicloud - //localhost:1099> s3 put gec12-tutorial-<rand> test /home/geni/Tutorials/DiCloud/s3_test Uploading file /home/geni/Tutorials/DiCloud/s3_test as object test in bucket gec12-tutorial-<rand> • dicloud - //localhost:1099> s3 get gec12-tutorial-<rand> test /home/geni/Tutorials/DiCloud/test Downloading file /home/geni/Tutorials/DiCloud/test from object test in bucket gec12-tutorial-<rand> • dicloud - //localhost:1099> s3 delete gec12-tutorial-<rand> us • dicloud - //localhost:1099>get balance Current balance is: $99.924958980703

  17. Tutorial Step 4d: EC2 VMs • Create EC2 VM • dicloud - //localhost:1099> ec2 start /geni_cert.pem /geni_pk.pemami-013d6c44 1 m1.small us-west-1 geni-keypair i-970bb3d0 • dicloud- //localhost:1099> ec2 probe /geni_cert.pem /geni_pk.pemi-970bb3d0 us-west-1 ascii Instance Id: i-970bb3d0AMI Id: ami-013d6c44 Public name: ec2-50-18-35-162.us-west-1.compute.amazonaws.com Private name: ip-10-166-207-140.us-west-1.compute.internal Public IP: 10.166.207.140 Private IP: ebs Instance State: runningInstance Type: m1.small Key pair: geni-keypair Start time: 2011-10-24T18:02:43+0000 Availability zone: us-west-1a

  18. Tutorial Step 4e: EC2 VMs • Logging into your EC2 VM • Open another terminal window • $ ssh –i /geni-keypair.pemroot@ec2-50-18-35-162.us-west-1.compute.amazonaws.com __| __|_ ) _| ( / Amazon Linux AMI ___|\___|___| See /usr/share/doc/system-release/ for latest release notes. No packages needed for security; 10 packages available [ec2-user@ip-10-166-207-140 ~]$ • Next lets create an EBS storage volume and attach it to ourVM

  19. Tutorial Step 4f: EBS storage volume • Create EBS storage volume • dicloud - //localhost:1099> ebs create /geni_cert.pem /geni_pk.pem1 us-west-1 us-west-1<availability zone> vol-7f2ca91e • dicloud- //localhost:1099>ebsattach /geni_cert.pem /geni_pk.pemus-west-1 vol-7f2ca91e i-970bb3d0sdg attaching • Switch back to the EC2 VM’s terminal • [ec2-user@ip-10-166-207-140 ~]$ ls /dev/sdg /dev/sdg

  20. Tutorial Step 4g: Console Scripting • DiCloud’s console is scriptable • $ cd ~/Tutorials/DiCloud/dicloud/bin • $ nanogetbalance.txt connect localhost1099 get balance quit • $ ./dicloud_console_script.sh < getbalance.txt dicloud - not connected>Connecting to //localhost:1099/dicloud/server dicloud- //localhost:1099>Current balance is: $0.0 dicloud- //localhost:1099> • Also able to hook into DiCloud programmatically using Java

  21. Tutorial Step 4h: Clean-up • Detach and destroy EBS storage volume • dicloud- //localhost:1099> ebsdetach /geni_cert.pem /geni_pk.pemus-west-1 vol-7f2ca91e detaching • dicloud - //localhost:1099> ebs delete /geni_cert.pem/geni_pk.pemvol-7f2ca91e us-west-1 • Terminate EC2 VM instance • dicloud - //localhost:1099> ec2 stop /geni_cert.pem/geni_pk.pemi-970bb3d0 us-west-1 geni-keypair • dicloud - //localhost:1099> get balance Current balance is: $99.867958980703

  22. Tutorial Step 5: Web Portal • Open Firefox web browser • Go to http://localhost/vise/dicloud • Login as user:admin, pw:geni • First, lets create a new non-admin user

  23. Tutorial Step 5: Web Portal • Give the new user the name geni with pw geniuser • Change access rights to view billing history

  24. Tutorial Step 5: Web Portal • Now click on “change” under “Certificates” for user geni • Enter file names and click “Update user certificates” • cert file: /geni_cert_web.pem • key file: /geni_pk_web.pem • keypair name: geni-keypair

  25. Tutorial Step 5: Web Portal • Now logout and log back in as user “geni” • Then click on “Request EC2 resources” • Use ami-013d6c44 as the AMI id

  26. Tutorial Step 5: Web Portal • Now logout and log back in as user “geni” • Then click on “Request EC2 resources” • Use ami-013d6c44 as the AMI id

  27. Tutorial Step 5: Web Portal • Now go back to your user page • Click on status next to the VM you just created

  28. Tutorial Step 5: Web Portal • Now lets create an S3 bucket • Click on “Request new S3 buckets”

  29. Tutorial Step 5: Web Portal • Once back at the user page, put something in your bucket • Select the s3_test file in ~/Tutorials/DiCloud/s3_test

  30. Tutorial Step 5: Web Portal • Now get that same thing back from your bucket

  31. Tutorial Step 5: Web Portal • Now lets create an EBS volume and attach it to our VM • Click request new EBS volumes • Make it 1GB, in us-west-1, in us-west-1<availability num>

  32. Tutorial Step 5: Web Portal • Attach the volume to our VM • Click on attach, enter instance id, and sdg for device

  33. Tutorial Step 5: Web Portal • If we ssh into our VM we can see that the volume is now attached

  34. Tutorial Step 5: Web Portal Cleanup • EBS • Detach and delete EBS volume • S3 • Delete S3 storage bucket • For EC2….. • Lets create 1 more VM, so we can test our Gush experiments across 2 VMs • Take note of the IP addresses for each machine • ec2-50-18-30-192.us-west-1.compute.amazonaws.com • ec2-50-18-138-224.us-west-1.compute.amazonaws.com

  35. Tutorial Step 6: Using Gush • Setup keys and fix hostnames • $ cp ~/Tutorials/DiCloud/ec2/keys/geni-keypair.pem ~/.ssh/id_rsa • $ cp ~/Tutorials/DiCloud/ec2/keys/geni-keypair.pub ~/.ssh/id_rsa.pub • $ keychain ~/.ssh/id_rsa • $ sshroot@amazon.machine.ip “hostname amazon.machine.ip” • $ sshroot@amazon.machine2.ip “hostname amazon.machine2.ip” • Add EC2 VMs to Gush’s directory.xml file • $ cd ~/Tutorials/DiCloud/gush • $ nanodirectory.xml <?xml version="1.0" encoding="UTF-8"?> <gush> <resource_manager type="ssh"> <node hostname="amazon.machine.ip:15400" user=“root" group="local" /> </resource_manager> <resource_manager type="ssh"> <node hostname="amazon.machine2.ip:15400" user=”root” group="local" /> </resource_manager> </gush>

  36. Tutorial Step 6: Using Gush • Run Gush experiment • $ cd ~/Tutorials/DiCloud/gush • $ ./gush –P 15000 • gush> info nodes There are 2 known nodes: [ U ] root@ec2-184-72-17-161.us-west-1.compute.amazonaws.com:15400(pref=0) (Disconnected.) [ U ] root@ec2-50-18-78-110.us-west-1.compute.amazonaws.com:15400(pref=0) (Disconnected.) • gush> load simple.xml Project "simple" is selected.Experiment "simple" is selected. • gush> run << see next slide for correct output >> • gush> disconnect gush> root@ec2-50-18-78-110.us-west-1.compute.amazonaws.com:15400 has decided to leave the mesh. root@ec2-184-72-17-161.us-west-1.compute.amazonaws.com:15400 has decided to leave the mesh. • gush> quit

  37. Tutorial Step 6: Using Gush • Run Gush experiment • gush> run Starting experiment run. Running experiment simple... gush> The configuration matcher has finished matching.The resource allocator has finished successfully. root@ec2-184-72-17-161.us-west-1.compute.amazonaws.com:15400 has joined the mesh. root@ec2-50-18-78-110.us-west-1.compute.amazonaws.com:15400 has joined the mesh. The file transfer of Package to ec2-184-72-17-161.us-west-1.compute.amazonaws.com has been completed. The software installation of Package on ec2-184-72-17-161.us-west-1.compute.amazonaws.com was successful. The file transfer of Package to ec2-50-18-78-110.us-west-1.compute.amazonaws.com has been completed. The software installation of Package on ec2-50-18-78-110.us-west-1.compute.amazonaws.com was successful. root@ec2-184-72-17-161.us-west-1.compute.amazonaws.com:15400,2261: Hello World! Your experiment was successful root@ec2-50-18-78-110.us-west-1.compute.amazonaws.com:15400,2313: Hello World! Your experiment was successful! The experiment has ended.

  38. Tutorial Step 6: Using Gush with Orca • Run Gush experiment using Orca and DiCloud resources • $ cd ~/Tutorials/DiCloud/gush • $ cat omni_config [ setup omni_config with correct ProtoGENI credentials ] • $ nanodirectory.xml [ change directory.xml to add orca slice ] <?xml version="1.0" encoding="UTF-8"?> <gush> <resource_manager type="geni-orca"> <port_map slice="gush-orca-david" port="11000"/> </resource_manager> <resource_manager type="ssh"> <node hostname="ec2-50-18-28-196.us-west-1.compute.amazonaws.com:15400" user="root" group="local" /> </resource_manager> <resource_manager type="ssh"> <node hostname="ec2-204-236-143-92.us-west-1.compute.amazonaws.com:15400" user="root" group="local" /> </resource_manager> </gush>

  39. Tutorial Step 6: Using Gush with Orca • Add both Amazon and Orca keys to keychain • $ cd ~/.ssh/; keychain --clear • $ keychain ~/.ssh/id_rsa_geni; keychain ~/.ssh/id_rsa • $ cd ~/Tutorials/DiCloud/gush • $ cpsimple.xml simple-orca.xml; nano simple-orca.xml [ Add orca resources to simple.xml example ] <?xml version="1.0" encoding="utf-8"?> <gush> …. …. <num_hosts>4</num_hosts> </rspec> <software name="SimpleSoftwareName" /> <resources> <resource type="ssh" group="local"/> <resource type="geni-orca" group="gush-orca-david" /> </resources> …. …. </gush>

  40. Tutorial Step 6: Using Gush with Orca • Create orca-new.rspec • $ nano orca-new.rspec <?xml version="1.0" encoding="UTF-8"?> <rspec type="request" xsi:schemaLocation="http://www.protogeni.net/resources/rspec/2 http://www.protogeni.net/resources/rspec/2/request.xsd" xmlns:flack="http://www.protogeni.net/resources/rspec/ext/flack/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns=http://www.protogeni.net/resources/rspec/2> <node client_id="geni1" component_manager_id="urn:publicid:IDN+acisrencivmsite+authority+cm"> <sliver_type name="raw-pc"> <disk_image name="http://geni-images.renci.org/images/gush/gush-deb5-i386.xml" version="25f53b64cfe44dd1604447f04b7b533bb67dea1e" /> </sliver_type> <interface client_id="geni1:0"> <ip address="172.16.1.1" netmask="255.255.0.0" /> </interface></node> <node client_id="geni2" component_manager_id="urn:publicid:IDN+acisrencivmsite+authority+cm"> <sliver_type name="raw-pc"> <disk_image name="http://geni-images.renci.org/images/gush/gush-deb5-i386.xml" version="25f53b64cfe44dd1604447f04b7b533bb67dea1e" /> </sliver_type> <interface client_id="geni2:0" > <ip address="172.16.1.2" netmask="255.255.0.0" /> </interface></node> <link client_id="center"> <interface_refclient_id="geni1:0" /> <interface_refclient_id="geni2:0" /> </link> </rspec>

  41. Tutorial Step 6: Using Gush with Orca • Create Orca slice and slivers; check status • $ helper-scripts/handle-geni.py -n -f pg -a https://www.emulab.net/protogeni/xmlrpc/am createslice gush-orca-david INFO:omni:Loadingconfig file omni_config INFO:omni:Usingcontrol framework pg Created slice with Name gush-orca-david, URN urn:publicid:IDN+emulab.net+slice+gush-orca-david • $ helper-scripts/handle-geni.py -n -f pg -a https://geni.renci.org:11443/orca/xmlrpccreatesliver gush-orca-david orca-new.rspec INFO:omni:Loadingconfig file omni_config INFO:omni:Usingcontrol framework pg Asked https://geni.renci.org:11443/orca/xmlrpc to reserve resources. Result: Request id: 8eaacd08-4f4b-4ac6-be0a-ce78dbd77893[ Slice UID: 4e8cec5d-a28a-49df-a485-0a03c46f3a95 | Reservation UID: f3cc03c5-6902-4fac-9f96-d6f94d1a43a1 | Resource Type: acisrencivmsite.vm | Resource Units: 1 ] [ Slice UID: 4e8cec5d-a28a-49df-a485-0a03c46f3a95 | Reservation UID: 2d6fc0e5-77bc-41a9-9c93-7e2d46134b9a | Resource Type: acisrencivmsite.vm | Resource Units: 1 ] [[ Slice UID: 4e8cec5d-a28a-49df-a485-0a03c46f3a95 | Reservation UID: 02db97c4-1ff1-492d-9c51-5736061682c0 | Resource Type: acisrencivmsite.vlan | Resource Units: 1 ]

  42. Tutorial Step 6: Using Gush with Orca • Create Orca slice and slivers; check status (cont’d) • $ helper-scripts/handle-geni.py -n -f pg -a https://geni.renci.org:11443/orca/xmlrpcsliverstatus gush-orca-david <?xml version="1.0"?> <gush> <slice name="gush-orca-david"> <expires>99999</expires> </slice> </gush> • $ helper-scripts/handle-geni.py -n -f pg -a https://geni.renci.org:11443/orca/xmlrpcsliverstatus gush-orca-david <?xml version="1.0"?> <gush> <slice name="gush-orca-david"> <expires>99999</expires> <node name="root@152.54.10.11" port="22"/> <node name="root@152.54.10.10" port="22"/> </slice> </gush>

  43. Tutorial Step 6: Using Gush with Orca • Must fix hostnames as before • $ sshroot@152.54.10.11 “hostname 152.54.10.11” • $ sshroot@152.54.10.10 “hostname 152.54.10.10”

  44. Tutorial Step 6: Using Gush with Orca • Run Gush experiment as before • $ ./gush –P 15000 • gush> Gush has learned about the slice gush-orca-david. • gush> Updated information on the slice gush-orca-david • gush> info nodes There are 4 known nodes: [ P ] root@152.54.10.10:11022(pref=0) (Disconnected.) [ P ] root@152.54.10.11:11022(pref=0) (Disconnected.) [ U ] root@ec2-204-236-143-92.us-west-1.compute.amazonaws.com:15400(pref=0) (Disconnected.) [ U ] root@ec2-50-18-28-196.us-west-1.compute.amazonaws.com:15400(pref=0) (Disconnected.) • gush> load simple-orca.xml Project "simple" is selected. Experiment "simple" is selected. • gush> run [ output on next slide ]

  45. Tutorial Step 6: Using Gush with Orca • Run Gush experiment as before • gush> run Starting experiment run. Running experiment simple... gush> The configuration matcher has finished matching. The resource allocator has finished successfully. root@ec2-50-18-28-196.us-west-1.compute.amazonaws.com:15400 has joined the mesh. root@ec2-204-236-143-92.us-west-1.compute.amazonaws.com:15400 has joined the mesh. The file transfer of Package to ec2-204-236-143-92.us-west-1.compute.amazonaws.com has been completed. The software installation of Package on ec2-204-236-143-92.us-west-1.compute.amazonaws.com was successful. The file transfer of Package to ec2-50-18-28-196.us-west-1.compute.amazonaws.com has been completed. The software installation of Package on ec2-50-18-28-196.us-west-1.compute.amazonaws.com was successful. root@152.54.10.10:11022 has joined the mesh. The file transfer of Package to 152.54.10.10 has been completed. The software installation of Package on 152.54.10.10 was successful. root@152.54.10.11:11022 has joined the mesh. The file transfer of Package to 152.54.10.11 has been completed. The software installation of Package on 152.54.10.11 was successful. root@152.54.10.10:11022,2053: Hello World! Your experiment was successful! root@152.54.10.11:11022,2028: Hello World! Your experiment was successful! root@ec2-50-18-28-196.us-west-1.compute.amazonaws.com:15400,3220: Hello World! Your experiment was successful!root@ec2-204-236-143-92.us-west-1.compute.amazonaws.com:15400,2281: Hello World! Your experiment was successful! The experiment has ended.

  46. Tutorial Step 6: Using Gush with Orca • Disconnect from gush and delete sliver • gush> disconnect gush> root@152.54.10.10:11022 has decided to leave the mesh. root@152.54.10.11:11022 has decided to leave the mesh. root@ec2-204-236-143-92.us-west-1.compute.amazonaws.com:15400 has decided to leave the mesh. root@ec2-50-18-28-196.us-west-1.compute.amazonaws.com:15400 has decided to leave the mesh. • gush> quit • $ helper-scripts/handle-geni.py -n -f pg -a https://geni.renci.org:11443/orca/xmlrpcdeletesliver gush-orca-david INFO:omni:Loadingconfig file omni_config INFO:omni:Usingcontrol framework pg Deleted sliver urn:publicid:IDN+emulab.net+slice+gush-orca-david on unspecified_AM_URN at https://geni.renci.org:11443/orca/xmlrpc

  47. Tutorial Step 6: Terminating EC2 VMs • Go back to the Firefox browser and terminate your VMs

More Related