Amazon Web Services

1. Amazon Web Services Justin DeBrabant CIS 570 - Advanced Systems - Fall 2013

2. Cloud Computing • A “pay what you use” service for hardware/software resources • analogy: utilities • Amazon Web Services (AWS) is a cloud computing provider • there are many others CIS 570 - Advanced Systems - Fall 2013

3. Benefits of Cloud Computing • elasticity • scale out or scale up easily* • *can be difficult depending on your application • scale down if demand decreases • no upfront capital • IT costs can be high, prohibitively so for a startup CIS 570 - Advanced Systems - Fall 2013

4. *Slide Courtesy Jeff Barr, Amazon CIS 570 - Advanced Systems - Fall 2013

5. Available Services • S3 (Simple Storage Service) • store and retrieve data • EC2 (Elastic Computer Cloud) • scalable compute capacity • EBS (Elastic Block Store) • predictable storage volumes that can be attached to a running EC2 instance • highly available, highly reliable CIS 570 - Advanced Systems - Fall 2013

6. Available Services (cont’d) • Databases • Relational: RDS (Relational Database Service) • NoSQL: DynamoDB • Scale-Out Caching: ElastiCache • EMR (Elastic MapReduce) • Hadoop-based • uses S3 and EC2 CIS 570 - Advanced Systems - Fall 2013

7. Other Important Terms vCPU (virtual CPU) – 1 hardware thread ECU – Amazon’s compute unit instance – EC2 virtual machine instance storage – amount of persistent storage for an instance AMI (Amazon Machine Image) – bundled software to be deployed on an instance CIS 570 - Advanced Systems - Fall 2013

8. Various Storages • S3 • available to instances within same region • analogy: DropBox • Instance Storage • available during the life of a specific instance, not shared across instances • analogy: local disk • EBS • persistent block storage that can be added to local instances, but is preserved across instance lifetimes • can be attached to only one running instance • analogy: removable storage media (e.g. external hard drive) CIS 570 - Advanced Systems - Fall 2013

9. AWS Regions • Amazon maintains different clusters worldwide • helps provide locality as well as improved geographic redundancy • each AWS service is available in specific regions • e.g. S3 • us-east-1 (Northern Virginia) • us-west-1 (Northern California) • us-west-2 (Oregon) • eu-west-1 (Ireland) • ap-southeast-1 (Singapore) • ap-southeast-2 (Sydney) • ap-northeast-1 (Tokyo) • sa-east-1 (Sao Paolo) • http://docs.aws.amazon.com/general/latest/gr/rande.html CIS 570 - Advanced Systems - Fall 2013

10. S3 • data organized into buckets • buckets names must be unique across S3 • data in bucket can be organized any way you wish • direct mapping between buckets and AWS subdomains • e.g. bucket name: mybucket, file: myfile.txt • http://mybucket.aws.amazon.com/myfile.txt • must specify region of bucket CIS 570 - Advanced Systems - Fall 2013

11. S3 (cont’d) • Guarantees: • 99.999999999% durability • 99.99% availability • designed to be recoverable against two simultaneous data losses • SLAs • if above guarantees are not met, partial refunds are given CIS 570 - Advanced Systems - Fall 2013

12. S3 Pricing • storage used • pay per GB • tiered, firstTB is most expensive • can reduce cost by reducing redundancy • http://aws.amazon.com/s3/pricing/ • data transfer • all data transfer in is free • data transfer out is not • pay per GB CIS 570 - Advanced Systems - Fall 2013

13. EC2 • machines classified as instances • each instance is a virtual machine • every instance type has different resources • vCPU • ECU • network bandwidth • memory • instance storage CIS 570 - Advanced Systems - Fall 2013

14. Instance Families • General Purpose • balanced, all-purpose machines with up to 8 vCPUs and 30 GB of memory • Memory Optimized • up to 244 GB of memory • Compute Optimized • up to 32 vCPUs and 10GBit network perf. • Storage Optimized • SSD and big disk backed instance storage CIS 570 - Advanced Systems - Fall 2013

15. Instance Software • Instances are virtual machines, and can support a wide range of operating systems • Linux (free) • Windows (have to pay for additional license) • Can create or use existing AMIs • prepackaged software bundle • includes OS and key software • e.g Oracle, MySQL, Apache WebServer • for this class, we’ll use existing AMIs CIS 570 - Advanced Systems - Fall 2013

16. Buying Instances • On-Demand Instances • request an instance, use it, terminate it • Reserved Instances • akin to buying a machine • permanently own instance • Spot Instances • bid for unused EC2 capacity • price can vary greatly • can set maximum bid and wait for available machines at or below that price CIS 570 - Advanced Systems - Fall 2013

17. On-Demand Instance Pricing • billed per instance hour • rounded up at per instance granularity! • e.g. 2 instance for 20 mins = 2 instance hours • free tier • 750 hours with micro instance • 30 GB EBS storage • 1 GB of data transfer • http://aws.amazon.com/ec2/pricing/ CIS 570 - Advanced Systems - Fall 2013

18. EMR • easy interface to launch Hadoop jobs using EC2 • just upload runnable jar containing Hadoop job • pricing • standard instance pricing + EMR instance pricing • e.g. 8 node cluster of m1.small for 1 hour • (8 * .06) + (8 * .015) CIS 570 - Advanced Systems - Fall 2013

19. Accessing AWS Resources • Console • easy interactive GUI • can launch EC2 instances, create S3 buckets, and start EMR jobs (among other things) • demo! • CLI (command line interface) • must install tools • http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/SettingUp_CommandLine.html CIS 570 - Advanced Systems - Fall 2013

20. Accessing a Running Instance • use SSH • must have private key file for key specified at instance launch • ssh –i <private key> ec2-user@public-dns • once SSH’d in, can use like any other machine • use SCP to copy in and out files • Warning: will be stored on ephemeral instance storage! CIS 570 - Advanced Systems - Fall 2013

21. Questions? CIS 570 - Advanced Systems - Fall 2013