Introduction
This presentation is the property of its rightful owner.
Sponsored Links
1 / 42

Introduction PowerPoint PPT Presentation


  • 64 Views
  • Uploaded on
  • Presentation posted in: General

Introduction. Dr. Ying Lu [email protected] Schorr Center 104. CSCE990 Advanced Distributed Systems Seminar. Some lecture notes are based on slides created by Dr. Zahorjan at Univ. of Washington, Dr. Konev at Univ. of Liverpool, Steve Crouch at Software Sustainability Institute,

Download Presentation

Introduction

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Introduction

Introduction

Dr. Ying Lu

[email protected]

Schorr Center 104

CSCE990 Advanced Distributed Systems Seminar


Introduction 3348019

Some lecture notes are based on slides created by

Dr. Zahorjan at Univ. of Washington,

Dr. Konev at Univ. of Liverpool,

Steve Crouch at Software Sustainability Institute,

Petru Eles at Linköpings University, and

Dr. Majd F. Sakr, Mohammad Hammoud, Vinay Kolar at CMU

I have modified them and added new slides

Giving credit where credit is due:


Types of distributed systems

Types of Distributed Systems?


Cloud grid utility computing

‘Cloud’ & ‘Grid’ – Utility Computing?

The Grid…

The Cloud…

j.o.h.n walker

charles.frith

Is it really like the grid?

Is it more like a fog?

But… they’re both about providing access to compute and data resources


The problem

The Problem

Basically, want to run compute/data intensive task

Don’t have enough resources to run job locally

At least, to return results within sensible timeframe

Would like to use another, more capable resource


Distributed computing in olden times

Distributed Computing in Olden Times

Small number of ‘fast’ computers

Very expensive

Centralized

Used nearly all the time

Time allocations for users

Not updated often

Michael L. Umbricht and Carl R. Friend

  • Punched cards

    • Wait time huge

    • MailNet, SneakerNet, etc…

  • Mainframes

  • Cray-1 1976 - $8.8 million, 160 megaflops, 8MB memory

Univac 1710

brewbooks

Cray X-MP(Cray -1 successor)


It s about scaling up

It’s About Scaling Up…

Compute and data – you need more, you go somewhere else to get it

  • Then… the march towards localization of computation, the Personal Computer

  • Computational Science develops in laboratories

  • Is this changing again?

Images: nasaimages, Extra Ketchup, Google Maps, Dave Page


What is cloud computing

What is Cloud Computing?

Many ways to define it i.e. one for every supplier of ‘cloud’

Key characteristics:

On demand, dynamic allocation of resources – ‘elasticity’

Abstraction

Self-managed

Billed for what you use e.g. in terms of CPU, storage

Standardized interfaces e.g. OCCI

… it’s more like an electricity grid than the Grid


How does it deliver

How Does it Deliver?

Cloud computing can deliver at any of these levels

These levels are often blurred and routinely disputed!

Resources provided on demand

Internet

End user/Customer

Developer/ Service Provider


Iaas infrastructure as a service

IaaS – Infrastructure as a Service

You get access to (usually) virtualised hardware

Servers, storage, networking

Operating system

Responsible for managing OS, middleware, runtime, data, application (development)

e.g. Amazon EC2


Amazon ec2 the idea

Amazon EC2 – The Idea

‘Elastic Computing’

Sign up

Select & configure virtualized resources

Emulated OS: RHEL, OEL, Windows Server, OpenSolaris, Fedora, Ubuntu, Debian, SUSE, Gentoo, Amazon Linux AMI

Infrastructure:

Data: IBM DB2, IBM Informix, Microsoft SQL, MySQL, Oracle

Web Hosting: Apache HTTP, IIS/Asp.NET, IBM WebSphere

Batch Processing: Hadoop, Condor, Open MPI

Newer addition - development environments:

IBM sMash, Ruby on Rails, Jboss Enterprise Application Platform

Moving towards PaaS! (Already there?)

Additional web services

S3: Simple Storage Solution – transfer data in/out, 1 byte to 5 TB

SQS: Simple Queue Service


Amazon ec2 pricing

Amazon EC2: Pricing

Free (at the start!):

Run single Amazon Micro Instance for a year

750 hours of EC2, 750 hours of Elastic Load Balancing plus 15 GB data processing

15 GB bandwidth in/out across all services

On demand instances:

Pay per hour, no long-term commitment

From $0.025/hour -> $0.76/hour

Reserved instances:

Upfront payment, with discount per hour

From $227/year + $0.01/hour -> $1820/year + $0.32/hour

Spot instances:

Bid for unused EC2 capacity:

Spot Price fluctuates with supply/demand, if bid over Spot Price, you get it

From $0.007/hour -> $0.68/hour


Ec2 application example

EC2 Application Example

  • Peter Harkins, a Senior Engineer at The Washington Post, used 200 EC2 instances (1,407 server hours) to convert 17,481 pages of Hillary Clinton’s travel documents into a form more friendly to use on the WWW within nine hours after they were released*

    *http://aws.amazon.com/solutions/case-studies/washington-post/


Paas platform as a service

PaaS – Platform as a Service

You get integrated development environment

e.g. application design, testing, deployment, hosting, frameworks for database integration, storage, app versioning, etc.

Develop applications on top

Responsible for managing data, application (development)

e.g. Google App Engine


Google app engine the idea

Google App Engine: The Idea

Sign up via Google Accounts

Develop App Engine web applications locally using SDK – emulates all services

Includes tool to upload application code, static files and config files

Can ‘version’ your web application instances

Apps run in a Java/Python ‘sandbox’

Automatic scaling and load balancing – abstract across underlying resources


Google app engine pricing

Google App Engine: Pricing

Free within a quota:

500MB storage, 5 million page views a month (~6.5 CPU hours, 1GB)

10 applications/developer

Billed model:

Each app $8/user (max $1000) a month

For each app:


Saas software as a service

SaaS – Software as a Service

Top layer consumed directly by end user – the ‘business’ functionality

Application software provided, you configure it (more or less)

Various levels of maturity:

Level 1: each customer has own customised version of application in own instance

Level 2: all instances use same application code, but configured individually

Level 3: single instance of application across all customers

Level 4: multiple customers served on load-balanced ‘farm’ of identical instances

Levels 3 & 4: separate customer data!

e.g. Gmail, Google Sites, Google Docs, Facebook


Summary of provision

Summary of Provision

Application Migration – adopt the level you want


Cloud open standards

Cloud Open Standards

Implementations typically have proprietary standards and interfaces

Vendors like this – often locked in to one implementation

Community ‘push’ towards open cloud standards:

Open Grid Forum (OGF) – Open Cloud Computing Interface (OCCI)

Distributed Management Task Force (DMTF) – Open Virtualisation Format (OVF)


Why should you study distributed systems

Why should you Study Distributed Systems?


Definition of a distributed system

Definition of a Distributed System

A distributed system is:

A collection of independent computers that appear to its users as a single coherent system (Tanenbaum book)

One in which components located at networked computers communicate and coordinate their actions only by passing messages (Coulouris book)


Why distributed systems

Why Distributed Systems?

Scale

Processing

Data

Diversity in Application Domains

Collaboration

Cost


Why distributed systems1

Why Distributed Systems?

Big data continues to grow:

In mid-2010, the information universe carried 1.2 zettabytes and 2020 predictions expect nearly 44 times more at 35 zettabytes coming our way.

Applications are becoming data-intensive.


Why distributed systems2

Why Distributed Systems?

Individual computers have limited resources compared to scale of current day problems & application domains:

Caches and Memory:

16KB- 64KB, 2-4 cycles

512KB- 8MB, 6-15 cycles

4MB- 32MB, 30-50 cycles

1GB- 4GB, 300+ cycles


Why distributed systems3

Why Distributed Systems?

Hard Disk Drive:

Limited capacity

Limited number of channels

Limited bandwidth


Why distributed systems4

Why Distributed Systems?

  • Processor:

  • The number of transistors that can be integrated on a single die has continued to grow at Moore’s pace.

  • Chip Multiprocessors (CMPs) are now available

P

P

P

P

P

L1

L1

L1

L1

L1

Interconnect

L2

L2 Cache

A single Processor Chip

A CMP


Why distributed systems5

Why Distributed Systems?

  • Processor (cont’d):

  • Up until a few years ago, CPU speed grew at the rate of 55% annually, while the memory speed grew at the rate of only 7% [H & P].

P

P

P

P

P

L1

L1

L1

L1

L1

Interconnect

L2

L2 Cache

Memory

Memory

P

Processor-Memory speed gap

M


Why distributed systems6

Why Distributed Systems?

  • Even if 100s or 1000s of cores are placed on a CMP, it is a challenge to deliver input data to these cores fast enough for processing.

10000 seconds (or 3 hours) to load data

P

P

P

P

L1

L1

L1

L1

Interconnect

L2 Cache

A Data Set

of 4 TBs

Memory

4 100MB/S IO Channels


Why distributed systems7

Why Distributed Systems?

Only 3 minutes to load data

P

P

100

Machines

Splits

L1

L1

L2

L2

Memory

Memory

A Data Set (data)

of 4 TBs


Requirements

Requirements

  • But this requires:

    • A way to express the problem as parallel processes and execute them on different machines (Programming Models and Concurrency).

    • A way for processes on different machines to exchange information (Communication).

    • A way for processes to cooperate, synchronize with one another and agree on shared values (Synchronization).

    • A way to enhance reliability and improve performance (Consistency and Replication).


Requirements1

Requirements

  • But this requires (Cont.):

    • A way to recover from partial failures (Fault Tolerance).

    • A way to secure communication and ensure that a process gets only those access rights it is entitled to (Security).

    • A way to extend interfaces so as to mimic the behavior of another system, reduce diversity of platforms, and provide a high degree of portability and flexibility (Virtualization)


Course objective

Course Objective

  • This is a course on advanced distributed systems, where we will understand the state of the art in distributed systems, in particular, data-intensive distributed computing systems, and how and why we got there and how to engage in systems research.


  • Login