Utility oriented cloud grid computing a vision hype and reality
Download
1 / 56

Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality - PowerPoint PPT Presentation

Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Gri d Computing and D istributed S ystems (GRIDS) Lab Dept. of Computer Science and Software Engineering The University of Melbourne, Australia www.gridbus.org www.buyya.com www.manjrasoft.com Dr. Rajkumar Buyya

Related searches for Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha

Download Presentation

Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality

Grid Computing and Distributed Systems (GRIDS) LabDept. of Computer Science and Software EngineeringThe University of Melbourne, Australiawww.gridbus.orgwww.buyya.comwww.manjrasoft.com

Dr. Rajkumar Buyya

Gridbus Sponsors


The GRIDS Lab @ Melbourne

R & D

Education

  • Youngest and one of the rapidly growing research labs in our School/University:

    • Founded in 2002

    • Houses 20+ researchers consisting of:

      • Research Fellows/PostDocs

      • Software Engineers

      • PhD candidates

      • Honours/Masters students

  • Funding

    • National and International organizations

    • Australian Research Council & DEST

    • Many industries (Sun, StorageTek, Microsoft, IBM, Microsoft)

  • University-wide collaboration:

    • Faculties of Science, Engineering, and Medicine

  • Many national and international collaborations.

    • Academics

    • Industries

  • Software:

    • Widely in academic and industrial users.

  • Publication:

    • My research team produces over 20% of our Dept’s research output.

+ Community Services: e.g., IEEE TC for Scalable Computing


Agenda

  • Introduction

    • Utility Networks and Grid Computing

    • Application Drivers and Various Types of Grid Services

  • Global Grids and Challenges

    • Security, resource management, pricing models, …

  • Service-Oriented Grid Architecture and Gridbus Solutions

    • Market-based Management, GMD, Grid Bank, Aneka

  • Grid Service Broker

    • Architecture, Design and Implementation

  • Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids

    • A Case Study in High Energy Physics

  • Summary and Conclusion


1969 – Leonard Kleinrock, ARPANET project

“As of now, computer networks are still in their infancy, but as they grow up and become sophisticated, we will probably see the spread of ‘computer utilities’, which, like present electric and telephone utilities, will service individual homes and offices across the country”

Computers Redefined

1984 – John Gage, Sun Microsystems

“The network is the computer”

2008 – David Patterson, U. C. Berkeley

“The data center is the computer. There are dramatic differences between of developing software for millions to use as a service versus distributing software for millions to run their PCs”

2008 – “Cloud is the computer” – Buyya!

“Computer Utilities” Vision: Implications of the Internet


Computing Paradigms and Attributes: Realizing the ‘Computer Utilities’ Vision

?

}

  • Web

  • Data Centres

  • Utility Computing

  • Service Computing

  • Grid Computing

  • P2P Computing

  • Market-Oriented Computing

  • Cloud Computing

+

  • -Ubiquitous access

  • -Reliability

  • Scalability

  • Autonomic

  • Dynamic discovery

  • Composability

  • -QoS

  • -SLA

  • - …

  • Trillion $ business

  • Who will own it?

Paradigms

Attributes/Capabilities


* Since Grids have been around for sometime (early 2000), do we have a unified vision of what Grids can do? * And did we make sufficient advances to turn vision of “computer utilities” into a reality?

- Let us take a look at views of

“industrial” practitioners & “academics”


“Industrial” vision of Grid computing

  • IBM

    • On Demand Computing

  • Microsoft

    • .NET

  • Oracle

    • 10g

  • Sun

    • N1 – Sun Grid Engine

  • HP

    • Adaptive Enterprise

  • Amazon

    • Elastic Compute Cloud Services

  • Manjrasoft

    • Aneka for building enterprise Grids and Clouds.


Most academics view: Cyberinfrastructure for conducting collaborative (e-)Science


database

How do Grids look like?A Bird Eye View of a Global Grid

Grid Information Service

Grid Resource Broker

Application

R2

R3

R4

R5

RN

Grid Resource Broker

R6

R1

Resource Broker

Grid Information Service


database

How do Grids look like?A Bird Eye View of a Global Grid

Grid Information Service

Grid Resource Broker

Application

R2

R3

R4

R5

RN

Grid Resource Broker

R6

R1

Resource Broker

Grid Information Service


How Are Grids Used?

Utility computing

High-performance computing

Collaborative design

Financial modeling

Collaborative data-sharing

High-energy physics

E-Business

Life sciences

Drug discovery

Data center automation

E-Science

Natural language processing

Business Intelligence

(Data Mining)


Agenda

  • Introduction

    • Utility Networks and Grid Computing

    • Application Drivers and Various Types of Grid Services

  • Global Grids and Challenges

    • Security, resource management, pricing models, …

  • Service-Oriented Grid Architecture and Gridbus Solutions

    • Market-based Management, GMD, Grid Bank, Aneka

  • Grid Service Broker

    • Architecture, Design and Implementation

  • Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids

    • A Case Study in High Energy Physics

  • Summary and Conclusion


Computational Economy

Security

Data locality

Resource Allocation

& Scheduling

Uniform Access

System Management

Resource Discovery

Application Construction

Network Management

Grid Challenges


Australia

Nimrod-G

Gridbus

DISCWorld

GrangeNet.

APACGrid

ARC eResearch

Brazil

OurGrid, EasyGrid

LNCC-Grid + many others

China

ChinaGrid – Education

CNGrid - application

Europe

UK eScience

EU Grids..

and many more...

India

Garuda

Japan

NAREGI

Korea...

N*Grid

Singapore

NGP

USA

Globus

TeraGrid

Cyberinfrasture

AutoMate

and many more...

Industry Initiatives

IBM On Demand Computing

HP Adaptive Computing

Sun N1

Microsoft - .NET

Oracle 10g

Amzon – Elastic Compute Cloud

Infosys – Enterprise Grid

Satyam – Business Grid

Manjrasoft – enterprise Clouds and Grids

and many more

Public Forums

Open Grid Forum

Conferences:

CCGrid

Grid

HPDC

E-Science

Some Grid Initiatives Worldwide

1.3 billion – 3 yrs

27 million

2? billion

120million – 5 yrs

450million – 5 yrs

486million – 5 yrs

1.3 billion (Rs)

1 billion – 5 yrs

http://www.gridcomputing.com


Open-Source Grid Middleware Projects

OurGrid

Slide by Hiro


Driving Theme:Community vs. Utility Grids


The Gridbus Project @ Melbourne:Enable Leasing of ICT Services on Demand

WWG

Gridbus

Pushes Grid computing into mainstream computing


Agenda

  • Introduction

    • Utility Networks and Grid Computing

    • Application Drivers and Various Types of Grid Services

  • Global Grids and Challenges

    • Security, resource management, pricing models, …

  • Service-Oriented Grid Architecture and Gridbus Solutions

    • Market-based Management, GMD, Grid Bank, Aneka

  • Grid Service Broker

    • Architecture, Design and Implementation

  • Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids

    • A Case Study in High Energy Physics

  • Summary and Conclusion


What do Grid players want & require?

  • Grid Service Consumers (GSCs): - minimize expenses, meet QoS

    • How do I express QoS requirements ?

    • How do I trade between timeframe & cost ?

    • How do I discover services and map jobs to meet my QoS needs?

    • How do I manage Grid dynamics and get my work done?

  • Grid Service Providers (GSPs):– maximise ROI

    • How do I decide service pricing models ?

    • How do I specify them ?

    • How do I translate them into resource allocations ?

    • How do I enforce them ?

    • How do I advertise & attract consumers ?

    • How do I do accounting and handle payments?

  • They need mechanisms, tools and technologies that help them in value expression, value translation, and value enforcement.


Service-Oriented Grid Architecture

Data Catalogue

Grid Bank

Information Service

Grid Market Services

Sign-on

HealthMonitor

Info ?

Grid Node N

Grid Explorer

Secure

ProgrammingEnvironments

Job Control

Agent

Grid Node1

Applications

Schedule Advisor

QoS

Pricing Algorithms

Trade Server

Trading

Trade Manager

Accounting

Resource

Reservation

Misc. services

Deployment Agent

JobExec

Resource Allocation

Storage

Grid Resource Broker

R1

R2

Rm

Core Middleware

Services

Grid Service Consumer

Grid Service Providers


CDB

PDB

Market-Oriented Grid Software: A union of Gridbus and other technologies

Grid

Applications

Science

Commerce

Engineering

Collaboratories

Grid Portals

APIs/Tools:

ExcellGrid

Workflow APIs

Task, Parametric, and Components Programming

MPI

User-LevelMiddleware

Grid Workflow Engine

Grid Scheduling:

Gridbus Resource Broker

Grid MarketDirectory

Grid Exchange & Federation

Globus

Unicore

Grid Storage Economy

GridBank

Core Grid

Middleware

Aneka Cloud (WS-based access + SLA

NorduGrid

XGrid

Grid Economy

JVM

Condor

PBS

SGE

Libra

Tomcat

.NET

Grid

Fabric

Software

Mac

Windows

Linux

AIX

IRIX

OSF1

Solaris

Grid

Fabric

Hardware

Worldwide Grid


Application Code

Explore data

1

Visual Application Composer

10

Results+Cost Info

2

GridResource Broker

Data Catalogue

5

4

Grid Info Service

12

6

3

ASP Catalogue

Grid Market Directory

9

7

Job

Results

8

Grid Service (GS)

(Globus)

Bill

Aneka

EC2

CPU

orPE

PE

GTS

11

GridbusGridBank

Resource Allocation

PE

GSP

(Accounting Service)

GSP

(e.g., IBM)

GSP

(e.g., Amazon)

GSP

(e.g., Microsoft)

On Demand Assembly of Services in Market-Oriented Grid Environments


On Demand Assembly of Services in Market-Oriented Grid Environments


Cloud Services

  • Infrastructure as a Services (IaaS)

    • CPU, Storage: Amazon.com et. al

  • Platform as a Services (PaaS)

    • Google App Engine, Microsoft Azure, andManjrasoft Aneka

  • Software as a Service (SaaS)

    • SalesForce.Com

Enterprise/Private Clouds

Clouds

Public/Internet Clouds


Software as a Service (SaaS)

e.g., ..SalesForce.com

Platform as a Service (PaaS)

e.g., ..Aneka

Infrastructure as a Service (IaaS)

e.g., Amazon, Nirvanix

Layered view of services within a Cloud stack


Aneka

A Software Platform for Building and Managing “Enterprise” Grids and Clouds


Aneka: A 3rd Gen enterprise Grid Technology  Cloud model


.NET based service-oriented platform for grid / cloud computing

Development and Run Time Environment

Includes Development and Management Tools

Suitable for

Development of Enterprise Grid / Cloud Applications

Grid / Cloud enabling legacy applications

Ideal for Corporate Developers, Software, SaaS, Hosting Vendors and Application / System Integrators

ANEKA – Product Overview (Alpha)

ANEKA Product Architecture


Enterprise/Private

Harness LAN connected resources

Application Development, Testing, Execution

Teaching and Learning

Sensitive applications

Public

Hosted by a 3rd party service provider owning a large Data Center (1000s of servers)

Offers subscription-based services to their shared infrastructure on “pay-as go” model.to many users from different organisations.

Amazon.com, Microsoft Azure

Aneka SDK + Execution Manger

Aneka Deployment Models

Enterprise/Private Clouds

Aneka

Public Clouds


Executor

Executor

Executor

Executor

Scheduler

ClientAgent

ClientAgent

Programming / Deployment Model

FIRST PRODUCT

Aneka: components

public DumbTask: ITask

{

public void Execute()

{

……

}

}

Aneka enterprise Cloud

for(int i=0; i<n; i++)

{

DumbTask task = newDumbTask();

app.SubmitExecution(task);

}

work units

internet

work units

Aneka Worker

Service

Aneka Manager

internet

Aneka Users


How does it solve the problem?

An Illustratioin

  • Divide the problem in to multiple small tasks and distribute them run in parallel on multiple computers within a Cloud.

Executor

Application

Manager

Manager / Executor

GThreads/Tasks


User scenario: GoFront(unit of China Southern Railway Group)

Aneka Maya Renderer

Use private Aneka Cloud

Case 2: Aneka Enterprise Cloud

Time (in hrs)

Case 1: Single Server

Using Maya Graphical Mode Directly

Single Server

Aneka Cloud

Aneka utilizes idle desktops (30) to decrease task time from days to hours

4 cores server

Application: Locomotive design CAD rendering

Raw Locomotive Design Files

(Using AutoDesk Maya)

GoFront Private Aneka Cloud

LAN network

(Running Maya Batch Mode on demand)


Aneka: How can get it?

  • Available to Download:

    • Software: www.manjrasoft.com

    • Manual: Setting up Cloud using your LAN-network computers

  • Teaching material

    • parallel and distributed computing and programming,

    • List of possible assignments for students

    • Possible Projects for Final year students..

  • Price – highly affordable 

    • = Fee you charge to 1 student (each year) and all students/teachers in entire college/university can use it!

  • Applications

    • Other Departments (Physics, Chemistry, Biology, Finance, Engineering) can use it for their applications.


On Demand Assembly of Services in Market-Oriented Grid Environments


Agenda

  • Introduction

    • Utility Networks and Grid Computing

    • Application Drivers and Various Types of Grid Services

  • Global Grids and Challenges

    • Security, resource management, pricing models, …

  • Service-Oriented Grid Architecture and Gridbus Solutions

    • Market-based Management, GMD, Grid Bank, Aneka

  • Grid Service Broker

    • Architecture, Design and Implementation

  • Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids

    • A Case Study in High Energy Physics

  • Summary and Conclusion


Grid Service Broker (GSB)

  • A resource broker for scheduling task farming data Grid applications with static or dynamic parameter sweeps on global Grids.

  • It uses computational economy paradigm for optimal selection of computational and data services depending on their quality, cost, and availability, and users’ QoS requirements (deadline, budget, & T/C optimisation)

  • Key Features

    • A single window to manage & control experiment

    • Programmable Task Farming Engine

    • Resource Discovery and Resource Trading

    • Optimal Data Source Discovery

    • Scheduling & Predications

    • Generic Dispatcher & Grid Agents

    • Transportation of data & sharing of results

    • Accounting


workload

Gridbus User Console/Portal/Application Interface

App, T, $, Optimization Preference

Gridbus Broker

Gridbus Farming Engine

Schedule Advisor

Trading Manager

RecordKeeper

Grid Dispatcher

Grid Explorer

TM TS

$

GE GIS, NWS

Core Middleware

Grid Info Server

RM & TS

G

$

Data

Catalog

Data

Node

C

$

U

G

Globus enabled node.

L

A

Amazon EC2/S3 Cloud.


Home Node/Portal

Gridbus

Broker

batch()

-PBS

-Condor

-SGE

-Aneka

-XGrid

fork()

Data Catalog

Globus

Aneka

Amazon EC2

SSH

Job manager

fork()

AMI

batch()

fork()

batch()

-PBS

-Condor

-SGE

-XGrid

-PBS

-Condor

-SGE

Gridbus

agent

Gridbus

agent

Gridbus Broker: Separating “applications” from “different” remote service access enablers and schedulers

Application Development Interface

Single-sign on security

Alogorithm1

SchedulingInterfaces

AlogorithmN

Plugin Actuators

Data Store

Access Technology

SRB

Grid FTP


Gridbus Services for eScience applications

  • Application Development Environment:

    • XML-based language for composition of task farming (legacy) applications as parameter sweep applications.

    • Task Farming APIs for new applications.

    • Web APIs (e.g., Portlets) for Grid portal development.

    • Threads-based Programming Interface

    • Workflow interface and Gridbus-enabled workflow engine.

    • … Grid Superscalar – in cooperation with BSC/UPC

  • Resource Allocation and Scheduling

    • Dynamic discovery of optional computational and data nodes that meet user QoS requirements.

  • Hide Low-Level Grid Middleware interfaces

    • Globus (v2, v4), SRB, Aneka, Unicore, and ssh-based access to local/remote resources managed by XGrid, PBS, Condor, SGE.


Click Here for Demo

Drug Design

Made Easy!


s


Agenda

  • Introduction

    • Utility Networks and Grid Computing

    • Application Drivers and Various Types of Grid Services

  • Global Grids and Challenges

    • Security, resource management, pricing models, …

  • Service-Oriented Grid Architecture and Gridbus Solutions

    • Market-based Management, GMD, Grid Bank, Aneka

  • Grid Service Broker

    • Architecture, Design and Implementation

  • Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids

    • A Case Study in High Energy Physics

  • Summary and Conclusion


Case Study: High Energy Physics and Data Grid

  • The Belle Experiment

    • KEK B-Factory, Japan

    • Investigating fundamental violation of symmetry in nature (Charge Parity) which may help explain “why do we have more antimatter in the universe OR imbalance of matter and antimatter in the universe?”.

    • Collaboration 1000 people, 50 institutes

    • 100’s TB data currently


Case Study: Event Simulation and Analysis

B0->D*+D*-Ks

  • Simulation and Analysis Package - Belle Analysis Software Framework (BASF)

  • Experiment in 2 parts – Generation of Simulated Data and Analysis of the distributed data

Analyzed 100 data files (30MB each) that were distributed among the five nodes within Australian Belle DataGrid platform.


Australian Belle Data Grid Testbed

VPACMelbourne


Belle Data Grid (GSP CPU Service Price: G$/sec)

G$4

NA

G$4

G$6

VPACMelbourne

G$2

Datanode


Belle Data Grid (Bandwidth Price: G$/MB)

32

33

36

G$4

31

30

34

NA

38

31

G$4

G$6

VPACMelbourne

G$2

Datanode


Deploying Application Scenario

  • A data grid scenario with 100 jobs and each accessing remote data of ~30MB

  • Deadline: 3hrs.

  • Budget: G$ 60K

  • Scheduling Optimisation Scenario:

    • Minimise Time

    • Minimise Cost

  • Results:


fleagle.ph.unimelb.edu.au

belle.anu.edu.au

belle.physics.usyd.edu.au

brecca-2.vpac.org

80

70

60

50

Number of jobs completed

40

30

20

10

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

Time (in mins.)

Time Minimization in Data Grids


fleagle.ph.unimelb.edu.au

belle.anu.edu.au

belle.physics.usyd.edu.au

brecca-2.vpac.org

100

90

80

70

60

50

Number of jobs completed

40

30

20

10

0

1

3

5

7

9

11

13

15

17

19

21

23

25

27

29

31

33

35

37

39

41

43

45

47

49

51

53

55

57

59

61

63

Time(in mins.)

Results : Cost Minimization in Data Grids


Observation


Agenda

  • Introduction

    • Utility Networks and Grid Computing

    • Application Drivers and Various Types of Grid Services

  • Global Grids and Challenges

    • Security, resource management, pricing models, …

  • Service-Oriented Grid Architecture and Gridbus Solutions

    • Market-based Management, GMD, Grid Bank, Aneka

  • Grid Service Broker

    • Architecture, Design and Implementation

  • Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids

    • A Case Study in High Energy Physics

  • Summary and Conclusion


Summary and Conclusion

  • Grids exploit synergies that result from cooperation of autonomous entities:

    • Resource sharing, dynamic provisioning, and aggregation at global level Great Science and Great Business!

  • Grids have emerged as enabler for Cyberinfrastructure that powers e-Science and e-Business applications.

  • SOA + Market-based Grid Management = Utility Grids

  • Grids allow users to dynamically lease Grid services at runtime based on their quality, cost, availability, and users QoS requirements.

    • Delivering ICT services as computing utilities.

  • Clouds are rapidly emerging, but more work is required

    • Federation of Clouds, Cloud Exchange, and Application Scaling


Convergence of Competing Paradigms/Communities Needed

?

}

  • Web

  • Data Centres

  • Utility Computing

  • Service Computing

  • Grid Computing

  • P2P Computing

  • Cloud Computing

  • Market-Oriented Computing

+

  • Ubiquitous access

  • Reliability

  • Scalability

  • Autonomic

  • Dynamic discovery

  • Composability

  • QoS

  • SLA

  • Trillion $ business

  • Who will own it?

Paradigms

Attributes/Capabilities


Thanks for your attention!

  • Are there any

    • Questions?

    • Comments/ Suggestions

We Welcome Cooperation in R&D and Business! http:/www.gridbus.org | www.Manjrasoft.com

rbuyya@unimelb.edu.au | raj@manjrasoft.com


ad
  • Login