Utility oriented cloud grid computing a vision hype and reality
Download
1 / 56

Utility- Oriented Cloud Grid Computing A Vision - PowerPoint PPT Presentation


  • 771 Views
  • Updated On :

Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Gri d Computing and D istributed S ystems (GRIDS) Lab Dept. of Computer Science and Software Engineering The University of Melbourne, Australia www.gridbus.org www.buyya.com www.manjrasoft.com Dr. Rajkumar Buyya

Related searches for Utility- Oriented Cloud Grid Computing A Vision

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Utility- Oriented Cloud Grid Computing A Vision' - paul


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Utility oriented cloud grid computing a vision hype and reality l.jpg

Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality

Grid Computing and Distributed Systems (GRIDS) LabDept. of Computer Science and Software EngineeringThe University of Melbourne, Australiawww.gridbus.orgwww.buyya.comwww.manjrasoft.com

Dr. Rajkumar Buyya

Gridbus Sponsors


The grids lab @ melbourne l.jpg
The GRIDS Lab @ Melbourne

R & D

Education

  • Youngest and one of the rapidly growing research labs in our School/University:

    • Founded in 2002

    • Houses 20+ researchers consisting of:

      • Research Fellows/PostDocs

      • Software Engineers

      • PhD candidates

      • Honours/Masters students

  • Funding

    • National and International organizations

    • Australian Research Council & DEST

    • Many industries (Sun, StorageTek, Microsoft, IBM, Microsoft)

  • University-wide collaboration:

    • Faculties of Science, Engineering, and Medicine

  • Many national and international collaborations.

    • Academics

    • Industries

  • Software:

    • Widely in academic and industrial users.

  • Publication:

    • My research team produces over 20% of our Dept’s research output.

+ Community Services: e.g., IEEE TC for Scalable Computing


Agenda l.jpg
Agenda

  • Introduction

    • Utility Networks and Grid Computing

    • Application Drivers and Various Types of Grid Services

  • Global Grids and Challenges

    • Security, resource management, pricing models, …

  • Service-Oriented Grid Architecture and Gridbus Solutions

    • Market-based Management, GMD, Grid Bank, Aneka

  • Grid Service Broker

    • Architecture, Design and Implementation

  • Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids

    • A Case Study in High Energy Physics

  • Summary and Conclusion


Computer utilities vision implications of the internet l.jpg

1969 – Leonard Kleinrock, ARPANET project

“As of now, computer networks are still in their infancy, but as they grow up and become sophisticated, we will probably see the spread of ‘computer utilities’, which, like present electric and telephone utilities, will service individual homes and offices across the country”

Computers Redefined

1984 – John Gage, Sun Microsystems

“The network is the computer”

2008 – David Patterson, U. C. Berkeley

“The data center is the computer. There are dramatic differences between of developing software for millions to use as a service versus distributing software for millions to run their PCs”

2008 – “Cloud is the computer” – Buyya!

“Computer Utilities” Vision: Implications of the Internet


Computing paradigms and attributes realizing the computer utilities vision l.jpg
Computing Paradigms and Attributes: Realizing the ‘Computer Utilities’ Vision

?

}

  • Web

  • Data Centres

  • Utility Computing

  • Service Computing

  • Grid Computing

  • P2P Computing

  • Market-Oriented Computing

  • Cloud Computing

+

  • -Ubiquitous access

  • -Reliability

  • Scalability

  • Autonomic

  • Dynamic discovery

  • Composability

  • -QoS

  • -SLA

  • - …

  • Trillion $ business

  • Who will own it?

Paradigms

Attributes/Capabilities


Let us take a look at views of industrial practitioners academics l.jpg

* Since Grids have been around for sometime (early 2000), do we have a unified vision of what Grids can do? * And did we make sufficient advances to turn vision of “computer utilities” into a reality?

- Let us take a look at views of

“industrial” practitioners & “academics”


Industrial vision of grid computing l.jpg
“Industrial” vision of Grid computing we have a unified vision of what Grids can do?

  • IBM

    • On Demand Computing

  • Microsoft

    • .NET

  • Oracle

    • 10g

  • Sun

    • N1 – Sun Grid Engine

  • HP

    • Adaptive Enterprise

  • Amazon

    • Elastic Compute Cloud Services

  • Manjrasoft

    • Aneka for building enterprise Grids and Clouds.



How do grids look like a bird eye view of a global grid l.jpg

database collaborative (e-)Science

How do Grids look like?A Bird Eye View of a Global Grid

Grid Information Service

Grid Resource Broker

Application

R2

R3

R4

R5

RN

Grid Resource Broker

R6

R1

Resource Broker

Grid Information Service


How do grids look like a bird eye view of a global grid10 l.jpg

database collaborative (e-)Science

How do Grids look like?A Bird Eye View of a Global Grid

Grid Information Service

Grid Resource Broker

Application

R2

R3

R4

R5

RN

Grid Resource Broker

R6

R1

Resource Broker

Grid Information Service


How are grids used l.jpg
How Are Grids Used? collaborative (e-)Science

Utility computing

High-performance computing

Collaborative design

Financial modeling

Collaborative data-sharing

High-energy physics

E-Business

Life sciences

Drug discovery

Data center automation

E-Science

Natural language processing

Business Intelligence

(Data Mining)


Agenda12 l.jpg
Agenda collaborative (e-)Science

  • Introduction

    • Utility Networks and Grid Computing

    • Application Drivers and Various Types of Grid Services

  • Global Grids and Challenges

    • Security, resource management, pricing models, …

  • Service-Oriented Grid Architecture and Gridbus Solutions

    • Market-based Management, GMD, Grid Bank, Aneka

  • Grid Service Broker

    • Architecture, Design and Implementation

  • Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids

    • A Case Study in High Energy Physics

  • Summary and Conclusion


Grid challenges l.jpg

Computational Economy collaborative (e-)Science

Security

Data locality

Resource Allocation

& Scheduling

Uniform Access

System Management

Resource Discovery

Application Construction

Network Management

Grid Challenges


Some grid initiatives worldwide l.jpg

Australia collaborative (e-)Science

Nimrod-G

Gridbus

DISCWorld

GrangeNet.

APACGrid

ARC eResearch

Brazil

OurGrid, EasyGrid

LNCC-Grid + many others

China

ChinaGrid – Education

CNGrid - application

Europe

UK eScience

EU Grids..

and many more...

India

Garuda

Japan

NAREGI

Korea...

N*Grid

Singapore

NGP

USA

Globus

TeraGrid

Cyberinfrasture

AutoMate

and many more...

Industry Initiatives

IBM On Demand Computing

HP Adaptive Computing

Sun N1

Microsoft - .NET

Oracle 10g

Amzon – Elastic Compute Cloud

Infosys – Enterprise Grid

Satyam – Business Grid

Manjrasoft – enterprise Clouds and Grids

and many more

Public Forums

Open Grid Forum

Conferences:

CCGrid

Grid

HPDC

E-Science

Some Grid Initiatives Worldwide

1.3 billion – 3 yrs

27 million

2? billion

120million – 5 yrs

450million – 5 yrs

486million – 5 yrs

1.3 billion (Rs)

1 billion – 5 yrs

http://www.gridcomputing.com


Open source grid middleware projects l.jpg
Open-Source Grid Middleware Projects collaborative (e-)Science

OurGrid

Slide by Hiro


Driving theme community vs utility grids l.jpg
Driving Theme: collaborative (e-)ScienceCommunity vs. Utility Grids


The gridbus project @ melbourne enable leasing of ict services on demand l.jpg
The Gridbus Project @ Melbourne: collaborative (e-)ScienceEnable Leasing of ICT Services on Demand

WWG

Gridbus

Pushes Grid computing into mainstream computing


Agenda19 l.jpg
Agenda collaborative (e-)Science

  • Introduction

    • Utility Networks and Grid Computing

    • Application Drivers and Various Types of Grid Services

  • Global Grids and Challenges

    • Security, resource management, pricing models, …

  • Service-Oriented Grid Architecture and Gridbus Solutions

    • Market-based Management, GMD, Grid Bank, Aneka

  • Grid Service Broker

    • Architecture, Design and Implementation

  • Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids

    • A Case Study in High Energy Physics

  • Summary and Conclusion


What do grid players want require l.jpg
What do Grid players want & require? collaborative (e-)Science

  • Grid Service Consumers (GSCs): - minimize expenses, meet QoS

    • How do I express QoS requirements ?

    • How do I trade between timeframe & cost ?

    • How do I discover services and map jobs to meet my QoS needs?

    • How do I manage Grid dynamics and get my work done?

  • Grid Service Providers (GSPs):– maximise ROI

    • How do I decide service pricing models ?

    • How do I specify them ?

    • How do I translate them into resource allocations ?

    • How do I enforce them ?

    • How do I advertise & attract consumers ?

    • How do I do accounting and handle payments?

  • They need mechanisms, tools and technologies that help them in value expression, value translation, and value enforcement.


Service oriented grid architecture l.jpg
Service-Oriented Grid Architecture collaborative (e-)Science

Data Catalogue

Grid Bank

Information Service

Grid Market Services

Sign-on

HealthMonitor

Info ?

Grid Node N

Grid Explorer

Secure

ProgrammingEnvironments

Job Control

Agent

Grid Node1

Applications

Schedule Advisor

QoS

Pricing Algorithms

Trade Server

Trading

Trade Manager

Accounting

Resource

Reservation

Misc. services

Deployment Agent

JobExec

Resource Allocation

Storage

Grid Resource Broker

R1

R2

Rm

Core Middleware

Services

Grid Service Consumer

Grid Service Providers


Market oriented grid software a union of gridbus and other technologies l.jpg

CDB collaborative (e-)Science

PDB

Market-Oriented Grid Software: A union of Gridbus and other technologies

Grid

Applications

Science

Commerce

Engineering

Collaboratories

Grid Portals

APIs/Tools:

ExcellGrid

Workflow APIs

Task, Parametric, and Components Programming

MPI

User-LevelMiddleware

Grid Workflow Engine

Grid Scheduling:

Gridbus Resource Broker

Grid MarketDirectory

Grid Exchange & Federation

Globus

Unicore

Grid Storage Economy

GridBank

Core Grid

Middleware

Aneka Cloud (WS-based access + SLA

NorduGrid

XGrid

Grid Economy

JVM

Condor

PBS

SGE

Libra

Tomcat

.NET

Grid

Fabric

Software

Mac

Windows

Linux

AIX

IRIX

OSF1

Solaris

Grid

Fabric

Hardware

Worldwide Grid


On demand assembly of services in market oriented grid environments l.jpg

Application Code collaborative (e-)Science

Explore data

1

Visual Application Composer

10

Results+Cost Info

2

GridResource Broker

Data Catalogue

5

4

Grid Info Service

12

6

3

ASP Catalogue

Grid Market Directory

9

7

Job

Results

8

Grid Service (GS)

(Globus)

Bill

Aneka

EC2

CPU

orPE

PE

GTS

11

GridbusGridBank

Resource Allocation

PE

GSP

(Accounting Service)

GSP

(e.g., IBM)

GSP

(e.g., Amazon)

GSP

(e.g., Microsoft)

On Demand Assembly of Services in Market-Oriented Grid Environments



Cloud services l.jpg
Cloud Services Environments

  • Infrastructure as a Services (IaaS)

    • CPU, Storage: Amazon.com et. al

  • Platform as a Services (PaaS)

    • Google App Engine, Microsoft Azure, andManjrasoft Aneka

  • Software as a Service (SaaS)

    • SalesForce.Com

Enterprise/Private Clouds

Clouds

Public/Internet Clouds


Layered view of services within a cloud stack l.jpg

Software as a Service (SaaS) Environments

e.g., ..SalesForce.com

Platform as a Service (PaaS)

e.g., ..Aneka

Infrastructure as a Service (IaaS)

e.g., Amazon, Nirvanix

Layered view of services within a Cloud stack


Aneka l.jpg

Aneka Environments

A Software Platform for Building and Managing “Enterprise” Grids and Clouds


Aneka a 3 rd gen enterprise grid technology cloud model l.jpg
Aneka: A 3 Environmentsrd Gen enterprise Grid Technology  Cloud model


Aneka product overview alpha l.jpg

.NET based service-oriented platform for grid / cloud computing

Development and Run Time Environment

Includes Development and Management Tools

Suitable for

Development of Enterprise Grid / Cloud Applications

Grid / Cloud enabling legacy applications

Ideal for Corporate Developers, Software, SaaS, Hosting Vendors and Application / System Integrators

ANEKA – Product Overview (Alpha)

ANEKA Product Architecture


Aneka deployment models l.jpg

Enterprise/Private computing

Harness LAN connected resources

Application Development, Testing, Execution

Teaching and Learning

Sensitive applications

Public

Hosted by a 3rd party service provider owning a large Data Center (1000s of servers)

Offers subscription-based services to their shared infrastructure on “pay-as go” model.to many users from different organisations.

Amazon.com, Microsoft Azure

Aneka SDK + Execution Manger

Aneka Deployment Models

Enterprise/Private Clouds

Aneka

Public Clouds


Aneka components l.jpg

Executor computing

Executor

Executor

Executor

Scheduler

ClientAgent

ClientAgent

Programming / Deployment Model

FIRST PRODUCT

Aneka: components

public DumbTask: ITask

{

public void Execute()

{

……

}

}

Aneka enterprise Cloud

for(int i=0; i<n; i++)

{

DumbTask task = newDumbTask();

app.SubmitExecution(task);

}

work units

internet

work units

Aneka Worker

Service

Aneka Manager

internet

Aneka Users


How does it solve the problem l.jpg
How does it solve the problem? computing

An Illustratioin

  • Divide the problem in to multiple small tasks and distribute them run in parallel on multiple computers within a Cloud.

Executor

Application

Manager

Manager / Executor

GThreads/Tasks


User scenario gofront unit of china southern railway group l.jpg
User scenario: GoFront computing(unit of China Southern Railway Group)

Aneka Maya Renderer

Use private Aneka Cloud

Case 2: Aneka Enterprise Cloud

Time (in hrs)

Case 1: Single Server

Using Maya Graphical Mode Directly

Single Server

Aneka Cloud

Aneka utilizes idle desktops (30) to decrease task time from days to hours

4 cores server

Application: Locomotive design CAD rendering

Raw Locomotive Design Files

(Using AutoDesk Maya)

GoFront Private Aneka Cloud

LAN network

(Running Maya Batch Mode on demand)


Aneka how can get it l.jpg
Aneka: How can get it? computing

  • Available to Download:

    • Software: www.manjrasoft.com

    • Manual: Setting up Cloud using your LAN-network computers

  • Teaching material

    • parallel and distributed computing and programming,

    • List of possible assignments for students

    • Possible Projects for Final year students..

  • Price – highly affordable 

    • = Fee you charge to 1 student (each year) and all students/teachers in entire college/university can use it!

  • Applications

    • Other Departments (Physics, Chemistry, Biology, Finance, Engineering) can use it for their applications.



Agenda36 l.jpg
Agenda Environments

  • Introduction

    • Utility Networks and Grid Computing

    • Application Drivers and Various Types of Grid Services

  • Global Grids and Challenges

    • Security, resource management, pricing models, …

  • Service-Oriented Grid Architecture and Gridbus Solutions

    • Market-based Management, GMD, Grid Bank, Aneka

  • Grid Service Broker

    • Architecture, Design and Implementation

  • Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids

    • A Case Study in High Energy Physics

  • Summary and Conclusion


Grid service broker gsb l.jpg
Grid Service Broker (GSB) Environments

  • A resource broker for scheduling task farming data Grid applications with static or dynamic parameter sweeps on global Grids.

  • It uses computational economy paradigm for optimal selection of computational and data services depending on their quality, cost, and availability, and users’ QoS requirements (deadline, budget, & T/C optimisation)

  • Key Features

    • A single window to manage & control experiment

    • Programmable Task Farming Engine

    • Resource Discovery and Resource Trading

    • Optimal Data Source Discovery

    • Scheduling & Predications

    • Generic Dispatcher & Grid Agents

    • Transportation of data & sharing of results

    • Accounting


Slide38 l.jpg

workload Environments

Gridbus User Console/Portal/Application Interface

App, T, $, Optimization Preference

Gridbus Broker

Gridbus Farming Engine

Schedule Advisor

Trading Manager

RecordKeeper

Grid Dispatcher

Grid Explorer

TM TS

$

GE GIS, NWS

Core Middleware

Grid Info Server

RM & TS

G

$

Data

Catalog

Data

Node

C

$

U

G

Globus enabled node.

L

A

Amazon EC2/S3 Cloud.


Gridbus broker separating applications from different remote service access enablers and schedulers l.jpg

Home Node/Portal Environments

Gridbus

Broker

batch()

-PBS

-Condor

-SGE

-Aneka

-XGrid

fork()

Data Catalog

Globus

Aneka

Amazon EC2

SSH

Job manager

fork()

AMI

batch()

fork()

batch()

-PBS

-Condor

-SGE

-XGrid

-PBS

-Condor

-SGE

Gridbus

agent

Gridbus

agent

Gridbus Broker: Separating “applications” from “different” remote service access enablers and schedulers

Application Development Interface

Single-sign on security

Alogorithm1

SchedulingInterfaces

AlogorithmN

Plugin Actuators

Data Store

Access Technology

SRB

Grid FTP


Gridbus services for escience applications l.jpg
Gridbus Services for eScience applications Environments

  • Application Development Environment:

    • XML-based language for composition of task farming (legacy) applications as parameter sweep applications.

    • Task Farming APIs for new applications.

    • Web APIs (e.g., Portlets) for Grid portal development.

    • Threads-based Programming Interface

    • Workflow interface and Gridbus-enabled workflow engine.

    • … Grid Superscalar – in cooperation with BSC/UPC

  • Resource Allocation and Scheduling

    • Dynamic discovery of optional computational and data nodes that meet user QoS requirements.

  • Hide Low-Level Grid Middleware interfaces

    • Globus (v2, v4), SRB, Aneka, Unicore, and ssh-based access to local/remote resources managed by XGrid, PBS, Condor, SGE.


Slide41 l.jpg

Click Here for Demo Environments

Drug Design

Made Easy!


Slide42 l.jpg
s Environments


Agenda43 l.jpg
Agenda Environments

  • Introduction

    • Utility Networks and Grid Computing

    • Application Drivers and Various Types of Grid Services

  • Global Grids and Challenges

    • Security, resource management, pricing models, …

  • Service-Oriented Grid Architecture and Gridbus Solutions

    • Market-based Management, GMD, Grid Bank, Aneka

  • Grid Service Broker

    • Architecture, Design and Implementation

  • Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids

    • A Case Study in High Energy Physics

  • Summary and Conclusion


Case study high energy physics and data grid l.jpg
Case Study: EnvironmentsHigh Energy Physics and Data Grid

  • The Belle Experiment

    • KEK B-Factory, Japan

    • Investigating fundamental violation of symmetry in nature (Charge Parity) which may help explain “why do we have more antimatter in the universe OR imbalance of matter and antimatter in the universe?”.

    • Collaboration 1000 people, 50 institutes

    • 100’s TB data currently


Case study event simulation and analysis l.jpg
Case Study: Event Simulation and Analysis Environments

B0->D*+D*-Ks

  • Simulation and Analysis Package - Belle Analysis Software Framework (BASF)

  • Experiment in 2 parts – Generation of Simulated Data and Analysis of the distributed data

Analyzed 100 data files (30MB each) that were distributed among the five nodes within Australian Belle DataGrid platform.


Australian belle data grid testbed l.jpg
Australian Belle Data Grid Testbed Environments

VPACMelbourne


Belle data grid gsp cpu service price g sec l.jpg
Belle Data Grid (GSP CPU Service Price: G$/sec) Environments

G$4

NA

G$4

G$6

VPACMelbourne

G$2

Datanode


Belle data grid bandwidth price g mb l.jpg
Belle Data Grid (Bandwidth Price: G$/MB) Environments

32

33

36

G$4

31

30

34

NA

38

31

G$4

G$6

VPACMelbourne

G$2

Datanode


Deploying application scenario l.jpg
Deploying Application Scenario Environments

  • A data grid scenario with 100 jobs and each accessing remote data of ~30MB

  • Deadline: 3hrs.

  • Budget: G$ 60K

  • Scheduling Optimisation Scenario:

    • Minimise Time

    • Minimise Cost

  • Results:


Time minimization in data grids l.jpg

fleagle.ph.unimelb.edu.au Environments

belle.anu.edu.au

belle.physics.usyd.edu.au

brecca-2.vpac.org

80

70

60

50

Number of jobs completed

40

30

20

10

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

Time (in mins.)

Time Minimization in Data Grids


Results cost minimization in data grids l.jpg

fleagle.ph.unimelb.edu.au Environments

belle.anu.edu.au

belle.physics.usyd.edu.au

brecca-2.vpac.org

100

90

80

70

60

50

Number of jobs completed

40

30

20

10

0

1

3

5

7

9

11

13

15

17

19

21

23

25

27

29

31

33

35

37

39

41

43

45

47

49

51

53

55

57

59

61

63

Time(in mins.)

Results : Cost Minimization in Data Grids


Observation l.jpg
Observation Environments


Agenda53 l.jpg
Agenda Environments

  • Introduction

    • Utility Networks and Grid Computing

    • Application Drivers and Various Types of Grid Services

  • Global Grids and Challenges

    • Security, resource management, pricing models, …

  • Service-Oriented Grid Architecture and Gridbus Solutions

    • Market-based Management, GMD, Grid Bank, Aneka

  • Grid Service Broker

    • Architecture, Design and Implementation

  • Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids

    • A Case Study in High Energy Physics

  • Summary and Conclusion


Summary and conclusion l.jpg
Summary and Conclusion Environments

  • Grids exploit synergies that result from cooperation of autonomous entities:

    • Resource sharing, dynamic provisioning, and aggregation at global level Great Science and Great Business!

  • Grids have emerged as enabler for Cyberinfrastructure that powers e-Science and e-Business applications.

  • SOA + Market-based Grid Management = Utility Grids

  • Grids allow users to dynamically lease Grid services at runtime based on their quality, cost, availability, and users QoS requirements.

    • Delivering ICT services as computing utilities.

  • Clouds are rapidly emerging, but more work is required

    • Federation of Clouds, Cloud Exchange, and Application Scaling


Convergence of competing paradigms communities needed l.jpg
Convergence of Competing Paradigms/Communities Needed Environments

?

}

  • Web

  • Data Centres

  • Utility Computing

  • Service Computing

  • Grid Computing

  • P2P Computing

  • Cloud Computing

  • Market-Oriented Computing

+

  • Ubiquitous access

  • Reliability

  • Scalability

  • Autonomic

  • Dynamic discovery

  • Composability

  • QoS

  • SLA

  • Trillion $ business

  • Who will own it?

Paradigms

Attributes/Capabilities


Thanks for your attention l.jpg
Thanks for your attention! Environments

  • Are there any

    • Questions?

    • Comments/ Suggestions

We Welcome Cooperation in R&D and Business! http:/www.gridbus.org | www.Manjrasoft.com

[email protected] | [email protected]


ad