pragma grid research projects in universiti sains malaysia
Download
Skip this Video
Download Presentation
PRAGMA Grid Research Projects in Universiti Sains Malaysia

Loading in 2 Seconds...

play fullscreen
1 / 41

PRAGMA - PowerPoint PPT Presentation


  • 298 Views
  • Uploaded on

PRAGMA Grid Research Projects in Universiti Sains Malaysia. Compute Power Market / P2P e-Science Grid. Presented by : Chan Huah Yong Gian Chand Sodhy Grid Computing Research Lab Universiti Sains Malaysia. PRAGMA Grid Research Projects in Universiti Sains Malaysia.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'PRAGMA ' - Solomon


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
pragma grid research projects in universiti sains malaysia

PRAGMAGrid Research Projects in Universiti Sains Malaysia

Compute Power Market / P2P

e-Science Grid

Presented by : Chan Huah Yong

Gian Chand Sodhy

Grid Computing Research Lab

Universiti Sains Malaysia

compute power market peer to peer
Compute Power Market/Peer to Peer
  • Compute Power Market (CPM)
    • computational market in peer-to-peer computing style,
    • introduces resource trading on idle computational power across the Internet.
cpm p2p framework
CPM/P2P Framework

Layer 4:

Applications

Applications

Portals

Scheduler

Layer 3:

CPM

Utilities

Market

Server

CPM

Agent

Trader

Dispatcher

Accounting

Job management

Layer 2:

P2P

Middleware

JXTA Protocols (Peer Groups, Peer Pipes, Monitoring etc)

JXTA Security

Layer 1:

P2P Fabric

Internet

USM

Melbourne

cpm design architecture
CPM Design Architecture

Three basic entities:

  • resource provider
  • resource consumer
  • market.

These entities are communicating with each other through a set of utilities and services that will cater for resource searching, pricing negotiation, resource allocation, job deployment, transaction accounting and so on.

cpm design architecture6
CPM Design Architecture

Two key players driving the Compute Power Market:

  • Compute Power Market Agent
  • Market Server

Like the conventional marketplace, CPM Agent works on behalf of both resource owner and the resource consumer, while the market server provides the necessary infrastructure including communication, information, security, transport access to remote resources, market repository and information services that able to bring these 2 parties together for resources trading.

entities interaction

Market Repository

EntitiesInteraction

Market

Market Server

1) Publish Resource Information

2) Discover Resource Information

CPM Agent

CPM Agent

3) Contract Negotiation

4) Job execution

5) Accounting

Consumer

Provider

cpm market server
CPM Market Server

Consists of two major components:

  • Membership Control
    • maintains the access authentication of the market by applying membership among the provider and consumer.
  • Resource Discovery component
    • manipulates the resource publishing and discovery process.
    • The resource publishing service allows the resource provider to advertise their resource and the resource discovery service provides consumer the facility to discover resource available in the market.
cpm agent
CPM Agent

Application that comprises the fundamental services of CPM, includes:

  • Scheduler– schedule and dispatch job
  • Trader- handles the trading and negotiation between the provider and consumer.
  • Job management - handles the job deployment and execution on the provider computer device and returns the result back to the consumer.
  • Accounting- handles the fees and pays for the trading using a resource meter   to keep track of the resources that have been consumed.
cpm agent cont
CPM Agent (cont…)

Users interact with their CPM Agent for

  • searching (search ideal resource provider base on consumer preferences),
  • managing(negotiating with CPM Agent from the provider for a particular resources trading) and
  • running (executing and scheduling job) resources in the CPM.
  • CPM Agent will represent resource owners and do the arrangement and management of the resources and play an important role in trading negotiation. When the job finished, CPM Agent completes the accounting process and bill the consumer for the charge of resource usage.
work completed
Work Completed
  • CPM/P2P Version 1.0
    • Trader
    • Job Management
    • Accounting
    • Generic Components
  • Basic components
  • Already uploaded to JXTA CVS on September 2002.
slide12

Benchmark + Retrieve + Publish Info

waiting for contract

Discover + Join Market

Receive + Accept/Reject Contract

Update log database

Receive + Execute job

Calculate Charge + prepare bill

Pack + Return Result + Bill

Complete System Flow for CPM Agent on Provider

Trader

provider

Generic Components [using JXTA]

Accounting

Job management

slide13

Set Trading condition

Discover resource

Discover + Join Market

Generate + send Contract

Update consumer account

Send job

receive Result + Bill

Check Bill + Pay provider

Complete System Flow for CPM Agent on Consumer

Trader

consumer

Generic Components [using JXTA]

Reject

Accept

Accounting

Job management

work in progress
Work In Progress
  • Scheduler
  • Market Repository
    • Provide advertisement manipulation on Market repository
    • Multi Rendezvous Model
scheduler

CPM Agent

CPM/P2P Scheduler (super peer)

Scheduler

Accounting

Submit job

Receive result

User

Trader

Job Management

Master

Worker

scheduler16
Scheduler
  • The scheduler has a Hierarchical Master-Worker architecture.
    • Master (metascheduler) schedules job based on global scheduling policies.
    • Worker (local scheduler) schedules and dispatches job based on local system policies.
  • Job scheduling is based on dateline and budget constraint.
market repository

CPM Agent

Market Repository

Market Server

  • Discovery
  • - Membership

Market Repository

User

  • Market repository.
    • Searching algorithm (advertisement)
    • Index manipulation (add new, update, delete)
  • Market-to-market Interaction

Resources

updating market repository
Updating Market Repository
  • Once peer leave the network
    • Update Repository indicate peer has left (delete the service offered by that peer)
  • If there are new services to be shared by existing peer
    • Update the old information (services) in Repository
future work
Future Work
  • Trading Policy
    • The resource usage policy (such as resource reservation, resource availability)
  • Security
    • Handle the security issues in CPM/P2P, include secure communications, intrusion detection.
  • Multi Market Model
    • Adopt various market model (auction, bargain) into CPM/P2P trader. Currently, only Commodity (Flat Price) market model is being adopted.
what is an e science grid portal
“A meeting point for

scientific communities to run

scientific applications through

distributed global collaborations

enabled by the Internet and

grid engine,

using distributed resources and

high performance platform.”

What is an e-Science grid Portal?
e science grid the complete framework

Simulations

Of

Materials

Scientific

Informatics

Mathematical

& Theoretical

Simulations

• • •

Phase II

Compute-

Intensive

Data-Intensive

Visualization &

Collaboration

Collaborative

Resource

Management

Resource

Monitoring

Phase I

Uniform

Resource

Access

Uniform Data

Access

Authentication &

authorization

Security

services

Grid Information

Service

Brokering

Co-scheduling

• • •

Data

Storage

Special

Instrument

Fast-Ethernet

based

PC Cluster

e-Science Grid The Complete Framework

Layer 4:

Portal & Application

Iterative Solver

Parallel Molecular

Modeling

Layer 3:

Application Toolkit

Data

Management

Security

Layer 2:

Core Grid Engine

Globus

Layer 1:

Infrastructure

Short-Area-

Network based

PC Cluster

Existing Server/

Super Computer

conceptual model of e science grid
Conceptual Model of e-Science grid
  • Service-oriented grid
      • Provider – organization provides services to the grid
  • Economic-based grid
    • Based on the compute power market
      • Contract – agreement on services usage between provider and consumer
      • Marketplace – intermediate/ communication medium for both parties (organizations)
difference between p2p and grid

I

O

I

I

I

O

O

Market

Market

O

Level 1

G

G

Level 2

I

I

I

Difference between P2P and grid

E-Science GRID

CPM/p2p

O

= Organization, etc USM, UKM, UTM

G

= Group (group by Individuals in an organization)

= Individual user, etc: grid007.cs.usm.my

portal framework

Myproxy Server

Compute Resources

with Globus deployed

Information Services

LDAP servers

Client browser

Web Server

Portal Framework

GSI

HTTPS

HTTPS,

Globus Services

LDAP

Tomcat, Java,

COG kit,

GPDK

e science grid architecture

Applications

Replica

Catalog

e-Science Portal

Replica Management

Service

File

Transfer

Job Manager

Intrusion Detection Service

Scheduler Specific plug-in

Bill

Directory Service

E-Science Grid Architecture

User

Mobile Agent

Resources

Account Manager

Resource Usage Tracking Agent

slide29

RESOURCE MONITORING

Resource Monitor

GRID RESOURCE ALLOCATION Overall Framework

RESOURCE ALLOCATION

Resource Broker

Available resources?

Grid

Scheduler

Cost? Bill?

Resource Usage?

Trader

Metering

Replica Management

Data? (by agent)

Select resources Sends agents

RESOURCES ON GRID

framework newly proposed
Framework (newly proposed)

grid.cs.usm.my

- Resource Broker

- GRAM

- DUROC

Job Submission request via http

GRAM

Gatekeeper

Job manager Condor

GRAM

Gatekeeper

Job manager Condor

Condor pool

PBS pool

Standalone PC’s

GRAM

Gatekeeper

Job manager fork

resource broker framework
Resource Broker Framework

Resource broker

User request to RSL mapper

Job Description in RSL

Resource request

Resource Monitoring

Matchmaker

Available resources

Job + resource description RSL

Resource Reservation

Dispatch job

Dispatch job

Dispatch job

Condor pool

PBS pool

Individual resources

metering design

Consumer

Provider

Administrator

Metering Design

Cost?

Budget/Deadline

Charging

Trading

Accept?

No

Yes

Credit

>=

Budget?

Yes

No

Job Scheduling

Message top-up

Done?

No

Usage Tracking

Message failure

Yes

Accounting

Billing

slide33

REPLICA MANAGEMENT Overall Framework

Information

Services

(4) List of available resources

(5) Conf. of each resources

Replica

Catalog

(2) List of proteins

Replica

Selection

(1) List of proteins to dock

(3) List of available resources

to a ligand

Not replicated protein files

Selected replicas

Compute Resources

Selected replicas

Set of replicated PDB

Grid Environments

resource monitoring
Resource Monitoring

User

Access

Visualization

Metering

(Project 2)

e-Sciences Grid Portal

(Dynamic Information Services)

Grid Scheduler

(Project 2)

Other Applications

(Project 5)

Processed Data

Authorized

Prediction

Platform Type, Operating System, CPU, Memory, Network, File System, Job Status

Resource Monitor

Event Publication Information

Dispatch Agent

Directory Service

Mobile Agents Facility

Invitation/Correction

Event Publication Information

Resources

focused area resource monitoring
Focused Area Resource Monitoring

Status Monitoring

Event Archive

Resource Monitor

Job Monitoring

Mobile Agents Facility

Reverse Registering

Directory Service

Resource Discovery

Fault Correction

Resources

prediction module
Prediction Module

Network Weather Service

Network

Monitoring

Name Server

network

measurement

Sensor

Persistent State

Forecaster

e-Science

Grid

forecasts

forecasts

E-Science Portal

Client

milestones
Milestones
  • Phase I(June - August 2002)– Final Model and Design
  • Phase II(Sep 2002 - May 2003)– Major Implementation & Integration
slide38

Manpower

  • 4 lecturers
  • 9 research officers
  • 5 MSc. students
  • 5 BSc. students
resources machines
Resources (Machines)
  • 16-node clusters x 2
  • Pentium-4 dual-processor servers x 2
  • Pentium-4 high-end workstations x 8
  • Pentium-4 PCs x 29
  • Pentium-3 PCs x 3
ad