Pragma grid research projects in universiti sains malaysia
Download
1 / 41

PRAGMA - PowerPoint PPT Presentation


  • 295 Views
  • Updated On :

PRAGMA Grid Research Projects in Universiti Sains Malaysia. Compute Power Market / P2P e-Science Grid. Presented by : Chan Huah Yong Gian Chand Sodhy Grid Computing Research Lab Universiti Sains Malaysia. PRAGMA Grid Research Projects in Universiti Sains Malaysia.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'PRAGMA ' - Solomon


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Pragma grid research projects in universiti sains malaysia l.jpg

PRAGMAGrid Research Projects in Universiti Sains Malaysia

Compute Power Market / P2P

e-Science Grid

Presented by : Chan Huah Yong

Gian Chand Sodhy

Grid Computing Research Lab

Universiti Sains Malaysia


Pragma grid research projects in universiti sains malaysia2 l.jpg

PRAGMAGrid Research Projects in Universiti Sains Malaysia

Compute Power Market / P2P


Compute power market peer to peer l.jpg
Compute Power Market/Peer to Peer

  • Compute Power Market (CPM)

    • computational market in peer-to-peer computing style,

    • introduces resource trading on idle computational power across the Internet.


Cpm p2p framework l.jpg
CPM/P2P Framework

Layer 4:

Applications

Applications

Portals

Scheduler

Layer 3:

CPM

Utilities

Market

Server

CPM

Agent

Trader

Dispatcher

Accounting

Job management

Layer 2:

P2P

Middleware

JXTA Protocols (Peer Groups, Peer Pipes, Monitoring etc)

JXTA Security

Layer 1:

P2P Fabric

Internet

USM

Melbourne


Cpm design architecture l.jpg
CPM Design Architecture

Three basic entities:

  • resource provider

  • resource consumer

  • market.

    These entities are communicating with each other through a set of utilities and services that will cater for resource searching, pricing negotiation, resource allocation, job deployment, transaction accounting and so on.


Cpm design architecture6 l.jpg
CPM Design Architecture

Two key players driving the Compute Power Market:

  • Compute Power Market Agent

  • Market Server

    Like the conventional marketplace, CPM Agent works on behalf of both resource owner and the resource consumer, while the market server provides the necessary infrastructure including communication, information, security, transport access to remote resources, market repository and information services that able to bring these 2 parties together for resources trading.


Entities interaction l.jpg

Market Repository

EntitiesInteraction

Market

Market Server

1) Publish Resource Information

2) Discover Resource Information

CPM Agent

CPM Agent

3) Contract Negotiation

4) Job execution

5) Accounting

Consumer

Provider


Cpm market server l.jpg
CPM Market Server

Consists of two major components:

  • Membership Control

    • maintains the access authentication of the market by applying membership among the provider and consumer.

  • Resource Discovery component

    • manipulates the resource publishing and discovery process.

    • The resource publishing service allows the resource provider to advertise their resource and the resource discovery service provides consumer the facility to discover resource available in the market.


Cpm agent l.jpg
CPM Agent

Application that comprises the fundamental services of CPM, includes:

  • Scheduler– schedule and dispatch job

  • Trader- handles the trading and negotiation between the provider and consumer.

  • Job management - handles the job deployment and execution on the provider computer device and returns the result back to the consumer.

  • Accounting- handles the fees and pays for the trading using a resource meter   to keep track of the resources that have been consumed.


Cpm agent cont l.jpg
CPM Agent (cont…)

Users interact with their CPM Agent for

  • searching (search ideal resource provider base on consumer preferences),

  • managing(negotiating with CPM Agent from the provider for a particular resources trading) and

  • running (executing and scheduling job) resources in the CPM.

  • CPM Agent will represent resource owners and do the arrangement and management of the resources and play an important role in trading negotiation. When the job finished, CPM Agent completes the accounting process and bill the consumer for the charge of resource usage.


Work completed l.jpg
Work Completed

  • CPM/P2P Version 1.0

    • Trader

    • Job Management

    • Accounting

    • Generic Components

  • Basic components

  • Already uploaded to JXTA CVS on September 2002.


Slide12 l.jpg

Benchmark + Retrieve + Publish Info

waiting for contract

Discover + Join Market

Receive + Accept/Reject Contract

Update log database

Receive + Execute job

Calculate Charge + prepare bill

Pack + Return Result + Bill

Complete System Flow for CPM Agent on Provider

Trader

provider

Generic Components [using JXTA]

Accounting

Job management


Slide13 l.jpg

Set Trading condition

Discover resource

Discover + Join Market

Generate + send Contract

Update consumer account

Send job

receive Result + Bill

Check Bill + Pay provider

Complete System Flow for CPM Agent on Consumer

Trader

consumer

Generic Components [using JXTA]

Reject

Accept

Accounting

Job management


Work in progress l.jpg
Work In Progress

  • Scheduler

  • Market Repository

    • Provide advertisement manipulation on Market repository

    • Multi Rendezvous Model


Scheduler l.jpg

CPM Agent

CPM/P2P Scheduler (super peer)

Scheduler

Accounting

Submit job

Receive result

User

Trader

Job Management

Master

Worker


Scheduler16 l.jpg
Scheduler

  • The scheduler has a Hierarchical Master-Worker architecture.

    • Master (metascheduler) schedules job based on global scheduling policies.

    • Worker (local scheduler) schedules and dispatches job based on local system policies.

  • Job scheduling is based on dateline and budget constraint.


Market repository l.jpg

CPM Agent

Market Repository

Market Server

  • Discovery

  • - Membership

Market Repository

User

  • Market repository.

    • Searching algorithm (advertisement)

    • Index manipulation (add new, update, delete)

  • Market-to-market Interaction

Resources


Updating market repository l.jpg
Updating Market Repository

  • Once peer leave the network

    • Update Repository indicate peer has left (delete the service offered by that peer)

  • If there are new services to be shared by existing peer

    • Update the old information (services) in Repository


Future work l.jpg
Future Work

  • Trading Policy

    • The resource usage policy (such as resource reservation, resource availability)

  • Security

    • Handle the security issues in CPM/P2P, include secure communications, intrusion detection.

  • Multi Market Model

    • Adopt various market model (auction, bargain) into CPM/P2P trader. Currently, only Commodity (Flat Price) market model is being adopted.



Pragma grid research projects in universiti sains malaysia21 l.jpg

PRAGMAGrid Research Projects in Universiti Sains Malaysia

e-Science Grid


What is an e science grid portal l.jpg

“A meeting point for

scientific communities to run

scientific applications through

distributed global collaborations

enabled by the Internet and

grid engine,

using distributed resources and

high performance platform.”

What is an e-Science grid Portal?


E science grid the complete framework l.jpg

Simulations

Of

Materials

Scientific

Informatics

Mathematical

& Theoretical

Simulations

• • •

Phase II

Compute-

Intensive

Data-Intensive

Visualization &

Collaboration

Collaborative

Resource

Management

Resource

Monitoring

Phase I

Uniform

Resource

Access

Uniform Data

Access

Authentication &

authorization

Security

services

Grid Information

Service

Brokering

Co-scheduling

• • •

Data

Storage

Special

Instrument

Fast-Ethernet

based

PC Cluster

e-Science Grid The Complete Framework

Layer 4:

Portal & Application

Iterative Solver

Parallel Molecular

Modeling

Layer 3:

Application Toolkit

Data

Management

Security

Layer 2:

Core Grid Engine

Globus

Layer 1:

Infrastructure

Short-Area-

Network based

PC Cluster

Existing Server/

Super Computer


Conceptual model of e science grid l.jpg
Conceptual Model of e-Science grid

  • Service-oriented grid

    • Provider – organization provides services to the grid

  • Economic-based grid

    • Based on the compute power market

      • Contract – agreement on services usage between provider and consumer

      • Marketplace – intermediate/ communication medium for both parties (organizations)


  • Difference between p2p and grid l.jpg

    I

    O

    I

    I

    I

    O

    O

    Market

    Market

    O

    Level 1

    G

    G

    Level 2

    I

    I

    I

    Difference between P2P and grid

    E-Science GRID

    CPM/p2p

    O

    = Organization, etc USM, UKM, UTM

    G

    = Group (group by Individuals in an organization)

    = Individual user, etc: grid007.cs.usm.my


    Visit us grid cs usm my portal index jsp l.jpg
    Visit us: grid.cs.usm.my/portal/index.jsp


    Portal framework l.jpg

    Myproxy Server

    Compute Resources

    with Globus deployed

    Information Services

    LDAP servers

    Client browser

    Web Server

    Portal Framework

    GSI

    HTTPS

    HTTPS,

    Globus Services

    LDAP

    Tomcat, Java,

    COG kit,

    GPDK


    E science grid architecture l.jpg

    Applications

    Replica

    Catalog

    e-Science Portal

    Replica Management

    Service

    File

    Transfer

    Job Manager

    Intrusion Detection Service

    Scheduler Specific plug-in

    Bill

    Directory Service

    E-Science Grid Architecture

    User

    Mobile Agent

    Resources

    Account Manager

    Resource Usage Tracking Agent


    Slide29 l.jpg

    RESOURCE MONITORING

    Resource Monitor

    GRID RESOURCE ALLOCATION Overall Framework

    RESOURCE ALLOCATION

    Resource Broker

    Available resources?

    Grid

    Scheduler

    Cost? Bill?

    Resource Usage?

    Trader

    Metering

    Replica Management

    Data? (by agent)

    Select resources Sends agents

    RESOURCES ON GRID


    Framework newly proposed l.jpg
    Framework (newly proposed)

    grid.cs.usm.my

    - Resource Broker

    - GRAM

    - DUROC

    Job Submission request via http

    GRAM

    Gatekeeper

    Job manager Condor

    GRAM

    Gatekeeper

    Job manager Condor

    Condor pool

    PBS pool

    Standalone PC’s

    GRAM

    Gatekeeper

    Job manager fork


    Resource broker framework l.jpg
    Resource Broker Framework

    Resource broker

    User request to RSL mapper

    Job Description in RSL

    Resource request

    Resource Monitoring

    Matchmaker

    Available resources

    Job + resource description RSL

    Resource Reservation

    Dispatch job

    Dispatch job

    Dispatch job

    Condor pool

    PBS pool

    Individual resources


    Metering design l.jpg

    Consumer

    Provider

    Administrator

    Metering Design

    Cost?

    Budget/Deadline

    Charging

    Trading

    Accept?

    No

    Yes

    Credit

    >=

    Budget?

    Yes

    No

    Job Scheduling

    Message top-up

    Done?

    No

    Usage Tracking

    Message failure

    Yes

    Accounting

    Billing


    Slide33 l.jpg

    REPLICA MANAGEMENT Overall Framework

    Information

    Services

    (4) List of available resources

    (5) Conf. of each resources

    Replica

    Catalog

    (2) List of proteins

    Replica

    Selection

    (1) List of proteins to dock

    (3) List of available resources

    to a ligand

    Not replicated protein files

    Selected replicas

    Compute Resources

    Selected replicas

    Set of replicated PDB

    Grid Environments


    Resource monitoring l.jpg
    Resource Monitoring

    User

    Access

    Visualization

    Metering

    (Project 2)

    e-Sciences Grid Portal

    (Dynamic Information Services)

    Grid Scheduler

    (Project 2)

    Other Applications

    (Project 5)

    Processed Data

    Authorized

    Prediction

    Platform Type, Operating System, CPU, Memory, Network, File System, Job Status

    Resource Monitor

    Event Publication Information

    Dispatch Agent

    Directory Service

    Mobile Agents Facility

    Invitation/Correction

    Event Publication Information

    Resources


    Focused area resource monitoring l.jpg
    Focused Area Resource Monitoring

    Status Monitoring

    Event Archive

    Resource Monitor

    Job Monitoring

    Mobile Agents Facility

    Reverse Registering

    Directory Service

    Resource Discovery

    Fault Correction

    Resources


    Prediction module l.jpg
    Prediction Module

    Network Weather Service

    Network

    Monitoring

    Name Server

    network

    measurement

    Sensor

    Persistent State

    Forecaster

    e-Science

    Grid

    forecasts

    forecasts

    E-Science Portal

    Client


    Milestones l.jpg
    Milestones

    • Phase I(June - August 2002)– Final Model and Design

    • Phase II(Sep 2002 - May 2003)– Major Implementation & Integration


    Slide38 l.jpg

    Manpower

    • 4 lecturers

    • 9 research officers

    • 5 MSc. students

    • 5 BSc. students


    Resources machines l.jpg
    Resources (Machines)

    • 16-node clusters x 2

    • Pentium-4 dual-processor servers x 2

    • Pentium-4 high-end workstations x 8

    • Pentium-4 PCs x 29

    • Pentium-3 PCs x 3


    Visit us at http grid cs usm my l.jpg
    Visit us at: http://grid.cs.usm.my



    ad