grid tools n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Grid Tools PowerPoint Presentation
Download Presentation
Grid Tools

Loading in 2 Seconds...

play fullscreen
1 / 40

Grid Tools - PowerPoint PPT Presentation


  • 133 Views
  • Uploaded on

Grid Tools. Jai Dayal & Casey Rathbone. Outline. Web Services Unicore BOINC Globus Condor. Web Services.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Grid Tools' - theola


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
grid tools

Grid Tools

Jai Dayal & Casey Rathbone

outline
Outline
  • Web Services
  • Unicore
  • BOINC
  • Globus
  • Condor
web services
Web Services

“A Web service is an abstract notion that must be implemented by a concrete agent. … The agent is the concrete piece of software or hardware that sends and receives messages, while the service is the resource characterized by the abstract set of functionality that is provided.”

web services2
Web Services
  • Remote Procedure Calls (RPC) - distributed function call interface
  • Service Oriented Architecture (SOA) - utilizes message passing to communicate to services also known as "message-oriented" services
  • Representational State Transfer (REST) - emulates HTTP (or similar protocols) to communicate stateful resources
unicore overview
Unicore - Overview
  • Single point of entry – security
    • Uses SSL
  • Uses web services
    • SOAP Messages
    • WSDL
  • Reputation for simple deployment
unicore architecture
Unicore - Architecture
  • Most basic resource
    • A cluster, a PC, web service, etc

Vsite

unicore architecture1
Unicore - Architecture
  • The gateway is the point of entry to the Unicore grid

Gateway

Vsites

unicore architecture2
Unicore - Architecture
  • Gateway + Vsites = Usite

Usite

Gateway

Vsites

unicore architecture3
Unicore - Architecture

UNICORE - Architecture

Usite

Gateway

Client

Vsites

unicore architecture4
Unicore - Architecture
  • Client communicates using SOAP
  • SOAP/HTTP binding

Usite

Gateway

Client

SOAP Message

Vsites

  • SOAP contains:
    • Address of gateway
    • Port number
    • Vsite name
    • task descriptions
unicore security
Unicore - Security
  • Gateway handles authentication
    • SSL used to authenticate the client
    • SSL also used by client to authenticate gateway
    • Both present digital certificates
  • Vsite handles authorization
    • Gateway passes along certificate in SOAP header
    • If client has privileges, Vsite authorizes
  • This model allows end to end security
  • Removes some strain from gateway
  • Trust between gateway and Vsite
boinc overview
BOINC - Overview

Grid

Public Resource Computing

Owned by institutions/orgs

Individuals own pc's

Managed/used by pro's

Used by commoners

High level of trust

Low (no?) level of trust

  • Grid computing vs. public resource computing
boinc overview1
BOINC - Overview

Grid

Public Resource Computing

Owned by institutions/orgs

Individuals own pc's

Managed/used by pro's

Used by commoners

High level of trust

Low (no?) level of trust

  • Grid computing vs. public resource computing
  • BOINC Goals
    • Cheap and easy
    • Project variety
    • Public participation → reward points
boinc overview2
BOINC - Overview

Grid

Public Resource Computing

Owned by institutions/orgs

Individuals own pc's

Managed/used by pro's

Used by commoners

High level of trust

Low (no?) level of trust

  • Grid computing vs. public resource computing
  • BOINC Goals
    • Cheap and easy
    • Project variety
    • Public participation → reward points
  • BOINC projects
    • SETI at home
    • Folding at home
    • Predictor at home
boinc architecture2
BOINC - Architecture

BOINC

Projects

Servers

Scheduling

Data

boinc architecture3
BOINC - Architecture

BOINC

Projects

Tools

Servers

Scheduling

Data

boinc architecture4
BOINC - Architecture

BOINC

Projects

Tools

Servers

Scheduling

Data

Client

boinc workflow
BOINC - Workflow
  • Client contacts Scheduling Server
  • Server returns XML Document
  • Workunit: inputs for a computation/job
    • Applications
    • Input files
    • Environment variables
  • Result:
    • Reference to workunit
    • Reference to output files
  • Client runs job and places results in output files
boinc fault tolerance
BOINC – Fault tolerance
  • Public Resource Computing is volatile
    • Users leave before job completing
    • Over-clocking can lead to bad results
    • Malicious users
  • Homogenous Redundancy
    • Computer architecture specific jobs
  • Transitioner
    • Identifies bad results
  • Validator
    • Determines canonical result
  • Assimilator
    • Handles canonical result
  • These daemons can run on different hosts
boinc client settings
BOINC – Client settings
  • Client implements “local scheduling policy”
    • Utilizes maximum resources
    • Meet job deadlines
    • Respect clients resource allocation wishes
    • Offers some variety to the client
  • Anonymous platform
    • Used for open source projects
    • Clients can modify apps to fit their resources
    • Communicate with scheduling server using XML config file.
globus toolkit 4
Globus Toolkit 4
  • GT4 is a open-source toolkit developed to build grids
  • It provides full capabilities for sharing computing power and databases
  • Usage of Globus is extensive throughout the scientific community within NSF, DOE, DARPA, IBM, NASA, and Microsoft projects
  • The Large Hadron Collider project utilizes a Globus based grid for the European Data Grid.
globus toolkit 41
Globus Toolkit 4

Globus Toolkit is a suite of Web-Services and Unix that controls provides an interface to submit, monitor, and cancel jobs on the Grid resources.

Globus Toolkit is an interface that allows for jobs to be submitted and executed at the appropriate resources within the grid

globus toolkit 4 security
Globus Toolkit 4 Security
  • GT4 uses (Grid Security Infrastructure) GSI with an public key cryptography
  • GSI Functional Layers - Authorization, Authentications, Delegation, Message Protection, and Message Format
globus toolkit 4 data management
Globus Toolkit 4 Data Management
  • Data Management tools within the toolkit fall into either of two categories data replication and data movement
  • Data Replication consists of Replica Location Service (RLS)‏
  • Data Movement consists of GridFTP and Reliable File Transfer (RFT)‏
gt4 data replication
GT4: Data Replication

Replica Location Service provides the capability to track and maintain multiple locations of data across the grid.  It is a distributed registry system that allows users and applications to register the locations of data.

gt4 data movement
GT4: Data Movement
  • GridFTP - GT4 provides a server implementation, scriptable command-line interface and several GUI client interfaces
  • Reliable File Transfer (RFT) - A Web Services Resource Framework that schedules file transfers based on a set of criteria of when specific resources and bandwidth is available
globus toolkit 4 execution management
Globus Toolkit 4 Execution Management
  • Community Scheduler Framework - Grid level meta-scheduling framework
  • Grid Teleoperation Control Protocol - A service interface that has been used to
  • Dynamic Accounts - This service provides functionality to create groups and accounts for specific jobs that need to be submitted
  • Grid Resource Allocation & Management - Used to locate, submit, monitor and cancel jobs on Grid computing resources.  It is the fabric that ties all the schedulers and job queues together to utilize the resources.
gt4 community scheduler framework
GT4: Community Scheduler Framework

Open-Source implementation of OSGA-based meta scheduler

Provides an interface to existing grid platforms making integration more efficient

gt4 community scheduler framework1
GT4: Community Scheduler Framework

The diagram below shows the basic layout of the scheduler framework

http://www.globus.org/grid_software/computation/csf.php

gt4 grid teleoperation control protocol
GT4: Grid Teleoperation Control Protocol

The GTCP is used to control heterogenous physical and computational simulations.

It exposes the WSRF-compliant service interface used to control remote instruments and simulations and a interface to the backend GTCP to allow for additional control abstraction.

globus toolkit 4 information services
Globus Toolkit 4 Information Services

Monitoring and Discovery Service is a suite of web services used to discover resources and services on a grid

This service is dedicated to handling the complex task of organizing resources into Virtual Organizations (VO)

http://www.globus.org/toolkit/docs/4.0/info/key-index.html

globus toolkit 4 common runtime
Globus Toolkit 4 Common Runtime
  • GT4 contains a multitude of implementations of libraries allowing the suite to be platform independent.
    • C Common Libraries
    • C WebService Libraries
    • Java WebService Libraries
    • Python WebService Libraries
    • XIO
condor overview
Condor - Overview
  • University of Wisconsin – Madison
  • Open source
  • Linux, Solaris, Mac OS, Windows
  • No program modification
  • Jobs tend to be architecture specific
    • Designed on x86 → job run on x86
    • Java applications possible exception
  • Overall concept is simple
    • Install it on each desired machine
    • Designate one as central manager
  • Practice is much harder
    • Installation is tricky
    • Lots of commands to learn
condor
Condor
  • Central Manager
    • Detects idle machines
    • Matches job with available resources
    • Considers each processor as separate resource
  • Jobs
    • Submitted via “submit file”
    • Universe, executable name, inputs, outputs, error, log file
  • Sequential Jobs
    • Standard universe
      • Used for long jobs (weeks, months)‏
      • Allows partial execution, pausing
      • Can relocate job to another machine
    • Vanilla universe
      • For short term jobs
    • Local universe
      • Submitter executes job
  • Parallel Jobs
    • Supports MPI and PVM
condor variety
Condor - variety
  • Flocking
    • Connects multiple pools
    • If current pool has no resources, job “flocks”
    • Set “flock_to” variable in config file
  • Condor-G
    • Distributed flocking
    • Utilizes Globus
    • Universe = Globus
    • Also specify “globusscheduler”
      • Points to the globus job manager
references
References

[1] D. P. Anderson, BOINC: A System for Public-Resource Computing and Storage.  In Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing. November, 2004.[2] S. Bagchi, E. Hung, A. Iyengar, N. Vogl, and N. Wadia, Capacity Planning Tools for Web and Grid Environments.  In Proceedings of the 1st international conference on Performance evaluation methodolgies and tools. October, 2006.[3] F. Hernández, P. Bangalore,  K. Reilly, End-User Tools for Grid Computing. In Proceedings of the first workshop on End-user software engineering. July, 2005.[4]I. Habib, Getting Started with Condor.  Linux Journal, Volume 2006, Issue 149, September 2006.

[5]Menday, R., "The Web Services Architecture and the UNICORE Gateway," Telecommunications, 2006. AICT-ICIW '06. International Conference on Internet and Web Applications and Services/Advanced International Conference on , vol., no., pp. 134-134, 19-25 Feb. 2006

[6]Hellinckx, P.; Stuer, G.; Hendrickx, W.; Arickx, F.; Broeckhove, J., "Grid-user driven grid research, the CoBRA grid," Cluster Computing and the Grid Workshops, 2006. Sixth IEEE International Symposium on , vol.2, no., pp. 8 pp.-, 16-19 May 2006