slide1 l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
The UCLA Grid Portal - Campus Grids and the UC Grid PowerPoint Presentation
Download Presentation
The UCLA Grid Portal - Campus Grids and the UC Grid

Loading in 2 Seconds...

play fullscreen
1 / 33

The UCLA Grid Portal - Campus Grids and the UC Grid - PowerPoint PPT Presentation


  • 319 Views
  • Uploaded on

The UCLA Grid Portal - Campus Grids and the UC Grid http://grid.ucla.edu Joan Slottow and Prakashan Korambath Research Computing Technologies UCLA Academic Technology Services UCLA Grid Architecture Head node Cluster I Credential mapped to local ID Grid Appliance Grid Portal https

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'The UCLA Grid Portal - Campus Grids and the UC Grid' - niveditha


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

The UCLA Grid Portal -

Campus Grids and the UC Grid

http://grid.ucla.edu

Joan Slottow and Prakashan Korambath

Research Computing Technologies

UCLA Academic Technology Services

ucla grid architecture
UCLA Grid Architecture

Head node Cluster I

Credential mapped to local ID

Grid Appliance

Grid Portal

https

Uniform browser based interface

Head node Cluster II

Credential mapped to local ID

MyProxy Server

Storage Server

ION Visualization Server

Grid Appliance

single campus architecture
Single Campus Architecture

Campus

Grid

Portal

and

CA

A

C

A

C

Campus

MyProxy

Server

Campus

Storage

Server

  • Grid Portal accesses the storage server
  • via GridFTP.
  • No other access is provided.

A = Grid Appliance

C = Cluster and Cluster Head Node

multi campus architecture for the uc
Multi-Campus Architecture for the UC

UC

Grid

CA

UC

Register

Service

UC

User

Database

UC

MyProxy

UC Portal

Single CA is shared among all campuses

A

C

UCLA

Portal

UCSB

Portal

A

C

C

A

Storage

Server

C

A

C

C

UCLA

MyProxy

Storage

Server

UCSB

MyProxy

A

A

UCI

Portal

Storage

Server

All appliances talk to both their

campus portal and the UC portal

UCI

MyProxy

pools
Pools
  • Resource Pool:
    • Clusters contribute cycles to the resource pool
    • Clusters contribute applications to a resource pool:
    • TeraGrid cycles purchased by the campus can easily be shared.
  • Pool Users:
    • Every cluster user is also a pool user.
    • Many students and faculty members do not currently have login ids on any of the clusters. They can easily sign up to be pool-only users.
    • Each pool-only user is assigned a Storage Area.
  • Pool use hides all the details
  • Pool jobs
    • Target cluster selected by the UCLA Grid Portal.
    • Currently run applications only.
  • A similar sharing of resources and applications can be applied across the entire UC.
user types
User Types
  • 2 Types of users:
    • Cluster User:
      • Has a login ID on at least 1 cluster
    • Pool-Only User:
      • Does not have a login ID on any cluster
  • Both types of users have certificates/usernames to login to the Grid Portal
user types vs portals
User Types vs. Portals

Portal

Campus Portal

UC Portal

U

s

e

r

T

y

p

e

Pool-only

User

Can submit jobs to the campus pool

Can submit jobs to the

UC pool

Best choice for those

with access to clusters

on different campuses

-- can use all clusters,

UC-wide, he/she can

Access

Can submit jobs to the

UC pool

Can use those

clusters on campus

he/she can access

Can submit jobs to

the campus pool

Cluster

User

to login to a portal a user needs
To Login to a Portal a User Needs:
  • A certificate
  • A gridsphere account on that Portal
  • Additionally:
    • A Cluster User must be added to the gridmap file on the appliance node of each of the clusters on which he/she has a login id
    • A pool-only user needs to be assigned a storage area on the Grid Portal’s Storage Server.
workflow for the uc grid 1
Workflow -- For the UC Grid -- 1

User

Requested username/password pending

UC

Grid

CA

UC

Register

Service

UC

MyProxy

UC

User

Database

All authentication

is done at the campus

level

Unique?

Sends a message

to campus grid admin

Campus

Grid

Portal

Pool-only user

Form requesting

Proposed

username

Authentication

Cluster user

Sends a message

to cluster admin

Usernames are unique across the entire UC

workflow for the uc grid 2
Workflow -- For the UC Grid -- 2

Requested username/password pending

Campus

Grid

Admin

UC

Register

Service

UC

User

Database

UC

MyProxy

UC

Grid

CA

Creates and signs user certificate

Creates gridsphere account at UC Portal

Campus

Grid

Portal

Authenticates

Clicks Approve

Pushes certificate to UC MyProxy

Pushes certificate to campus MyProxy

Sends request +

Host Cert

Creates

GridSphere

account

(assigns storage

area)

Campus

MyProxy

UC Register

Client

pool job submission
Pool Job Submission
  • Submitting a pool job
how pools work
How Pools Work
  • Currently applications only
  • A cluster that contributes to the pool must create:
    • Guest login id
    • Certificate for Grid Portal
    • Put guest in the gridmap file
    • Mechanism that allocates resources for pool jobs is up to the cluster administrator
pool job submission14
Pool Job Submission

On Storage Server or Originating Custer

Database

User

Storage Area

Portal

Find the best cluster

Update

A

Select a guest user

C

Generate guest user proxy

Status

Change

?

Create a uniquely-named subdirectory

Add Database entry for this job

Guest User

Storage Area

Stage input files to target cluster

Pending

Active

Done

Error

Target Cluster

Job submitted via GRAM Job Service and the

guest user proxy

When notified by GRAM of a status change

Update the database

teragrid
TeraGrid
  • UGP acts as a client to the TeraGrid.
  • Their Globus Toolkit nodes are open to all of their users.
  • They allowed us to get the CA Signing Policy and CA Certificate Public Key for each of their clusters.
  • It only goes one way.

myproxy.ncsa.uiuc.edu

UCLA

Grid

Portal

Request for user proxy

User Proxy

Certificate

User proxy certificate returned because

UCLA Grid Portal can authenticate

user proxy certificate

DataStar at

SDSC

UCLA pool jobs currently can run on two TeraGrid clusters.

submitting jobs to the teragrid
Submitting Jobs to the TeraGrid

Submitting a job to the TeraGrid

grid development environment and ajax technologies
Grid Development Environment and Ajax Technologies

Ajax

  • Ajax -- Asynchronous JavaScript and XML
  • Create web applications that are more interactive
  • Only those parts of a page that change are updated

GDE

  • Create project, import files, save project
  • Fully integrated editor interface
  • GDE-generated Makefile or use pre-existing user Makefile
  • Can for as many windows as you want. (Can be on different clusters)
  • Tested in IE, Safari, Firefox
  • Uses Zimbra Open AJAX Toolkit (BSD license)
  • Compilations submitted to instantaneous queues (must run on the cluster, not on the appliance)
slide20
GDE

Ajax Grid Development Environment

GDE Project Directory

pools now
Pools -- Now
  • Single Pool
    • Every campus cluster user is also a pool user.
    • Pool-only users can only submit to the pool
    • Any cluster can contribute cycles to the pool
      • It can choose which of its applications it will contribute to the pool
  • When a pool job is submitted, the Portal decides which cluster will run it.
futures specialty pools
Futures -- Specialty Pools
  • Clusters can form Specialty Pools
    • A user must be approved by the pool admin to
    • A specialty pool includes a list of applications
    • Grid Portal databases for: pools and pool users
futures specialty pools24
Futures -- Specialty Pools

UCLA

UC

UCLA

UCI

Matlab

Amber

Q-Chem

Mathematica

Mathematica

Matlab

Amber

Q-Chem

UCSB

Mathematica

*

Amber

Fluent

Gromacs

Jaguar

*

Vasp

*

*

Mathematica

Corsica

Osiris

baby b2

TeraGrid

Joan is in the default pools

Frank is in the default pools and the UCLA Physics Pool

Gromacs

Jaguar

*

Prakashan is in the default and chemistry pools

* = CNSI

futures redundancy and failover

Futures -- Redundancy and Failover

This is a feature of MySQL which automatically

mirrors the database on the slaves.

MySQL Master

Slave

Portal

Load

Balancer

.

.

Disk

May be a shared file system

RAID5 or just like an appliance

LB

Portal

Load Balancer

LB

  • Also called big IP
  • It could be Apache or it could be hardware
  • Farms out work to the portals

MyProxy

MyProxy

RAID5

workflow for cluster user now single campus
Workflow for Cluster User -- Now -- Single Campus

User

Cluster

Head

node

Apply Page

Sends a message

to cluster admin

Sends a message

To grid admin

Approves and

adds to the

gridmap

file

SSH

authentication

Creates

gridsphere

account

Sends a message

To the user

Request for password, create and sign certificate, push it to MyProxy

workflow for pool only user now single campus
Workflow for Pool-only User -- Now -- Single Campus

User

ISIS

used at

UCLA

User authenticated if he/she belongs to the

UCLA community

Apply Page

Sends a message

To grid admin

link

Purse

(on MyProxy

Server)

Creates gridsphere account and assigns storage area on

storage server

Sends a message

To the user

User can now log in

workflow add a cluster on another campus 1
Workflow -- Add a Cluster on Another Campus -- 1

User

UC

Grid

CA

UC

Register

Service

UC

User

Database

UC

MyProxy

Other

Campus

Grid

Portal

Main Page

Username

Authentication

Sends a message

to cluster admin

Sends a message

to other campus grid admin

GridSphere account

not required

workflow add a cluster on another campus 2
Workflow -- Add a cluster on Another Campus -- 2

UC

User

Database

Other

Campus

Grid

Admin

UC

Register

Service

UC

MyProxy

UC

CA

Pushes certificate to Other campus MyProxy

Other

Campus

Grid

Portal

Authenticates

Clicks Approve

Sends request +

Host Cert

Creates

GridSphere

account

Other

Campus

MyProxy

UC Register

Client

portal decides where the job will run
Portal Decides Where the Job will Run

Amber Job Submitted

Clusters with Amber

Requirements?

Time,

Number of nodes

Can resources

meet requirements?

Current Load

Queue

Length

Select one

ucsb batch queue prediction algorithm
UCSB Batch Queue Prediction Algorithm

UCSB Prediction Algorithm:

Daniel Nurmi

75% of the jobs queued start in

317 seconds ~ 5 minutes

95% of the jobs queued start in

54,000 seconds ~1.5 hours

Hoffman cluster Parallel Queue

futures job status and the uc grid portal
Futures -- Job Status and the UC Grid Portal

Joan submits a job to:

- UCLA C1

- UCI C1

- UC Pool

From the UC Grid Portal

UC Portal

Jobs

DB

A

C1

UCLA

Portal

A

C2

Jobs

DB

C1

C2

A

A

Joan submits a job to:

- UCLA C1

- UCLA C2

- UCLA pool

From the UCLA Grid Portal

UCI

Portal

Jobs

DB