grid computing concepts appplications and technologies n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Grid Computing: Concepts, Appplications, and Technologies PowerPoint Presentation
Download Presentation
Grid Computing: Concepts, Appplications, and Technologies

Loading in 2 Seconds...

play fullscreen
1 / 188

Grid Computing: Concepts, Appplications, and Technologies - PowerPoint PPT Presentation


  • 83 Views
  • Uploaded on

Grid Computing: Concepts, Appplications, and Technologies. Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer Science The University of Chicago http://www.mcs.anl.gov/~foster.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Grid Computing: Concepts, Appplications, and Technologies' - trella


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
grid computing concepts appplications and technologies

Grid Computing:Concepts, Appplications, and Technologies

Ian Foster

Mathematics and Computer Science Division

Argonne National Laboratory

and

Department of Computer Science

The University of Chicago

http://www.mcs.anl.gov/~foster

Grid Computing in Canada Workshop, University of Alberta, May 1, 2002

outline
Outline
  • The technology landscape
  • Grid computing
  • The Globus Toolkit
  • Applications and technologies
    • Data-intensive; distributed computing; collaborative; remote access to facilities
  • Grid infrastructure
  • Open Grid Services Architecture
  • Global Grid Forum
  • Summary and conclusions
outline1
Outline
  • The technology landscape
  • Grid computing
  • The Globus Toolkit
  • Applications and technologies
    • Data-intensive; distributed computing; collaborative; remote access to facilities
  • Grid infrastructure
  • Open Grid Services Architecture
  • Global Grid Forum
  • Summary and conclusions
living in an exponential world 1 computing sensors
Living in an Exponential World(1) Computing & Sensors

Moore’s Law: transistor count doubles each 18 months

Magnetohydro-

dynamics

star formation

living in an exponential world 2 storage
Living in an Exponential World:(2) Storage
  • Storage density doubles every 12 months
  • Dramatic growth in online data (1 petabyte = 1000 terabyte = 1,000,000 gigabyte)
    • 2000 ~0.5 petabyte
    • 2005 ~10 petabytes
    • 2010 ~100 petabytes
    • 2015 ~1000 petabytes?
  • Transforming entire disciplines in physical and, increasingly, biological sciences; humanities next?
data intensive physical sciences
Data Intensive Physical Sciences
  • High energy & nuclear physics
    • Including new experiments at CERN
  • Gravity wave searches
    • LIGO, GEO, VIRGO
  • Time-dependent 3-D systems (simulation, data)
    • Earth Observation, climate modeling
    • Geophysics, earthquake modeling
    • Fluids, aerodynamic design
    • Pollutant dispersal scenarios
  • Astronomy: Digital sky surveys
ongoing astronomical mega surveys
Ongoing Astronomical Mega-Surveys
  • Large number of new surveys
    • Multi-TB in size, 100M objects or larger
    • In databases
    • Individual archives planned and under way
  • Multi-wavelength view of the sky
    • > 13 wavelength coverage within 5 years
  • Impressive early discoveries
    • Finding exotic objects by unusual colors
      • L,T dwarfs, high redshift quasars
    • Finding objects by time variability
      • Gravitational micro-lensing

MACHO

2MASS

SDSS

DPOSS

GSC-II

COBE MAP

NVSS

FIRST

GALEX

ROSAT

OGLE

...

crab nebula in 4 spectral regions
Crab Nebula in 4 Spectral Regions

X-ray

Optical

Infrared

Radio

coming floods of astronomy data
Coming Floods of Astronomy Data
  • The planned Large Synoptic Survey Telescope will produce over 10 petabytes per year by 2008!
    • All-sky survey every few days, so will have fine-grain time series for the first time
data intensive biology and medicine
Data Intensive Biology and Medicine
  • Medical data
    • X-Ray, mammography data, etc. (many petabytes)
    • Digitizing patient records (ditto)
  • X-ray crystallography
  • Molecular genomics and related disciplines
    • Human Genome, other genome databases
    • Proteomics (protein structure, activities, …)
    • Protein interactions, drug delivery
  • Virtual Population Laboratory (proposed)
    • Simulate likely spread of disease outbreaks
  • Brain scans (3-D, time dependent)
a brain is a lot of data mark ellisman ucsd
A Brainis a Lotof Data!(Mark Ellisman, UCSD)

And comparisons must be

made among many

We need to get to one micron to know location of every cell. We’re just now

starting to get to 10 microns – Grids will help get us there and further

an exponential world 3 networks or coefficients matter
An Exponential World: (3) Networks(Or, Coefficients Matter …)
  • Network vs. computer performance
    • Computer speed doubles every 18 months
    • Network speed doubles every 9 months
    • Difference = order of magnitude per 5 years
  • 1986 to 2000
    • Computers: x 500
    • Networks: x 340,000
  • 2001 to 2010
    • Computers: x 60
    • Networks: x 4000

Moore’s Law vs. storage improvements vs. optical improvements. Graph from Scientific American (Jan-2001) by Cleo Vilett, source Vined Khoslan, Kleiner, Caufield and Perkins.

outline2
Outline
  • The technology landscape
  • Grid computing
  • The Globus Toolkit
  • Applications and technologies
    • Data-intensive; distributed computing; collaborative; remote access to facilities
  • Grid infrastructure
  • Open Grid Services Architecture
  • Global Grid Forum
  • Summary and conclusions
evolution of the scientific process
Evolution of the Scientific Process
  • Pre-electronic
    • Theorize &/or experiment, alone or in small teams; publish paper
  • Post-electronic
    • Construct and mine very large databases of observational or simulation data
    • Develop computer simulations & analyses
    • Exchange information quasi-instantaneously within large, distributed, multidisciplinary teams
evolution of business
Evolution of Business
  • Pre-Internet
    • Central corporate data processing facility
    • Business processes not compute-oriented
  • Post-Internet
    • Enterprise computing is highly distributed, heterogeneous, inter-enterprise (B2B)
    • Outsourcing becomes feasible => service providers of various sorts
    • Business processes increasingly computing- and data-rich
the grid
The Grid

“Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations”

an example virtual organization cern s large hadron collider
An Example Virtual Organization: CERN’s Large Hadron Collider

1800 Physicists, 150 Institutes, 32 Countries

100 PB of data by 2010; 50,000 CPUs?

grid communities applications data grids for high energy physics

~PBytes/sec

~100 MBytes/sec

Offline Processor Farm

~20 TIPS

There is a “bunch crossing” every 25 nsecs.

There are 100 “triggers” per second

Each triggered event is ~1 MByte in size

~100 MBytes/sec

Online System

Tier 0

CERN Computer Centre

~622 Mbits/sec or Air Freight (deprecated)

Tier 1

FermiLab ~4 TIPS

France Regional Centre

Germany Regional Centre

Italy Regional Centre

~622 Mbits/sec

Tier 2

Tier2 Centre ~1 TIPS

Caltech ~1 TIPS

Tier2 Centre ~1 TIPS

Tier2 Centre ~1 TIPS

Tier2 Centre ~1 TIPS

HPSS

HPSS

HPSS

HPSS

HPSS

~622 Mbits/sec

Institute ~0.25TIPS

Institute

Institute

Institute

Physics data cache

~1 MBytes/sec

1 TIPS is approximately 25,000

SpecInt95 equivalents

Physicists work on analysis “channels”.

Each institute will have ~10 physicists working on one or more channels; data for these channels should be cached by the institute server

Pentium II 300 MHz

Pentium II 300 MHz

Pentium II 300 MHz

Pentium II 300 MHz

Tier 4

Physicist workstations

Grid Communities & Applications:Data Grids for High Energy Physics

www.griphyn.org www.ppdg.net www.eu-datagrid.org

slide19

Data Integration and Mining: (credit Sara Graves)

From Global Information to Local Knowledge

Emergency Response

Precision Agriculture

Urban

Environments

Weather

Prediction

the grid a brief history
The Grid:A Brief History
  • Early 90s
    • Gigabit testbeds, metacomputing
  • Mid to late 90s
    • Early experiments (e.g., I-WAY), academic software projects (e.g., Globus, Legion), application experiments
  • 2002
    • Dozens of application communities & projects
    • Major infrastructure deployments
    • Significant technology base (esp. Globus ToolkitTM)
    • Growing industrial interest
    • Global Grid Forum: ~500 people, 20+ countries
the grid world current status
The Grid World: Current Status
  • Dozens of major Grid projects in scientific & technical computing/research & education
    • www.mcs.anl.gov/~foster/grid-projects
  • Considerable consensus on key concepts and technologies
    • Open source Globus Toolkit™ a de facto standard for major protocols & services
  • Industrial interest emerging rapidly
    • IBM, Platform, Microsoft, Sun, Compaq, …
  • Opportunity: convergence of eScience and eBusiness requirements & technologies
outline3
Outline
  • The technology landscape
  • Grid computing
  • The Globus Toolkit
  • Applications and technologies
    • Data-intensive; distributed computing; collaborative; remote access to facilities
  • Grid infrastructure
  • Open Grid Services Architecture
  • Global Grid Forum
  • Summary and conclusions
grid technologies resource sharing mechanisms that
Grid Technologies:Resource Sharing Mechanisms That …
  • Address security and policy concerns of resource owners and users
  • Are flexible enough to deal with many resource types and sharing modalities
  • Scale to large number of resources, many participants, many program components
  • Operate efficiently when dealing with large amounts of data & computation
aspects of the problem
Aspects of the Problem
  • Need for interoperability when different groups want to share resources
    • Diverse components, policies, mechanisms
    • E.g., standard notions of identity, means of communication, resource descriptions
  • Need for shared infrastructure services to avoid repeated development, installation
    • E.g., one port/service/protocol for remote access to computing, not one per tool/appln
    • E.g., Certificate Authorities: expensive to run
  • A common need for protocols & services
the hourglass model
The Hourglass Model
  • Focus on architecture issues
    • Propose set of core services as basic infrastructure
    • Use to construct high-level, domain-specific solutions
  • Design principles
    • Keep participation cost low
    • Enable local control
    • Support for adaptation
    • “IP hourglass” model

A p p l i c a t i o n s

Diverse global services

Core

services

Local OS

layered grid architecture by analogy to internet architecture

Application

Application

Internet Protocol Architecture

“Coordinating multiple resources”: ubiquitous infrastructure services, app-specific distributed services

Collective

“Sharing single resources”: negotiating access, controlling use

Resource

“Talking to things”: communication (Internet protocols) & security

Connectivity

Transport

Internet

“Controlling things locally”: Access to, & control of, resources

Fabric

Link

Layered Grid Architecture(By Analogy to Internet Architecture)
globus toolkit
Globus Toolkit™
  • A software toolkit addressing key technical problems in the development of Grid-enabled tools, services, and applications
    • Offer a modular set of orthogonal services
    • Enable incremental development of grid-enabled tools and applications
    • Implement standard Grid protocols and APIs
    • Available under liberal open source license
    • Large community of developers & users
    • Commercial support
general approach
General Approach
  • Define Grid protocols & APIs
    • Protocol-mediated access to remote resources
    • Integrate and extend existing standards
    • “On the Grid” = speak “Intergrid” protocols
  • Develop a reference implementation
    • Open source Globus Toolkit
    • Client and server SDKs, services, tools, etc.
  • Grid-enable wide variety of tools
    • Globus Toolkit, FTP, SSH, Condor, SRB, MPI, …
  • Learn through deployment and applications
key protocols
Key Protocols
  • The Globus Toolkit™ centers around four key protocols
    • Connectivity layer:
      • Security: Grid Security Infrastructure (GSI)
    • Resource layer:
      • Resource Management: Grid Resource Allocation Management (GRAM)
      • Information Services: Grid Resource Information Protocol (GRIP) and Index Information Protocol (GIIP)
      • Data Transfer: Grid File Transfer Protocol (GridFTP)
  • Also key collective layer protocols
    • Info Services, Replica Management, etc.
globus toolkit structure

Job

manager

Job

manager

Globus Toolkit Structure

Service naming

Soft state

management

Reliable invocation

GRAM

MDS

GridFTP

MDS

???

Notification

GSI

GSI

GSI

Other Service

or Application

Compute

Resource

Data

Resource

connectivity layer protocols services
Connectivity LayerProtocols & Services
  • Communication
    • Internet protocols: IP, DNS, routing, etc.
  • Security: Grid Security Infrastructure (GSI)
    • Uniform authentication, authorization, and message protection mechanisms in multi-institutional setting
    • Single sign-on, delegation, identity mapping
    • Public key technology, SSL, X.509, GSS-API
    • Supporting infrastructure: Certificate Authorities, certificate & key management, …

GSI: www.gridforum.org/security/gsi

why grid security is hard
Why Grid Security is Hard
  • Resources being used may be extremely valuable & the problems being solved extremely sensitive
  • Resources are often located in distinct administrative domains
    • Each resource may have own policies & procedures
  • The set of resources used by a single computation may be large, dynamic, and/or unpredictable
    • Not just client/server
  • It must be broadly available & applicable
    • Standard, well-tested, well-understood protocols
    • Integration with wide variety of tools
grid security requirements
Grid Security Requirements

User View

Resource Owner View

1) Specify local access control

2) Auditing, accounting, etc.

3) Integration w/ local systemKerberos, AFS, license mgr.

4) Protection from compromisedresources

1) Easy to use

2) Single sign-on

3) Run applicationsftp,ssh,MPI,Condor,Web,…

4) User based trust model

5) Proxies/agents (delegation)

Developer View

API/SDK with authentication, flexible message protection,

flexible communication, delegation, ...Direct calls to various security functions (e.g. GSS-API)Or security integrated into higher-level SDKs: E.g. GlobusIO, Condor-G, MPICH-G2, HDF5, etc.

grid security infrastructure gsi
Grid Security Infrastructure (GSI)
  • Extensions to existing standard protocols & APIs
    • Standards: SSL/TLS, X.509 & CA, GSS-API
    • Extensions for single sign-on and delegation
  • Globus Toolkit reference implementation of GSI
    • SSLeay/OpenSSL + GSS-API + delegation
    • Tools and services to interface to local security
      • Simple ACLs; SSLK5 & PKINIT for access to K5, AFS, etc.
    • Tools for credential management
      • Login, logout, etc.
      • Smartcards
      • MyProxy: Web portal login and delegation
      • K5cert: Automatic X.509 certificate creation
gsi in action create processes at a and b that communicate access files at c

Single sign-on via “grid-id”

& generation of proxy cred.

Or: retrieval of proxy cred.

from online repository

Remote process

creation requests*

GSI-enabled

GRAM server

Authorize

Map to local id

Create process

Generate credentials

Ditto

GSI-enabled

GRAM server

Process

Process

Communication*

Local id

Local id

Kerberos

ticket

Restricted

proxy

Remote file

access request*

Restricted

proxy

User Proxy

GSI-enabled

FTP server

Proxy

credential

Authorize

Map to local id

Access file

* With mutual authentication

GSI in Action: “Create Processes at A and B that Communicate & Access Files at C”

User

Site B

(Unix)

Site A

(Kerberos)

Computer

Computer

Site C

(Kerberos)

Storage

system

gsi working group documents
GSI Working Group Documents
  • Grid Security Infrastructure (GSI) Roadmap
    • Informational draft overview of working group activities and documents
  • Grid Security Protocols & Syntax
    • X.509 Proxy Certificates
    • X.509 Proxy Delegation Protocol
    • The GSI GSS-API Mechanism
  • Grid Security APIs
    • GSS-API Extensions for the Grid
    • GSI Shell API
gsi futures
GSI Futures
  • Scalability in numbers of users & resources
    • Credential management
    • Online credential repositories (“MyProxy”)
    • Account management
  • Authorization
    • Policy languages
    • Community authorization
  • Protection against compromised resources
    • Restricted delegation, smartcards
community authorization

1. CAS request, with

user/group

CAS

resource names

membership

Does the

and operations

collective policy

resource/collective

authorize this

2. CAS reply, with

membership

request for this

capability

and resource CA info

user?

collective policy

information

Resource

3. Resource request,

authenticated with

Is this request

capability

authorized by

the

local policy

capability?

information

4. Resource reply

Is this request

authorized for

the CAS?

Community Authorization

User

Laura Pearlman, Steve Tuecke, Von Welch, others

resource layer protocols services
Resource LayerProtocols & Services
  • Grid Resource Allocation Management (GRAM)
    • Remote allocation, reservation, monitoring, control of compute resources
  • GridFTP protocol (FTP extensions)
    • High-performance data access & transport
  • Grid Resource Information Service (GRIS)
    • Access to structure & state information
  • Others emerging: Catalog access, code repository access, accounting, etc.
  • All built on connectivity layer: GSI & IP

GRAM, GridFTP, GRIS: www.globus.org

resource management
Resource Management
  • The Grid Resource Allocation Management (GRAM) protocol and client API allows programs to be started and managed on remote resources, despite local heterogeneity
  • Resource Specification Language (RSL) is used to communicate requirements
  • A layered architecture allows application-specific resource brokers and co-allocators to be defined in terms of GRAM services
    • Integrated with Condor, PBS, MPICH-G2, …
resource management architecture

Broker

Co-allocator

Resource Management Architecture

RSL

specialization

RSL

Application

Information Service

Queries

& Info

Ground RSL

Simple ground RSL

Local

resource

managers

GRAM

GRAM

GRAM

LSF

Condor

NQE

data access transfer
Data Access & Transfer
  • GridFTP: extended version of popular FTP protocol for Grid data access and transfer
  • Secure, efficient, reliable, flexible, extensible, parallel, concurrent, e.g.:
    • Third-party data transfers, partial file transfers
    • Parallelism, striping (e.g., on PVFS)
    • Reliable, recoverable data transfers
  • Reference implementations
    • Existing clients and servers: wuftpd, ncftp
    • Flexible, extensible libraries in Globus Toolkit
the grid information problem
The Grid Information Problem
  • Large numbers of distributed “sensors” with different properties
  • Need for different “views” of this information, depending on community membership, security constraints, intended purpose, sensor type
the globus toolkit solution mds 2
The Globus Toolkit Solution: MDS-2

Registration & enquiry protocols, information models, query languages

  • Provides standard interfaces to sensors
  • Supports different “directory” structures supporting various discovery/access strategies
globus applications and deployments
Globus Applications and Deployments
  • Application projects include
    • GriPhyN, PPDG, NEES, EU DataGrid, ESG, Fusion Collaboratory, etc., etc.
  • Infrastructure deployments include
    • DISCOM, NASA IPG, NSF TeraGrid, DOE Science Grid, EU DataGrid, etc., etc.
    • UK Grid Center, U.S. GRIDS Center
  • Technology projects include
    • Data Grids, Access Grid, Portals, CORBA, MPICH-G2, Condor-G, GrADS, etc., etc.
globus futures
Globus Futures
  • Numerous large projects are pushing hard on production deployment & application
    • Much will be learned in next 2 years!
  • Active R&D program, focused for example on
    • Security & policy for resource sharing
    • Flexible, high-perf., scalable data sharing
    • Integration with Web Services etc.
    • Programming models and tools
  • Community code development producing a true Open Grid Architecture
outline4
Outline
  • The technology landscape
  • Grid computing
  • The Globus Toolkit
  • Applications and technologies
    • Data-intensive; distributed computing; collaborative; remote access to facilities
  • Grid infrastructure
  • Open Grid Services Architecture
  • Global Grid Forum
  • Summary and conclusions
important grid applications
Important Grid Applications
  • Data-intensive
  • Distributed computing (metacomputing)
  • Collaborative
  • Remote access to, and computer enhancement of, experimental facilities
outline5
Outline
  • The technology landscape
  • Grid computing
  • The Globus Toolkit
  • Applications and technologies
    • Data-intensive; distributed computing; collaborative; remote access to facilities
  • Grid infrastructure
  • Open Grid Services Architecture
  • Global Grid Forum
  • Summary and conclusions
data intensive science 2000 2015
Data Intensive Science: 2000-2015
  • Scientific discovery increasingly driven by IT
    • Computationally intensive analyses
    • Massive data collections
    • Data distributed across networks of varying capability
    • Geographically distributed collaboration
  • Dominant factor: data growth (1 Petabyte = 1000 TB)
    • 2000 ~0.5 Petabyte
    • 2005 ~10 Petabytes
    • 2010 ~100 Petabytes
    • 2015 ~1000 Petabytes?

How to collect, manage,

access and interpret this

quantity of data?

Drives demand for “Data Grids” to handleadditional dimension of data access & movement

data grid projects
Data Grid Projects
  • Particle Physics Data Grid (US, DOE)
    • Data Grid applications for HENP expts.
  • GriPhyN (US, NSF)
    • Petascale Virtual-Data Grids
  • iVDGL (US, NSF)
    • Global Grid lab
  • TeraGrid (US, NSF)
    • Dist. supercomp. resources (13 TFlops)
  • European Data Grid (EU, EC)
    • Data Grid technologies, EU deployment
  • CrossGrid (EU, EC)
    • Data Grid technologies, EU emphasis
  • DataTAG (EU, EC)
    • Transatlantic network, Grid applications
  • Japanese Grid Projects (APGrid) (Japan)
    • Grid deployment throughout Japan
  • Collaborations of application scientists & computer scientists
  • Infrastructure devel. & deployment
  • Globus based
grid communities applications data grids for high energy physics1

~PBytes/sec

~100 MBytes/sec

Offline Processor Farm

~20 TIPS

There is a “bunch crossing” every 25 nsecs.

There are 100 “triggers” per second

Each triggered event is ~1 MByte in size

~100 MBytes/sec

Online System

Tier 0

CERN Computer Centre

~622 Mbits/sec or Air Freight (deprecated)

Tier 1

FermiLab ~4 TIPS

France Regional Centre

Germany Regional Centre

Italy Regional Centre

~622 Mbits/sec

Tier 2

Tier2 Centre ~1 TIPS

Caltech ~1 TIPS

Tier2 Centre ~1 TIPS

Tier2 Centre ~1 TIPS

Tier2 Centre ~1 TIPS

HPSS

HPSS

HPSS

HPSS

HPSS

~622 Mbits/sec

Institute ~0.25TIPS

Institute

Institute

Institute

Physics data cache

~1 MBytes/sec

1 TIPS is approximately 25,000

SpecInt95 equivalents

Physicists work on analysis “channels”.

Each institute will have ~10 physicists working on one or more channels; data for these channels should be cached by the institute server

Pentium II 300 MHz

Pentium II 300 MHz

Pentium II 300 MHz

Pentium II 300 MHz

Tier 4

Physicist workstations

Grid Communities & Applications:Data Grids for High Energy Physics

www.griphyn.org www.ppdg.net www.eu-datagrid.org

biomedical informatics research network birn
Biomedical InformaticsResearch Network (BIRN)
  • Evolving reference set of brains provides essential data for developing therapies for neurological disorders (multiple sclerosis, Alzheimer’s, etc.).
  • Today
    • One lab, small patient base
    • 4 TB collection
  • Tomorrow
    • 10s of collaborating labs
    • Larger population sample
    • 400 TB data collection: more brains, higher resolution
    • Multiple scale data integration and analysis
digital radiology hollebeek u pennsylvania

mammograms X-rays

MRI

CAT scans

endoscopies ...

Digital Radiology(Hollebeek, U. Pennsylvania)
  • Hospital digital data
    • Very large data sources: great clinical value to digital storage and manipulation and significant cost savings
    • 7 Terabytes per hospital per year
    • Dominated by digital images
  • Why mammography
    • Clinical need for film recall & computer analysis
    • Large volume ( 4,000 GB/year ) (57% of total)
    • Storage and records standards exist
    • Great clinical value
earth system grid anl lbnl llnl ncar isi ornl
Earth System Grid(ANL, LBNL, LLNL, NCAR, ISI, ORNL)
  • Enable a distributed community [of thousands] to perform computationally intensive analyses on large climate datasets
  • Via
    • Creation of Data Grid supporting secure, high-performance remote access
    • “Smart data servers” supporting reduction and analyses
    • Integration with environmental data analysis systems, protocols, and thin clients

www.earthsystemgrid.org (soon)

earth system grid architecture
Earth System Grid Architecture

Attribute Specification

Replica Catalog

Metadata Catalog

Application

Multiple Locations

Logical Collection and Logical File Name

MDS

Selected

Replica

Replica

Selection

Performance

Information &

Predictions

GridFTP commands

NWS

Disk Cache

TapeLibrary

Disk Array

Disk Cache

Replica Location 1

Replica Location 2

Replica Location 3

a universal access transport protocol
A UniversalAccess/Transport Protocol
  • Suite of communication libraries and related tools that support
    • GSI security
    • Third-party transfers
    • Parameter set/negotiate
    • Partial file access
    • Reliability/restart
    • Logging/audit trail
  • All based on a standard, widely deployed protocol
  • Integrated instrumentation
  • Parallel transfers
  • Striping (cf DPSS)
  • Policy-based access control
  • Server-side computation
  • [later]
and the universal protocol is gridftp
And the Universal Protocol is … GridFTP
  • Why FTP?
    • Ubiquity enables interoperation with many commodity tools
    • Already supports many desired features, easily extended to support others
  • We use the term GridFTP to refer to
    • Transfer protocol which meets requirements
    • Family of tools which implement the protocol
  • Note GridFTP > FTP
  • Note that despite name, GridFTP is not restricted to file transfer!
gridftp basic approach
GridFTP: Basic Approach
  • FTP is defined by several IETF RFCs
  • Start with most commonly used subset
    • Standard FTP: get/put etc., 3rd-party transfer
  • Implement RFCed but often unused features
    • GSS binding, extended directory listing, simple restart
  • Extend in various ways, while preserving interoperability with existing servers
    • Stripe/parallel data channels, partial file, automatic & manual TCP buffer setting, progress and extended restart
the gridftp family of tools
The GridFTP Family of Tools
  • Patches to existing FTP code
    • GSI-enabled versions of existing FTP client and server, for high-quality production code
  • Custom-developed libraries
    • Implement full GridFTP protocol, targeting custom use, high-performance
  • Custom-developed tools
    • E.g., high-performance striped FTP server
gridftp for efficient wan transfer
GridFTP for Efficient WAN Transfer
  • Transfer Tb+ datasets
    • Highly-secure authentication
    • Parallel transfer for speed
  • LLNL->Chicago transfer (slow site network interfaces):
  • FUTURE: Integrate striped GridFTP with parallel storage systems, e.g., HPSS

Parallel TransferFully utilizes bandwidth of

network interface on single nodes.

Parallel Filesystem

Parallel Filesystem

Striped TransferFully utilizes bandwidth of

Gb+ WAN using multiple nodes.

gridftp for user friendly visualization setup
GridFTP for User-Friendly Visualization Setup
  • High-res visualization is too large for display on a single system
    • Needs to be tiled, 24bit->16bit depth
    • Needs to be staged to display units
  • GridFTP/ActiveMural integration application performs tiling, data reduction, and staging in a single operation
    • PVFS/MPI-IO on server
    • MPI process group transforms data as needed before transfer
    • Performance is currently bounded by 100Mb/s NICs on display nodes
distributed computing visualization
Distributed Computing+Visualization

Remote CenterGenerates Tb+ datasets fromsimulation code

FLASH data transferredto ANL for visualizationGridFTP parallelismutilizes high bandwidth

(Capable of utilizing>Gb/s WAN links)

WAN Transfer

Chiba City

Visualization

code constructsand stores

high-resolutionvisualizationframes fordisplay onmany devices

Job SubmissionSimulation code submitted to

remote center for execution

on 1000s of nodes

LAN/WAN Transfer

User-friendly striped GridFTP application tiles the frames and stages tiles onto display nodes

ActiveMural DisplayDisplays very high resolutionlarge-screen dataset animations

  • FUTURE (1-5 yrs)
  • 10s Gb/s LANs, WANs
  • End-to-end QoS
  • Automated replica management
  • Server-side data reduction & analysis
  • Interactive portals
sc 2001 experiment simulation of hep tier 1 site
SC’2001 Experiment:Simulation of HEP Tier 1 Site
  • Tiered (Hierarchical) Site Structure
    • All data generated at lower tiers must be forwarded to the higher tiers
    • Tier 1 sites may have many sites transmitting to them simultaneously and will need to sink a substantial amount of bandwidth
    • We demonstrated the ability of GridFTP to support this at SC 2001 in the Bandwidth Challenge
    • 16 Sites, with 27 Hosts, pushed a peak of 2.8 Gbs to the showfloor in Denver with a sustained bandwidth of nearly 2 Gbs
the replica management problem
The ReplicaManagement Problem
  • Maintain a mapping between logical names for files and collections and one or more physical locations
  • Important for many applications
  • Example: CERN high-level trigger data
    • Multiple petabytes of data per year
    • Copy of everything at CERN (Tier 0)
    • Subsets at national centers (Tier 1)
    • Smaller regional centers (Tier 2)
    • Individual researchers will have copies
our approach to replica management
Our Approach to Replica Management
  • Identify replica cataloging and reliable replication as two fundamental services
    • Layer on other Grid services: GSI, transport, information service
    • Use as a building block for other tools
  • Advantage
    • These services can be used in a wide variety of situations
replica catalog structure a climate modeling example
Replica Catalog Structure: A Climate Modeling Example

Replica Catalog

Logical Collection

C02 measurements 1998

Logical Collection

C02 measurements 1999

Filename: Jan 1998

Filename: Feb 1998

Logical File Parent

Location

jupiter.isi.edu

Location

sprite.llnl.gov

Filename: Mar 1998

Filename: Jun 1998

Filename: Oct 1998

Protocol: gsiftp

UrlConstructor:

gsiftp://jupiter.isi.edu/

nfs/v6/climate

Filename: Jan 1998

Filename: Dec 1998

Protocol: ftp

UrlConstructor:

ftp://sprite.llnl.gov/

pub/pcmdi

Logical File Jan 1998

Logical File Feb 1998

Size: 1468762

giggle a scalable replication location service
Giggle: A Scalable Replication Location Service
  • Local replica catalogs maintain definitive information about replicas
  • Publish (perhaps approximate) information using soft state techniques
  • Variety of indexing strategies possible
griphyn app science cs grids
GriPhyN = App. Science + CS + Grids
  • GriPhyN = Grid Physics Network
    • US-CMS High Energy Physics
    • US-ATLAS High Energy Physics
    • LIGO/LSC Gravity wave research
    • SDSS Sloan Digital Sky Survey
    • Strong partnership with computer scientists
  • Design and implement production-scale grids
    • Develop common infrastructure, tools and services
    • Integration into the 4 experiments
    • Application to other sciences via “Virtual Data Toolkit”
  • Multi-year project
    • R&D for grid architecture (funded at $11.9M +$1.6M)
    • Integrate Grid infrastructure into experiments through VDT
griphyn institutions
U Florida

U Chicago

Boston U

Caltech

U Wisconsin, Madison

USC/ISI

Harvard

Indiana

Johns Hopkins

Northwestern

Stanford

U Illinois at Chicago

U Penn

U Texas, Brownsville

U Wisconsin, Milwaukee

UC Berkeley

UC San Diego

San Diego Supercomputer Center

Lawrence Berkeley Lab

Argonne

Fermilab

Brookhaven

GriPhyN Institutions
griphyn petascale virtual data grids
GriPhyN: PetaScale Virtual Data Grids

Production Team

Individual Investigator

Workgroups

~1 Petaop/s

~100 Petabytes

Interactive User Tools

Request Planning &

Request Execution &

Virtual Data Tools

Management Tools

Scheduling Tools

Resource

Other Grid

  • Resource
  • Security and
  • Other Grid

Security and

Management

  • Management
  • Policy
  • Services

Policy

Services

Services

  • Services
  • Services

Services

Transforms

Distributed resources(code, storage, CPUs,networks)

Raw data

source

griphyn ppdg data grid architecture

MCAT; GriPhyN catalogs

MDS

MDS

GDMP

DAGMAN, Kangaroo

GSI, CAS

Globus

GRAM

GridFTP; GRAM; SRM

GriPhyN/PPDGData Grid Architecture

Application

= initial solution is operational

DAG

Catalog Services

Monitoring

Planner

Info Services

DAG

Repl. Mgmt.

Executor

Policy/Security

Reliable Transfer

Service

Compute Resource

Storage Resource

Ewa Deelman, Mike Wilde

griphyn research agenda
GriPhyN Research Agenda
  • Virtual Data technologies
    • Derived data, calculable via algorithm
    • Instantiated 0, 1, or many times (e.g., caches)
    • “Fetch value” vs. “execute algorithm”
    • Potentially complex (versions, cost calculation, etc)
  • E.g., LIGO: “Get gravitational strain for 2 minutes around 200 gamma-ray bursts over last year”
  • For each requested data value, need to
    • Locate item materialization, location, and algorithm
    • Determine costs of fetching vs. calculating
    • Plan data movements, computations to obtain results
    • Execute the plan
virtual data in action

Fetch item

Virtual Data in Action
  • Data request may
    • Compute locally
    • Compute remotely
    • Access local data
    • Access remote data
  • Scheduling based on
    • Local policies
    • Global policies
    • Cost

Major facilities, archives

Regional facilities, caches

Local facilities, caches

griphyn research agenda cont
GriPhyN Research Agenda (cont.)
  • Execution management
    • Co-allocation (CPU, storage, network transfers)
    • Fault tolerance, error reporting
    • Interaction, feedback to planning
  • Performance analysis (with PPDG)
    • Instrumentation, measurement of all components
    • Understand and optimize grid performance
  • Virtual Data Toolkit (VDT)
    • VDT = virtual data services + virtual data tools
    • One of the primary deliverables of R&D effort
    • Technology transfer to other scientific domains
programs as community resources data derivation and provenance
Programs as Community Resources:Data Derivation and Provenance
  • Most scientific data are not simple “measurements”; essentially all are:
    • Computationally corrected/reconstructed
    • And/or produced by numerical simulation
  • And thus, as data and computers become ever larger and more expensive:
    • Programs are significant community resources
    • So are the executions of those programs
slide82

“I’ve come across some interesting data, but I need to understand the nature of the corrections applied when it was constructed before I can trust it for my purposes.”

“I’ve detected a calibration error in an instrument and want to know which derived data to recompute.”

Data

consumed-by/

generated-by

created-by

Transformation

Derivation

execution-of

“I want to apply an astronomical analysis program to millions of objects. If the results already exist, I’ll save weeks of computation.”

“I want to search an astronomical database for galaxies with certain characteristics. If a program that performs this analysis exists, I won’t have to write one from scratch.”

the chimera virtual data system griphyn project
The Chimera Virtual Data System(GriPhyN Project)
  • Virtual data catalog
    • Transformations, derivations, data
  • Virtual data language
    • Data definition + query
  • Applications include browsers and data analysis applications
cluster finding data pipeline

tsObj

tsObj

catalog

cluster

core

brg

field

tsObj

tsObj

field

brg

brg

field

core

brg

field

5

4

3

2

1

1

3

2

1

2

1

2

Cluster-finding Data Pipeline
virtual data in cms
Virtual Data in CMS

Virtual Data Long Term Vision of CMS:

CMS Note 2001/047, GRIPHYN 2001-16

early griphyn challenge problem cms data reconstruction
Early GriPhyN Challenge Problem:CMS Data Reconstruction

2) Launch secondary job on WI pool; input files via Globus GASS

Master Condor job running at Caltech

Secondary Condor job on WI pool

5) Secondary reports complete to master

Caltech workstation

6) Master starts reconstruction jobs via Globus jobmanager on cluster

3) 100 Monte Carlo jobs on Wisconsin Condor pool

9) Reconstruction job reports complete to master

4) 100 data files transferred via GridFTP, ~ 1 GB each

7) GridFTP fetches data from UniTree

NCSA Linux cluster

NCSA UniTree - GridFTP-enabled FTP server

8) Processed objectivity database stored to UniTree

Scott Koranda, Miron Livny, others

trace of a condor g physics run

Pre / Simulation Jobs / Post (UW Condor)

ooDigis at NCSA

ooHits at NCSA

Delay due to script error

Trace of a Condor-G Physics Run
outline6
Outline
  • The technology landscape
  • Grid computing
  • The Globus Toolkit
  • Applications and technologies
    • Data-intensive; distributed computing; collaborative; remote access to facilities
  • Grid infrastructure
  • Open Grid Services Architecture
  • Global Grid Forum
  • Summary and conclusions
distributed computing
Distributed Computing
  • Aggregate computing resources & codes
    • Multidisciplinary simulation
    • Metacomputing/distributed simulation
    • High-throughput/parameter studies
  • Challenges
    • Heterogeneous compute & network capabilities, latencies, dynamic behaviors
  • Example tools
    • MPICH-G2: Grid-aware MPI
    • Condor-G, Nimrod-G: parameter studies
multidisciplinary simulations aviation safety

Human Models

Multidisciplinary Simulations: Aviation Safety

Wing Models

  • Lift Capabilities
  • Drag Capabilities
  • Responsiveness

Stabilizer Models

Airframe Models

  • Deflection capabilities
  • Responsiveness

Crew Capabilities

- accuracy

- perception

- stamina

- re-action times

- SOPs

Engine Models

  • Braking performance
  • Steering capabilities
  • Traction
  • Dampening capabilities
  • Thrust performance
  • Reverse Thrust performance
  • Responsiveness
  • Fuel Consumption

Landing Gear Models

Whole system simulations are produced by coupling all of the sub-system simulations

mpich g2 a grid enabled mpi
MPICH-G2: A Grid-Enabled MPI
  • A complete implementation of the Message Passing Interface (MPI) for heterogeneous, wide area environments
    • Based on the Argonne MPICH implementation of MPI (Gropp and Lusk)
  • Requires services for authentication, resource allocation, executable staging, output, etc.
  • Programs run in wide area without change
    • Modulo accommodating heterogeneous communication performance
  • See also: MetaMPI, PACX, STAMPI, MAGPIE

www.globus.org/mpi

grid based computation challenges
Grid-based Computation: Challenges
  • Locate “suitable” computers
  • Authenticate with appropriate sites
  • Allocate resources on those computers
  • Initiate computation on those computers
  • Configure those computations
  • Select “appropriate” communication methods
  • Compute with “suitable” algorithms
  • Access data files, return output
  • Respond “appropriately” to resource changes
mpich g2 use of grid services

Locates

hosts

MDS

Submits multiple jobs

globusrun

Stages

executables

GASS

Coordinates startup

DUROC

Authenticates

GRAM

GRAM

GRAM

Detects termination

Initiates job

fork

LSF

LoadLeveler

Monitors/controls

P2

P2

P2

P1

P1

P1

Communicates via vendor-MPI and TCP/IP (globus-io)

% grid-proxy-init

% mpirun -np 256 myprog

MPICH-G2 Use of Grid Services

Generates

resource specification

mpirun

cactus allen dramlitsch seidel shalf radke
Cactus(Allen, Dramlitsch, Seidel, Shalf, Radke)
  • Modular, portable framework for parallel, multidimensional simulations
  • Construct codes by linking
    • Small core (flesh): mgmt services
    • Selected modules (thorns): Numerical methods, grids & domain decomps, visualization and steering, etc.
  • Custom linking/configuration tools
  • Developed for astrophysics, but not astrophysics-specific

Thorns

Cactus “flesh”

www.cactuscode.org

cactus an application framework for dynamic grid computing
Cactus: An ApplicationFramework for Dynamic Grid Computing
  • Cactus thorns for active management of application behavior and resource use
  • Heterogeneous resources, e.g.:
    • Irregular decompositions
    • Variable halo for managing message size
    • Msg compression (comp/comm tradeoff)
    • Comms scheduling for comp/comm overlap
  • Dynamic resource behaviors/demands, e.g.:
    • Perf monitoring, contract violation detection
    • Dynamic resource discovery & migration
    • User notification and steering
cactus example terascale computing

Gig-E

100MB/sec

17

4

2

2

12

OC-12 line

But only 2.5MB/sec)

12

5

5

SDSC IBM SP

1024 procs

5x12x17 =1020

NCSA Origin Array

256+128+128

5x12x(4+2+2) =480

Cactus Example:Terascale Computing
  • Solved EEs for gravitational waves (real code)
    • Tightly coupled, communications required through derivatives
    • Must communicate 30MB/step between machines
    • Time step take 1.6 sec
  • Used 10 ghost zones along direction of machines: communicate every 10 steps
  • Compression/decomp. on all data passed in this direction
  • Achieved 70-80% scaling, ~200GF (only 14% scaling without tricks)
cactus example 2 migration in action

Running

At UC

3 successive

contract

violations

Resource

discovery

& migration

Running

At UIUC

Load

applied

(migration

time not to scale)

Cactus Example (2):Migration in Action
ipg milestone 3 large computing node completed 12 2000

Whitcomb

IPG Milestone 3:Large Computing NodeCompleted 12/2000

high-lift subsonicwind tunnel model

Glenn

Cleveland, OH

Ames

Moffett Field, CA

Langley

Hampton, VA

Sharp

OVERFLOW on IPGusingGlobus and MPICH-G2 for intra-problem, wide area communication

Lomax

512 node SGI Origin 2000

Application POC: Mohammad J. Djomehri

Slide courtesy Bill Johnston, LBNL & NASA

high throughput computing condor
High-Throughput Computing:Condor
  • High-throughput computing platform for mapping many tasks to idle computers
  • Three major components
    • Scheduler manages pool(s) of [distributively owned or dedicated] computers
    • DAGman manages user task pools
    • Matchmaker schedules tasks to computers
  • Parameter studies, data analysis
  • Condor-G extensions support wide area execution in Grid environment

www.cs.wisc.edu/condor

defining a dag

Job A

Job B

Job C

Job D

Defining a DAG
  • A DAG is defined by a .dag file, listing each of its nodes and their dependencies:

# diamond.dag

Job A a.sub

Job B b.sub

Job C c.sub

Job D d.sub

Parent A Child B C

Parent B C Child D

  • Each node runs the Condor job specified by its accompanying Condor submit file
high throughput computing mathematicians solve nug30
High-Throughput Computing:Mathematicians Solve NUG30
  • Looking for the solution to the NUG30 quadratic assignment problem
  • An informal collaboration of mathematicians and computer scientists
  • Condor-G delivered 3.46E8 CPU seconds in 7 days (peak 1009 processors) in U.S. and Italy (8 sites)
  • 14,5,28,24,1,3,16,15,
  • 10,9,21,2,4,29,25,22,
  • 13,26,17,30,6,20,19,
  • 8,18,7,27,12,11,23

MetaNEOS: Argonne, Iowa, Northwestern, Wisconsin

outline7
Outline
  • The technology landscape
  • Grid computing
  • The Globus Toolkit
  • Applications and technologies
    • Data-intensive; distributed computing; collaborative; remote access to facilities
  • Grid infrastructure
  • Open Grid Services Architecture
  • Global Grid Forum
  • Summary and conclusions
access grid

Presenter

mic

Presenter

camera

Ambient mic

(tabletop)

Audience camera

Access Grid
  • High-end group work and collaboration technology
  • Grid services being used for discovery, configuration, authentication
  • O(50) systems deployed worldwide
  • Basis for SC’2001 SC Global event in November 2001
    • www.scglobal.org

www.accessgrid.org

outline8
Outline
  • The technology landscape
  • Grid computing
  • The Globus Toolkit
  • Applications and technologies
    • Data-intensive; distributed computing; collaborative; remote access to facilities
  • Grid infrastructure
  • Open Grid Services Architecture
  • Global Grid Forum
  • Summary and conclusions
grid enabled research facilities leverage investments
Grid-Enabled Research Facilities:Leverage Investments
  • Research instruments, satellites, particle accelerators, MRI machines, etc., cost a great deal
  • Data from those devices can be accessed and analyzed by many more scientists
    • Not just the team that gathered the data
  • More productive use of instruments
    • Calibration, data sampling during a run, via on-demand real-time processing
telemicroscopy grid based computing
Telemicroscopy & Grid-Based Computing

DATA ACQUISITION

PROCESSING,ANALYSIS

ADVANCEDVISUALIZATION

NETWORK

COMPUTATIONALRESOURCES

IMAGING

INSTRUMENTS

LARGE-SCALE

DATABASES

apan trans pacific telemicroscopy collaboration osaka u ucsd isi slide courtesy mark ellisman@ucsd
APAN Trans-PacificTelemicroscopy Collaboration, Osaka-U, UCSD, ISI(slide courtesy Mark Ellisman@UCSD)

1st

UHVEM

(Osaka, Japan)

NCMIR

(San Diego)

(Chicago)

STAR TAP

(San Diego)

SDSC

Tokyo XP

TransPAC

vBNS

Globus

CRL/MPT

UCSD

2nd

UHVEM

(Osaka, Japan)

NCMIR

(San Diego)

network for earthquake engineering simulation
Network forEarthquake Engineering Simulation
  • NEESgrid: US national infrastructure to couple earthquake engineers with experimental facilities, databases, computers, & each other
  • On-demand access to experiments, data streams, computing, archives, collaboration

Argonne, Michigan, NCSA, UIUC, USC www.neesgrid.org

experimental facilities can include field sites
“Experimental Facilities” Can Include Field Sites
  • Remotely controlled sensor grids for field studies, e.g., in seismology and biology
    • Wireless/satellite communications
    • Sensor net technology for low-cost communications
outline9
Outline
  • The technology landscape
  • Grid computing
  • The Globus Toolkit
  • Applications and technologies
    • Data-intensive; distributed computing; collaborative; remote access to facilities
  • Grid infrastructure
  • Open Grid Services Architecture
  • Global Grid Forum
  • Summary and conclusions
nature and role of grid infrastructure
Nature and Role of Grid Infrastructure
  • Persistent Grid infrastructure is critical to the success of many eScience projects
    • High-speed networks, certainly
    • Remotely accessible compute & storage
    • Persistent, standard services: PKI, directories, reservation, …
    • Operational & support procedures
  • Many projects creating such infrastructures
    • Production operation is the goal, but much to learn about how to create & operate
a national grid infrastructure

A

A

A

A

A

A

A

A

REGIONAL

A

A

A

A

A

REGIONAL

A

A

A

REGIONAL

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

REGIONAL

A

REGIONAL

A

A

A

A

A

A

A

A

A

A

A

A National Grid Infrastructure
example grid infrastructure projects
Example Grid Infrastructure Projects
  • I-WAY (1995): 17 U.S. sites for one week
  • GUSTO (1998): 80 sites worldwide, exp
  • NASA Information Power Grid (since 1999)
    • Production Grid linking NASA laboratories
  • INFN Grid, EU DataGrid, iVDGL, … (2001+)
    • Grids for data-intensive science
  • TeraGrid, DOE Science Grid (2002+)
    • Production Grids link supercomputer centers
  • U.S. GRIDS Center
    • Software packaging, deployment, support
the 13 6 tf teragrid computing at 40 gb s
The 13.6 TF TeraGrid:Computing at 40 Gb/s

Site Resources

Site Resources

26

HPSS

HPSS

4

24

External Networks

External Networks

8

5

Caltech

Argonne

External Networks

External Networks

NCSA/PACI

8 TF

240 TB

SDSC

4.1 TF

225 TB

Site Resources

Site Resources

HPSS

UniTree

NCSA, SDSC, Caltech, Argonne www.teragrid.org

teragrid details

574p IA-32 Chiba City

32

256p HP

X-Class

32

Argonne

64 Nodes

1 TF

0.25 TB Memory

25 TB disk

32

32

Caltech

32 Nodes

0.5 TF

0.4 TB Memory

86 TB disk

128p Origin

24

32

128p HP

V2500

32

HR Display & VR Facilities

24

8

8

5

5

92p IA-32

HPSS

24

HPSS

OC-12

ESnet

HSCC

MREN/Abilene

Starlight

Extreme

Black Diamond

4

Chicago & LA DTF Core Switch/Routers

Cisco 65xx Catalyst Switch (256 Gb/s Crossbar)

OC-48

Calren

OC-48

OC-12

NTON

GbE

OC-12 ATM

Juniper M160

NCSA

500 Nodes

8 TF, 4 TB Memory

240 TB disk

SDSC

256 Nodes

4.1 TF, 2 TB Memory

225 TB disk

Juniper M40

Juniper M40

OC-12

vBNS

Abilene

Calren

ESnet

OC-12

vBNS

Abilene

MREN

OC-12

OC-12

2

2

OC-12

OC-3

OC-3

Myrinet Clos Spine

8

4

UniTree

8

HPSS

2

= 32x 1GbE

Sun

Starcat

Myrinet Clos Spine

4

1024p IA-32

320p IA-64

1176p IBM SP

Blue Horizon

16

14

= 64x Myrinet

4

= 32x Myrinet

1500p Origin

Sun E10K

= 32x FibreChannel

= 8x FibreChannel

10 GbE

32 quad-processor McKinley Servers

(128p @ 4GF, 8GB memory/server)

32 quad-processor McKinley Servers

(128p @ 4GF, 12GB memory/server)

Fibre Channel Switch

16 quad-processor McKinley Servers

(64p @ 4GF, 8GB memory/server)

IA-32 nodes

Cisco 6509 Catalyst Switch/Router

TeraGrid (Details)
targeted starlight optical network connections

NW Univ (Chicago) StarLight Hub

UIC

Atlanta

I-WIRE

Chicago Cross connect

Ill Inst of Tech

ANL

Univ of Chicago

Indianapolis (Abilene NOC)

St Louis

GigaPoP

AMPATH

NCSA/UIUC

Targeted StarLightOptical Network Connections

CERN

Asia-Pacific

SURFnet

CA*net4

Vancouver

Seattle

NTON

Portland

U Wisconsin

San Francisco

NYC

Chicago

PSC

NTON

IU

NCSA

Asia-Pacific

DTF 40Gb

Los Angeles

Atlanta

San Diego

(SDSC)

AMPATH

www.startap.net

ca net 4 architecture
CA*net 4 Architecture

CANARIE

GigaPOP

ORAN DWDM

Carrier DWDM

Edmonton

Saskatoon

St. John’s

Calgary

Regina

Quebec

Winnipeg

Charlottetown

Thunder Bay

Montreal

Victoria

Ottawa

Vancouver

Fredericton

Halifax

Boston

Seattle

Chicago

New York

CA*net 4 node)

Toronto

Possible future CA*net 4 node

Windsor

slide120

Japan

Europe

Korea

STAR TAP

(USA)

China

Hong Kong

  • Philippines

Thailand

Vietnam

Malaysia

  • Sri Lanka

Singapore

Indonesia

Current status

2001(plan)

  • Australia
  • APAN Network Topology

2001.9.3

622Mbps x 2

ivdgl a global grid laboratory
iVDGL: A Global Grid Laboratory

“We propose to create, operate and evaluate, over asustained period of time, an international researchlaboratory for data-intensive science.”

From NSF proposal, 2001

  • International Virtual-Data Grid Laboratory
    • A global Grid laboratory (US, Europe, Asia, South America, …)
    • A place to conduct Data Grid tests “at scale”
    • A mechanism to create common Grid infrastructure
    • A laboratory for other disciplines to perform Data Grid tests
    • A focus of outreach efforts to small institutions
  • U.S. part funded by NSF (2001-2006)
    • $13.7M (NSF) + $2M (matching)
initial us ivdgl data grid

Tier1 (FNAL)

Proto-Tier2

Tier3 university

Initial US-iVDGL Data Grid

SKC

BU

Wisconsin

PSU

BNL

Fermilab

Hampton

Indiana

JHU

Caltech

UCSD

Florida

Brownsville

Other sites to be added in 2002

ivdgl international virtual data grid laboratory

Tier0/1 facility

Tier2 facility

Tier3 facility

10 Gbps link

2.5 Gbps link

622 Mbps link

Other link

iVDGL:International Virtual Data Grid Laboratory

U.S. PIs: Avery, Foster, Gardner, Newman, Szalay www.ivdgl.org

us ivdgl interoperability
US iVDGL Interoperability
  • US-iVDGL-1 Milestone (August 02)

US-iVDGL-1

Aug 2002

iGOC

ATLAS

SDSS/NVO

CMS

LIGO

1

1

2

2

1

1

2

2

transatlantic interoperability
Transatlantic Interoperability
  • iVDGL-2 Milestone (November 02)

DataTAG

iVDGL-2

Nov 2002

iGOC

Outreach

ATLAS

SDSS/NVO

CMS

LIGO

ANL

CS Research

UC

BNL

FNAL

CERN

BU

CIT

CIT

JHU

INFN

HU

PSU

UCSD

UK PPARC

IU

ANL

UTB

UF

U of A

LBL

UC

UWM

FNAL

UM

UCB

OU

IU

UTA

ISI

NU

UW

another example infn grid in italy
Another Example:INFN Grid in Italy
  • 20 sites, ~200 persons, ~90 FTEs, ~20 IT
  • Preliminary budget for 3 years: 9 M Euros
  • Activities organized around
    • S/w development with Datagrid, DataTAG….
    • Testbeds (financed by INFN) for DataGrid, DataTAG, US-EU Intergrid
    • Experiments applications
    • Tier1..Tiern prototype infrastructure
  • Large scale testbeds provided by LHC experiments, Virgo…..
u s grids center nsf middleware infrastructure program
U.S. GRIDS CenterNSF Middleware Infrastructure Program
  • GRIDS = Grid Research, Integration, Deployment, & Support
  • NSF-funded center to provide
    • State-of-the-art middleware infrastructure to support national-scale collaborative science and engineering
    • Integration platform for experimental middleware technologies
  • ISI, NCSA, SDSC, UC, UW
  • NMI software release one: May 2002

www.grids-center.org

outline10
Outline
  • The technology landscape
  • Grid computing
  • The Globus Toolkit
  • Applications and technologies
    • Data-intensive; distributed computing; collaborative; remote access to facilities
  • Grid infrastructure
  • Open Grid Services Architecture
  • Global Grid Forum
  • Summary and conclusions
globus toolkit evaluation
Globus Toolkit: Evaluation (+)
  • Good technical solutions for key problems, e.g.
    • Authentication and authorization
    • Resource discovery and monitoring
    • Reliable remote service invocation
    • High-performance remote data access
  • This & good engineering is enabling progress
    • Good quality reference implementation, multi-language support, interfaces to many systems, large user base, industrial support
    • Growing community code base built on tools
globus toolkit evaluation1
Globus Toolkit: Evaluation (-)
  • Protocol deficiencies, e.g.
    • Heterogeneous basis: HTTP, LDAP, FTP
    • No standard means of invocation, notification, error propagation, authorization, termination, …
  • Significant missing functionality, e.g.
    • Databases, sensors, instruments, workflow, …
    • Virtualization of end systems (hosting envs.)
  • Little work on total system properties, e.g.
    • Dependability, end-to-end QoS, …
    • Reasoning about system properties
globus toolkit structure1

Job

manager

Job

manager

Globus Toolkit Structure

Service naming

Soft state

management

Reliable invocation

GRAM

MDS

GridFTP

MDS

???

Notification

GSI

GSI

GSI

Other Service

or Application

Compute

Resource

Data

Resource

Lots of good mechanisms, but (with the exception of GSI) not that easily

incorporated into other systems

open grid services architecture
Open Grid Services Architecture
  • Service orientation to virtualize resources
  • Define fundamental Grid service behaviors
    • Core set required, others optional
    • A unifying framework for interoperability & establishment of total system properties
  • Integration with Web services and hosting environment technologies
    • Leverage tremendous commercial base
    • Standard IDL accelerates community code
  • Delivery via open source Globus Toolkit 3.0
    • Leverage GT experience, code, mindshare
web services
“Web Services”
  • Increasingly popular standards-based framework for accessing network applications
    • W3C standardization; Microsoft, IBM, Sun, others
  • WSDL: Web Services Description Language
    • Interface Definition Language for Web services
  • SOAP: Simple Object Access Protocol
    • XML-based RPC protocol; common WSDL target
  • WS-Inspection
    • Conventions for locating service descriptions
  • UDDI: Universal Desc., Discovery, & Integration
    • Directory for Web services
web services example database service
Web Services Example:Database Service
  • WSDL definition for “DBaccess” porttype defines operations and bindings, e.g.:
    • Query(QueryLanguage, Query, Result)
    • SOAP protocol
  • Client C, Java, Python, etc., APIs can then be generated

DBaccess

transient service instances
Transient Service Instances
  • “Web services” address discovery & invocation of persistent services
    • Interface to persistent state of entire enterprise
  • In Grids, must also support transient service instances, created/destroyed dynamically
    • Interfaces to the states of distributed activities
    • E.g. workflow, video conf., dist. data analysis
  • Significant implications for how services are managed, named, discovered, and used
    • In fact, much of our work is concerned with the management of service instances
the grid service interfaces service data
The Grid Service =Interfaces + Service Data

Reliable invocation

Authentication

Service data access

Explicit destruction

Soft-state lifetime

Notification

Authorization

Service creation

Service registry

Manageability

Concurrency

GridService

… other interfaces …

Service

data

element

Service

data

element

Service

data

element

Implementation

Hosting environment/runtime

(“C”, J2EE, .NET, …)

open grid services architecture fundamental structure
Open Grid Services Architecture:Fundamental Structure

1) WSDL conventions and extensions for describing and structuring services

  • Useful independent of “Grid” computing

2) Standard WSDL interfaces & behaviors for core service activities

  • portTypes and operations => protocols
wsdl conventions extensions
WSDL Conventions & Extensions
  • portType (standard WSDL)
    • Define an interface: a set of related operations
  • serviceType (extensibility element)
    • List of port types: enables aggregation
  • serviceImplementation (extensibility element)
    • Represents actual code
  • service (standard WSDL)
    • instanceOf extension: map descr.->instance
  • compatibilityAssertion (extensibility element)
    • portType, serviceType, serviceImplementation
structure of a grid service

instanceOf

instanceOf

instanceOf

instanceOf

serviceImplementation

serviceImplementation

cA

serviceType

cA

serviceType

cA

cA

compatibilityAssertion

=

Structure of a Grid Service

service

service

service

service

Service

Instantiation

Service

Description

PortType

PortType

PortType

=

Standard WSDL

standard interfaces behaviors four interrelated concepts
Standard Interfaces & Behaviors:Four Interrelated Concepts
  • Naming and bindings
    • Every service instance has a unique name, from which can discover supported bindings
  • Information model
    • Service data associated with Grid service instances, operations for accessing this info
  • Lifecycle
    • Service instances created by factories
    • Destroyed explicitly or via soft state
  • Notification
    • Interfaces for registering interest and delivering notifications
ogsa interfaces and operations defined to date
OGSA Interfaces and OperationsDefined to Date
  • GridService Required
    • FindServiceData
    • Destroy
    • SetTerminationTime
  • NotificationSource
    • SubscribeToNotificationTopic
    • UnsubscribeToNotificationTopic
  • NotificationSink
    • DeliverNotification
  • Factory
    • CreateService
  • PrimaryKey
    • FindByPrimaryKey
    • DestroyByPrimaryKey
  • Registry
    • RegisterService
    • UnregisterService
  • HandleMap
    • FindByHandle

Authentication, reliability are binding properties

Manageability, concurrency, etc., to be defined

service data
Service Data
  • A Grid service instance maintains a set of service data elements
    • XML fragments encapsulated in standard <name, type, TTL-info> containers
    • Includes basic introspection information, interface-specific data, and application data
  • FindServiceData operation (GridService interface) queries this information
    • Extensible query language support
  • See also notification interfaces
    • Allows notification of service existence and changes in service data
grid service example database service
Grid Service Example:Database Service
  • A DBaccess Grid service will support at least two portTypes
    • GridService
    • DBaccess
  • Each has service data
    • GridService: basic introspection information, lifetime, …
    • DBaccess: database type, query languages supported, current load, …, …

Grid

Service

DBaccess

Name, lifetime, etc.

DB info

lifetime management
Lifetime Management
  • GS instances created by factory or manually; destroyed explicitly or via soft state
    • Negotiation of initial lifetime with a factory (=service supporting Factory interface)
  • GridService interface supports
    • Destroy operation for explicit destruction
    • SetTerminationTime operation for keepalive
  • Soft state lifetime management avoids
    • Explicit client teardown of complex state
    • Resource “leaks” in hosting environments
factory
Factory
  • Factory interface’s CreateService operation creates a new Grid service instance
    • Reliable creation (once-and-only-once)
  • CreateService operation can be extended to accept service-specific creation parameters
  • Returns a Grid Service Handle (GSH)
    • A globally unique URL
    • Uniquely identifies the instance for all time
    • Based on name of a home handleMap service
transient database services
Transient Database Services

“Create a database service”

“What services can you create?”

Grid

Service

Grid

Service

DBaccess

Factory

DBaccess

Instance name, etc.

Name, lifetime, etc.

Factory info

DB info

“What database services exist?”

Grid

Service

Grid

Service

Registry

DBaccess

Instance name, etc.

Name, lifetime, etc.

Registry info

DB info

example data mining for bioinformatics
Example:Data Mining for Bioinformatics

Community

Registry

Mining

Factory

Database

Service

BioDB 1

Compute Service Provider

User

Application

.

.

.

.

.

.

“I want to create

a personal database

containing data on

e.coli metabolism”

Database

Service

Database

Factory

BioDB n

Storage Service Provider

example data mining for bioinformatics1
Example:Data Mining for Bioinformatics

“Find me a data mining service, and somewhere to store data”

Community

Registry

Mining

Factory

Database

Service

BioDB 1

Compute Service Provider

User

Application

.

.

.

.

.

.

Database

Service

Database

Factory

BioDB n

Storage Service Provider

example data mining for bioinformatics2
Example:Data Mining for Bioinformatics

Community

Registry

Mining

Factory

GSHs for Mining

and Database

factories

Database

Service

BioDB 1

Compute Service Provider

User

Application

.

.

.

.

.

.

Database

Service

Database

Factory

BioDB n

Storage Service Provider

example data mining for bioinformatics3
Example:Data Mining for Bioinformatics

Community

Registry

Mining

Factory

Database

Service

“Create a data mining service with initial lifetime 10”

BioDB 1

Compute Service Provider

User

Application

.

.

.

.

.

.

“Create a

database with initial lifetime 1000”

Database

Service

Database

Factory

BioDB n

Storage Service Provider

example data mining for bioinformatics4
Example:Data Mining for Bioinformatics

Community

Registry

Mining

Factory

Database

Service

“Create a data mining service with initial lifetime 10”

BioDB 1

Miner

Compute Service Provider

User

Application

.

.

.

.

.

.

“Create a

database with initial lifetime 1000”

Database

Service

Database

Factory

BioDB n

Database

Storage Service Provider

example data mining for bioinformatics5
Example:Data Mining for Bioinformatics

Community

Registry

Mining

Factory

Database

Service

Query

BioDB 1

Miner

Compute Service Provider

User

Application

.

.

.

.

.

.

Query

Database

Service

Database

Factory

BioDB n

Database

Storage Service Provider

example data mining for bioinformatics6
Example:Data Mining for Bioinformatics

Community

Registry

Mining

Factory

Database

Service

Query

BioDB 1

Miner

Keepalive

Compute Service Provider

User

Application

.

.

.

.

.

.

Query

Database

Service

Database

Factory

Keepalive

BioDB n

Database

Storage Service Provider

example data mining for bioinformatics7
Example:Data Mining for Bioinformatics

Community

Registry

Mining

Factory

Database

Service

BioDB 1

Miner

Keepalive

Compute Service Provider

User

Application

.

.

.

.

.

.

Results

Database

Service

Database

Factory

Keepalive

Results

BioDB n

Database

Storage Service Provider

example data mining for bioinformatics8
Example:Data Mining for Bioinformatics

Community

Registry

Mining

Factory

Database

Service

BioDB 1

Miner

Compute Service Provider

User

Application

.

.

.

.

.

.

Database

Service

Database

Factory

Keepalive

BioDB n

Database

Storage Service Provider

example data mining for bioinformatics9
Example:Data Mining for Bioinformatics

Community

Registry

Mining

Factory

Database

Service

BioDB 1

Compute Service Provider

User

Application

.

.

.

.

.

.

Database

Service

Database

Factory

Keepalive

BioDB n

Database

Storage Service Provider

notification interfaces
Notification Interfaces
  • NotificationSource for client subscription
    • One or more notification generators
      • Generates notification message of a specific type
      • Typed interest statements: E.g., Filters, topics, …
      • Supports messaging services, 3rd party filter services, …
    • Soft state subscription to a generator
  • NotificationSink for asynchronous delivery of notification messages
  • A wide variety of uses are possible
    • E.g. Dynamic discovery/registry services, monitoring, application error notification, …
notification example
Notification Example
  • Notifications can be associated with any (authorized) service data elements

Grid

Service

Grid

Service

Notification

Sink

DBaccess

Name, lifetime, etc.

Name, lifetime, etc.

Notification

Source

DB info

DB info

Subscribers

notification example1
Notification Example
  • Notifications can be associated with any (authorized) service data elements

Grid

Service

Grid

Service

Notification

Sink

“Notify me of

new data about

membrane proteins”

DBaccess

Name, lifetime, etc.

Name, lifetime, etc.

Notification

Source

DB info

DB info

Subscribers

notification example2
Notification Example
  • Notifications can be associated with any (authorized) service data elements

Grid

Service

Grid

Service

Notification

Sink

DBaccess

Keepalive

Name, lifetime, etc.

Name, lifetime, etc.

Notification

Source

DB info

DB info

Subscribers

notification example3
Notification Example
  • Notifications can be associated with any (authorized) service data elements

Grid

Service

Grid

Service

Notification

Sink

DBaccess

New data

Name, lifetime, etc.

Name, lifetime, etc.

Notification

Source

DB info

DB info

Subscribers

open grid services architecture summary
Open Grid Services Architecture:Summary
  • Service orientation to virtualize resources
    • Everything is a service
  • From Web services
    • Standard interface definition mechanisms: multiple protocol bindings, local/remote transparency
  • From Grids
    • Service semantics, reliability and security models
    • Lifecycle management, discovery, other services
  • Multiple “hosting environments”
    • C, J2EE, .NET, …
recap the grid service
Recap: The Grid Service

Reliable invocation

Authentication

Service data access

Explicit destruction

Soft-state lifetime

Notification

Authorization

Service creation

Service registry

Manageability

Concurrency

GridService

… other interfaces …

Service

data

element

Service

data

element

Service

data

element

Implementation

Hosting environment/runtime

(“C”, J2EE, .NET, …)

ogsa and the globus toolkit
OGSA and the Globus Toolkit
  • Technically, OGSA enables
    • Refactoring of protocols (GRAM, MDS-2, etc.)—while preserving all GT concepts/features!
    • Integration with hosting environments: simplifying components, distribution, etc.
    • Greatly expanded standard service set
  • Pragmatically, we are proceeding as follows
    • Develop open source OGSA implementation
      • Globus Toolkit 3.0; supports Globus Toolkit 2.0 APIs
    • Partnerships for service development
    • Also expect commercial value-adds
gt3 an open source ogsa compliant globus toolkit
GT3: An Open Source OGSA-Compliant Globus Toolkit
  • GT3 Core
    • Implements Grid service interfaces & behaviors
    • Reference impln of evolving standard
    • Java first, C soon, C#?
  • GT3 Base Services
    • Evolution of current Globus Toolkit capabilities
    • Backward compatible
  • Many other Grid services

Other Grid

GT3

Services

Data

Services

GT3 Base Services

GT3 Core

hmm isn t this just another object model
Hmm, Isn’t This Just Another Object Model?
  • Well, yes, in a sense
    • Strong encapsulation
    • We (can) profit greatly from experiences of previous object-based systems
  • But
    • Focus on encapsulation not inheritance
    • Does not require OO implementations
    • Value lies in specific behaviors: lifetime, notification, authorization, …, …
    • Document-centric not type-centric
grids and ogsa research challenges
Grids and OGSA:Research Challenges
  • Grids pose profound problems, e.g.
    • Management of virtual organizations
    • Delivery of multiple qualities of service
    • Autonomic management of infrastructure
    • Software and system evolution
  • OGSA provides foundation for tackling these problems in a rigorous fashion?
    • Structured establishment/maintenance of global properties
    • Reasoning about total system properties
summary
Summary
  • OGSA represents refactoring of current Globus Toolkit protocols and integration with Web services technologies
  • Several desirable features
    • Significant evolution of functionality
    • Uniform IDL facilitates code sharing
    • Allows for alignment of potentially divergent directions (e.g., info service, service registry, monitoring)
evolution
Evolution
  • This is not happening all at once
    • We have an early prototype of Core (alpha release May?)
    • Next we will work on Base, others
    • Full release by end of 2002??
    • Establishing partnerships for other services
  • Backward compatibility
    • API level seems straightforward
    • Protocol level: gateways?
    • We need input on best strategies
for more information
For More Information
  • OGSA architecture and overview
    • “The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration”, at www.globus.org/ogsa
  • Grid service specification
    • At www.globus.org/ogsa
  • Open Grid Services Infrastructure WG, GGF
    • www.gridforum.org/ogsi (?), soon
  • Globus Toolkit OGSA prototype
    • www.globus.org/ogsa
outline11
Outline
  • The technology landscape
  • Grid computing
  • The Globus Toolkit
  • Applications and technologies
    • Data-intensive; distributed computing; collaborative; remote access to facilities
  • Grid infrastructure
  • Open Grid Services Architecture
  • Global Grid Forum
  • Summary and conclusions
ggf objectives
GGF Objectives
  • An open process for development of standards
    • Grid “Recommendations” process modeled after Internet Standards Process (IETF)
  • A forum for information exchange
    • Experiences, patterns, structures
  • A regular gathering to encourage shared effort
    • In code development: libraries, tools…
    • Via resource sharing: shared Grids
    • In infrastructure: consensus standards
ggf groups
Working Groups

Tightly focused on development of a spec or set of related specs

Protocol, API, etc.

Finite set of objectives and schedule of milestones

Research Groups

More exploratory than Working Groups

Focused on understanding requirements, taxonomies, models, methods for solving a particular set of related problems

May be open-ended but with a definite set of objectives and milestones to drive progress

GGF Groups
  • Groups are approved and evaluated by a GGF Steering Group (GFSG) based on written charters. Among the criteria for group formation:
  • Is this work better done (or already being done) elsewhere, e.g. IETF, W3C?
  • Are the leaders involved and/or in touch with relevant efforts elsewhere?
getting involved
Getting Involved
  • Participate in a GGF Meeting
    • 3x/year, last one had 500 people
    • July 21-24, 2002 in Edinburgh (with HPDC)
    • October 15-17, 2002 in Chicago
  • Join a working group or research group
    • Electronic participation via mailing lists (see www.gridforum.org)
grid events
Grid Events
  • Global Grid Forum: working meeting
    • Meets 3 times/year, alternates U.S.-Europe, with July meeting as major event
  • HPDC: major academic conference
    • HPDC’11 in Scotland with GGF’5, July 2002
  • Other meetings with Grid content include
    • SC’XY, CCGrid, Globus Retreat

www.gridforum.org, www.hpdc.org

outline12
Outline
  • The technology landscape
  • Grid computing
  • The Globus Toolkit
  • Applications and technologies
    • Data-intensive; distributed computing; collaborative; remote access to facilities
  • Grid infrastructure
  • Open Grid Services Architecture
  • Global Grid Forum
  • Summary and conclusions
summary1
Summary
  • The Grid problem: Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations
    • Real application communities emerging
    • Significant infrastructure deployments
    • Substantial open source/architecture technology base: Globus Toolkit
    • Pathway defined to industrial adoption, via open source + OGSA
    • Rich set of intellectual challenges
major application communities are emerging
Major Application Communities are Emerging
  • Intellectual buy-in, commitment
    • Earthquake engineering: NEESgrid
    • Exp. physics, etc.: GriPhyN, PPDG, EU Data Grid
    • Simulation: Earth System Grid, Astrophysical Sim. Collaboratory
    • Collaboration: Access Grid
  • Emerging, e.g.
    • Bioinformatics Grids
    • National Virtual Observatory
major infrastructure deployments are underway
Major Infrastructure Deployments are Underway
  • For example:
    • NSF “National Technology Grid”
    • NASA “Information Power Grid”
    • DOE ASCI DISCOM Grid
    • DOE Science Grid’
    • EU DataGrid
    • iVDGL
    • NSF Distributed Terascale Facility (“TeraGrid”)
    • DOD MOD Grid
a rich technology base has been constructed
A Rich Technology Basehas been Constructed
  • 6+ years of R&D have produced a substantial code base based on open architecture principles: esp. the Globus Toolkit, including
    • Grid Security Infrastructure
    • Resource directory and discovery services
    • Secure remote resource access
    • Data Grid protocols, services, and tools
  • Essentially all projects have adopted this as a common suite of protocols & services
  • Enabling wide range of higher-level services
pathway defined to industrial adoption
Pathway Defined toIndustrial Adoption
  • Industry need
    • eScience applications, service provider models, need to integrate internal infrastructures, collaborative computing in general
  • Technical capability
    • Maturing open source technology base
    • Open Grid Services Architecture enables integration with industry standards
  • Result likely to be exponential industrial uptake
rich set of intellectual challenges
Rich Set of Intellectual Challenges
  • Transforming the Internet into a robust, usable computational platform
  • Delivering (multi-dimensional) qualities of service within large systems
  • Community dynamics and collaboration modalities
  • Program development methodologies and tools for Internet-scale applications
  • Etc., etc., etc.
new programs
New Programs
  • U.K. eScience program
  • EU 6th Framework
  • U.S. Committee on Cyberinfrastructure
  • Japanese Grid initiative
u s cyberinfrastructure draft recommendations
U.S. Cyberinfrastructure:Draft Recommendations
  • New INITIATIVE to revolutionize science and engineering research at NSF and worldwide to capitalize on new computing and communications opportunities 21st Century Cyberinfrastructure includes supercomputing, but also massive storage, networking, software, collaboration, visualization, and human resources
    • Current centers (NCSA, SDSC, PSC) are a key resource for the INITIATIVE
    • Budget estimate: incremental $650M/year (continuing)
  • An INITIATIVE OFFICE with a highly placed, credible leader empowered to
    • Initiate competitive, discipline-driven path-breaking applications within NSF of cyberinfrastructure which contribute to the shared goals of the INITIATIVE
    • Coordinate policy and allocations across fields and projects. Participants across NSF directorates, Federal agencies, and international e-science
    • Develop high quality middleware and other software that is essential and special to scientific research
    • Manage individual computational, storage, and networking resources at least 100x larger than individual projects or universities can provide.
for more information1
For More Information
  • The Globus Project™
    • www.globus.org
  • Grid concepts, projects
    • www.mcs.anl.gov/~foster
  • Open Grid Services Architecture
    • www.globus.org/ogsa
  • Global Grid Forum
    • www.gridforum.org
  • GriPhyN project
    • www.griphyn.org