the emergence of open grid standards
Download
Skip this Video
Download Presentation
The Emergence of Open Grid Standards

Loading in 2 Seconds...

play fullscreen
1 / 114

The Emergence of Open Grid Standards - PowerPoint PPT Presentation


  • 215 Views
  • Uploaded on

Managed shared virtual systems. Computer science research. Open Grid Services Arch. Web services, etc. Real standards Multiple implementations. Globus Toolkit. Internet standards. Defacto standard Single implementation. The Emergence of Open Grid Standards. Increased functionality,

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'The Emergence of Open Grid Standards' - Gabriel


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
the emergence of open grid standards

Managed shared

virtual systems

Computer science research

Open Grid

Services Arch

Web services, etc.

Real standards

Multiple implementations

Globus Toolkit

Internet

standards

Defacto standard

Single implementation

The Emergence ofOpen Grid Standards

Increased functionality,

standardization

Custom

solutions

1990

1995

2000

2005

2010

open grid services architecture
Open Grid Services Architecture
  • Service orientation to virtualize resources
    • Everything is a service
  • From Web services
    • Standard interface definition mechanisms
    • Evolving set of other standards: security, etc.
  • From Grids (Globus Toolkit)
    • Service semantics, reliability & security models
    • Lifecycle management, discovery, other services
  • A framework for the definition & management of composable, interoperable services

“The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration”, Foster, Kesselman, Nick, Tuecke, 2002

web services
Web Services
  • XML-based distributed computing technology
  • Web service = a server process that exposes typed ports to the network
  • Described by the Web Services Description Language, an XML document that contains
    • Type of message(s) the service understands & types of responses & exceptions it returns
    • “Methods” bound together as “port types”
    • Port types bound to protocols as “ports”
  • A WSDL document completely defines a service and how to access it
wsdl example
WSDL Example

<wsdl:definitions targetNamespace=“…”>

<wsdl:types>

<schema>

<xsd:element name=“fooInput” …/>

<xsd:element name=“fooOutput” …/>

</schema>

</wsdl:types>

<wsdl:message name=“fooInputMessage”>

<part name=“parameters” element=“fooInput”/>

</wsdl:message>

<wsdl:message name=“fooOutputMessage”>

<part name=“parameters” element=“fooOutput”/>

</wsdl:message>

<wsdl:portType name=“fooInterface”>

<wsdl:operation name=“foo”>

<input message=“fooInput”/>

<output message = “fooOutput”/>

</wsdl:operation>

</wsdl:portType>

</wsdl:definitions>

transient service instances
Transient Service Instances
  • “Web services” address discovery & invocation of persistent services
    • Interface to persistent state of entire enterprise
  • In Grids, must also support transient services, created/destroyed dynamically
    • Interfaces to the states of distributed activities
    • E.g. workflow, video conf., dist. data analysis
  • Significant implications for how services are managed, named, discovered, and used
    • In fact, much of our work is concerned with the management of services
ogsa structure
OGSA Structure
  • A standard substrate: the Grid service
    • Standard interfaces and behaviors that address key distributed system issues: naming, service state, lifetime, notification
    • A Grid service is a Web service
  • … supports standard service specifications
    • Agreement, data access & integration, workflow, security, policy, diagnostics, etc.
    • Target of current & planned GGF efforts
  • … and arbitrary application-specific services based on these & other definitions
overview
Overview
  • Grid background
  • Open Grid Services Architecture
  • Open Grid Services Infrastructure
  • Beyond OGSI: other OGSA services
  • Globus Toolkit v3 implementation
  • Early GT3 performance results
  • Scientific and commercial perspectives
  • Summary
ogsi specification
OGSI Specification
  • Defines WSDL conventions and extensions
    • For describing and naming services
    • Working with W3C WSDL working group to drive OGSI extensions into WSDL 1.2
  • Defines fundamental interfaces (using extended WSDL) and behaviors that define a Grid Service
    • A unifying framework for interoperability & establishment of total system properties
  • http://www.ggf.org/ogsi-wg
fundamental interfaces behaviors
FundamentalInterfaces & Behaviors
  • OGSI defines basic patterns of interaction, which can be combined with each other and with custom patterns in a myriad of ways
  • OGSI Specification focuses on:
    • Atomic, composable patterns in the form of portTypes/interfaces
      • Define operations & associated service data elements
    • A model for how these are composed
      • Compatible with WSDL 1.2
  • Complete service descriptions are left to other groups that are defining real services
ogsi standard web services interfaces behaviors
OGSI: Standard Web Services Interfaces & Behaviors
  • Naming and bindings (basis for virtualization)
    • Every service instance has a unique name, from which can discover supported bindings
  • Lifecycle (basis for fault resilient state management)
    • Service instances created by factories
    • Destroyed explicitly or via soft state
  • Information model (basis for monitoring & discovery)
    • Service data (attributes) associated with GS instances
    • Operations for querying and setting this info
    • Asynchronous notification of changes to service date
  • Service Groups (basis for registries & collective svcs)
    • Group membership rules & membership management
  • Base Fault type
open grid services infrastructure ogsi

Resource allocation

Create Service

Grid Service Handle

Service data Keep-alives Notifications Service invocation

Service discovery

Register Service

Service instances

Open Grid ServicesInfrastructure (OGSI)

Authentication & authorization are applied to all requests

Service factory

Service requestor (e.g. user application)

Service registry

Interactions standardized using WSDL

open grid services infrastructure

Client

  • Introspection:
  • What port types?
  • What policy?
  • What state?

GridService

(required)

Other standard interfaces:

factory,

notification,

collections

Grid Service

Handle

Service

data

element

Service

data

element

Service

data

element

handle

resolution

Grid Service

Reference

Open Grid Services Infrastructure
  • Lifetime management
  • Explicit destruction
  • Soft-state lifetime

Data

access

Implementation

Hosting environment/runtime

(“C”, J2EE, .NET, …)

gwsdl
GWSDL
  • OGSI requires interface extension/composition
  • We worked within W3C WSDL working group to define standard interface extension in WSDL 1.2 that meets OGSI requirements
  • But could not wait for WSDL 1.2
  • So defined gwsdl:portType that extends WSDL 1.1 portType with:
    • WSDL 1.2 portType extension
    • WSDL 1.2 open content model
  • Define GWSDL  WSDL 1.1 & 1.2 mappings
gwsdl example
GWSDL Example

<wsdl:definitions>

<wsdl:types>…</wsdl:types>

<wsdl:message>…</wsdl:message>

<gwsdl:portType name=“foo”extends=“ns:bar ogsi:GridService”>

<wsdl:operation name=“op1”>…</wsdl:operation>

<wsdl:operation name=“op2”>…</wsdl:operation>

<ogsi:serviceData … />

</gwsdl:portType>

</wsdl:definitions>

ogsi service data
OGSI Service Data
  • Attributes: Publicly visible state of the service
  • Want to bring full power of XML to attributes
    • getXXX/setXXX is too limiting
      • How to get/set multiple?
      • Want richer queries across attributes (e.g., join)
    • Use XML Schema, XPath, XQuery, XSLT, etc.
    • OGSI service data:
      • Attributes defined using XML Schema
      • Attributes combined into a single (logical) document within the service
      • Rich pull/push/set operations against service data document
  • Should declare attributes in WSDL interface
example reliable file transfer service

Client

Client

Client

Request and manage file transfer operations

Notf’n

Source

Policy

Grid

Service

Fault

Monitor

Pending

interfaces

Query &/or

subscribe

to service data

Performance

service

data

elements

Policy

Perf.

Monitor

Faults

Example:Reliable File Transfer Service

File

Transfer

Internal

State

Data transfer operations

ogsi implementations
OGSI Implementations
  • Globus Toolkit version 3.0 (Java, C client)
  • U Virginia OGSI.NET (.NET)
  • LBNL pyGlobus (Python)
  • U Edinburgh (.NET)
  • U Manchester (PERL)
  • Fujitsu Unicore (Java)
overview1
Overview
  • Grid background
  • Open Grid Services Architecture
  • Open Grid Services Infrastructure
  • Beyond OGSI: other OGSA services
  • Globus Toolkit v3 implementation
  • Early GT3 performance results
  • Scientific and commercial perspectives
  • Summary
open grid services architecture1

Structured Data

Integration

Job Submission

Brokering

Workflow

Registry

Banking

Authorisation

Transformation

Structured Data Access

Data Transport

Resource Usage

Web Services: Basic Functionality

Structured Data

Relational

XML

Semi-structured

Open Grid Services Architecture

Users in Problem Domain X

Applications in Problem Domain X

Application & Integration Technology for Problem Domain X

Generic Virtual Service Access and Integration Layer

OGSA

OGSI: Interface to Grid Infrastructure

Compute, Data & Storage Resources

-

Distributed

Virtual Integration Architecture

ogsa standardization implementation
OGSA Standardization & Implementation
  • OGSI defines core interfaces and behaviors for manageable services
    • Supported by strong open source technology & major commercial vendors
  • Efforts are underway within GGF, OASIS, and other bodies to define standards for
    • Agreement negotiation
    • Common management model
    • Data access and integration
    • Security and policy
    • Etc., etc., etc.
transactions contexts
Transactions & Contexts
  • WS-Coordination & WS-Transaction
    • IBM/MS (not in standards org)
  • WS-CAF (Coordinated Application Framework)
    • Sun/Oracle/Arjuna/Fujitsu (not in standards org)
    • WS-CTX (Context)
    • WS-CF (Coordination Framework)
    • WS-TXM (Transaction Management)
  • Both take a “contextualization” approach
    • Context (id) threaded through SOAP header
    • OGSI for context creation, naming & lifecycle???
security standards
Security Standards
  • Many core security standards are from IETF
    • X.509, Kerberos, etc.
    • X.509 Proxy Certificates (RFC soon hopefully)
      • Used by Globus Toolkit GSI
  • OASIS appears to be leader in Web services security standards
    • WS-Security: SOAP message security
    • SAML: signed assertions using XML
    • XACML: access control lists using XML
  • GGF OGSA Security WG evaluating security specifications for applicability to OGSA
ibm microsoft ws security architecture
IBM/MicrosoftWS Security Architecture
  • Large set of specifications for doing Web services security, most of which should be appropriate for OGSA
  • Announced April 2002
  • Initial spec in July 2002 (WS-Security)
    • Submitted to OASIS
  • New crops of specs arrive periodically
    • WS-Policy*, WS-Trust, WS-Federation, etc.
    • But… Not yet in any standards organization
ws security current proposed wss specs
WS SecurityCurrent/Proposed WSS-specs

WS-Secure

Conversation

WS-Federation

WS-Authorization

WS-Policy

WS-Trust

WS-Privacy

WS-Security

In progress

SOAP Foundation

proposed

promised

oasis saml xacml
OASIS SAML & XACML
  • SAML: Security Assertion Markup Language
    • Good for asserting properties such as group membership, etc
  • XACML: eXtensible Access Control Markup Language
    • For defining access control policies
  • These are gaining considerable momentum, but WS-Policy* leaves them in question
project liberty alliance
Project Liberty Alliance
  • V1.x specifications for identity federation
    • Allows cross-organization identification
    • Privacy preserving model
  • Based on SAML
ws agreement
WS-Agreement
  • Recall key criteria of a Grid:
    • Coordinates resources that are not subject to centralized control …
    • using standard, open, general-purpose protocols and interfaces …
    • to deliver non-trivial qualities of service.
  • Implies need to express and negotiate agreements that govern the delivery of services to clients
    • Agreement = what will be done, QoS, billing, compliance monitoring
ws agreement contents
WS-Agreement Contents
  • Standard agreement language
    • A composition of a set of terms that govern a service’s behavior with respect to clients
    • Agreement language uses WS-Policy (currently)
    • Standard attributes for terms that express current state of negotiation
    • Other groups define specific terms
  • Standard agreement negotiation protocol
    • Establish, monitor, re-negotiate agreement
    • Expressed using OGSI GWSDL interfaces
    • Each agreement represented by a service
ws agreement applicability
WS-Agreement Applicability
  • All interesting Web/Grid services interactions will be governed by agreements!
  • WS-Agreement (language and interfaces) should be used by specifications that define domain-specific services
    • Data services
    • Job submission
    • Specialized services
    • Etc.
platform s community scheduling framework ws agreement
Platform’s Community Scheduling Framework & WS-Agreement

GT3.0

GT3.0

LSF

Queuing

Service

RM Adapter

Internet

Job

Service

Site B

Reservation

Service

GT3.0

PBS

Site A

RM Adapter

Site C

Reservation Agreement Exchange

GT3.0

SGE

RM

Adaptor

wsdm wsmf cmm
WSDM / WSMF / CMM
  • OASIS Web Services Distributed Management (WSDM) technical committee
    • Management using/of Web Services
    • HP submitted its Web Services Management Framework (WSMF) to WSDM in July 2003
      • WS-Events: event schema, subscription, message queues
      • WSMF-Foundation: management using Web services
      • WSM: management of Web services
  • GGF Common Management Model (CMM) WG
    • IBM submission overlaps WSMF-Foundation
  • Working to bring WSDM & CMM together, based on OGSI foundation
data as service ogsa data access integration
Data as Service:OGSA Data Access & Integration
  • Service-oriented treatment of data appears to have significant advantages
    • Leverage OGSI introspection, lifetime, etc.
    • Compatibility with Web services
  • Standard service interfaces being defined
    • Service data: e.g., schema
    • Derive new data services from old (views)
    • Externalize to e.g. file/database format
    • Perform queries or other operations
data services
Data Services
  • GGF Data Access and Integration Svcs (DAIS)
    • OGSI-compliant interfaces to access relational and XML databases
    • Needs to be generalized to encompass other data sources (see next slide…)
  • Generalized DAIS becomes the foundation for:
    • Replication: Data located in multiple locations
    • Federation: Composition of multiple sources
    • Provenance: How was data generated?
ogsa data services foster tuecke unger eds
“OGSA Data Services”(Foster, Tuecke, Unger, eds.)
  • Describes conceptual model for representing all manner of data sources as Web services
    • Database, filesystems, devices, programs, …
    • Integrates WS-Agreement
  • Data service is an OGSI-compliant Web service that implements one or more of base data interfaces:
    • DataDescription, DataAccess, DataFactory, DataManagement
    • These would be extended and combined for specific domains (including DAIS)
data access integration services

1a. Request to Registry for sources of data about “x”

SOAP/HTTP

service creation

API interactions

Registry

1b. Registry responds with Factory handle

2a. Request to Factory for access to database

Factory

Client

2c. Factory returns handle of GDS to client

2b. Factory creates GridDataService to manage access

3a. Client queries GDS with XPath, SQL, etc

XML / Relational database

Grid Data Service

3c. Results of query returned to client as XML

3b. GDS interacts with database

Data Access & Integration Services

Slide Courtesy Malcolm Atkinson, UK eScience Center

overview2
Overview
  • Grid background
  • Open Grid Services Architecture
  • Open Grid Services Infrastructure
  • Beyond OGSI: other OGSA services
  • Globus Toolkit v3 implementation
  • Early GT3 performance results
  • Scientific and commercial perspectives
  • Summary
the globus alliance
The Globus Alliance
  • A group of people with a common mission:
    • “Make Grid computing an everyday reality.”
  • Argonne/U.Chicago, USC-ISI, EPCC, & KTH-PDC
    • Led by Ian Foster (Argonne), Carl Kesselman (ISI), Malcolm Atkinson (EPCC), Lennart Johnsson (PDC)
    • Includes researchers, software developers, software architects & designers, systems engineers, & others
    • Collaborations (or at least acquaintances) with most Grid activities in the world
  • All activities contribute to our common mission
    • Research
    • Software development (prototypes, reference implns)
    • Application consulting
    • Infrastructure consulting
globus toolkit a story of evolution
Globus Toolkit:A Story of Evolution
  • Definition of Grid problem has been stable since original Globus Project proposal in 1995
    • Though we’ve gotten better at articulating it
  • But our approach to its solution has evolved:
    • From APIs and custom protocols…
    • to standard protocols…
    • to Grid services (OGSA)
  • Driven by experience implementing and deploying the Globus Toolkit, and building real applications with it
globus toolkit v3 0
Globus Toolkit® v3.0
  • All of the GT v2.4 services and clients
  • Complete Java implementation of OGSI v1.0
    • Rich, container-based implementation
    • Built on Apache Axis
  • Globus “proprietary” services built on OGSI
    • Managed Jobs (akin to GT2 GRAM)
    • Reliable File Transfer (RFT)
    • Index Services (akin to GT2 GIIS)
  • Some services not yet OGSI-fied:
    • GridFTP, Replica Location Services (RLS)
gt3 distribution
GT3 Distribution

GT-OGSA

Grid Service

Infrastructure

GT2

Components

RLS

The focus of this presentation

slide52

GT-OGSA Grid Service Infrastructure

Grid Service Container

User-Defined Services

Base Services

System-Level Services

OGSI Spec Implementation

Security Infrastructure

Web Service Engine

Hosting Environment

slide53

GT-OGSA Grid Service Infrastructure

Grid Service Container

User-Defined Services

Base Services

System-Level Services

OGSI Spec Implementation

Security Infrastructure

Web Service Engine

Hosting Environment

ogsi implementation
OGSI Implementation
  • GT3 includes a set of primitives that fully implement the interfaces and behaviors defined in the OGSI Specification
    • Defines how entities can create, discover and interact with a Grid service
  • The OGSI Specification defines a protocol: GT3 provides a programming model for that protocol
implementation of the gridservice porttype gridserviceimpl and persistentgridserviceimpl
destroy()

setServiceData()

findServiceData()

requestTerminationAfter()

requestTerminationBefore()

addOperationProvider()

(See docs for complete set of methods)

Implementation

of the OGSI Spec

Implementation of the GridService portType: GridServiceImpl and PersistentGridServiceImpl

Additional

functionality can be

added to a

Grid Service using

OperationProviders

Deployment descriptor

<parameter name=“operationProviders” value=“<className>”>

building an ogsi compliant grid service using gt3
Building an OGSI-Compliant Grid Service using GT3

Write service-specific

logic that also

implements the GT3

OperationProvider

interface

building an ogsi compliant grid service using gt31
Building an OGSI-Compliant Grid Service using GT3

Write service-specific

logic that also

implements the GT3

OperationProvider

interface

Combine with one of the

two GT3 implementations

of base GridService

functionality:

GridServiceImpl or

PersistentGridServiceImpl

building an ogsi compliant grid service using gt32
Building an OGSI-Compliant Grid Service using GT3

Write service-specific

logic that also

implements the GT3

OperationProvider

interface

Combine with one of the

two GT3 implementations

of base GridService

functionality:

GridServiceImpl or

PersistentGridServiceImpl

An

OGSI-Compliant

grid service

building an ogsi compliant grid service using gt33
Building an OGSI-Compliant Grid Service using GT3

OperationProviders are

configured at deployment time

or added at runtime

Write service-specific

logic that also

implements the GT3

OperationProvider

interface

Combine with one of the

two GT3 implementations

of base GridService

functionality:

GridServiceImpl or

PersistentGridServiceImpl

An

OGSI-Compliant

grid service

a grid service can be composed of multiple operationproviders
A Grid Service Can be Composed of Multiple OperationProviders
  • OPs can be designed as atomic bits of functionality to facilitate reuse
  • OP approach eases the task of bringing legacy code into OGSI-compliance
  • OPs allow Grid Services to be formed dynamically (in contrast to the inheritance approach)
several operationproviders are included in the gt3 distribution
Several OperationProviders are Included in the GT3 Distribution

NotificationSourceProvider

HandleResolverProvider

ServiceGroupProvider

ServiceGroupRegistrationProvider

FactoryProvider

ogsi specification notifications
OGSI Specification: Notifications
  • Our NotificationSourceProvider implementation allows any Grid Service to become a sender of notification messages
  • A subscribe request on a NotificationSource triggers the creation of a NotificationSubscription service
  • A NotificationSink can receive notification msgs from NotificationSources
    • Sinks are not required to implement the GridService portType
  • Notifications can be set on SDEs
ogsi specification factory
OGSI Specification: Factory

Factory portType

  • Factories create services
  • Factories are typically persistent services
  • Factory is an optional OGSI interface

(Grid Services can also be instantiated by other mechanisms)

ogsi specification service groups
OGSI Specification: Service Groups

Service group portTypes

  • A ServiceGroup is a grid service that maintains information about a group of other grid services
  • The classic registry model can be implemented with the ServiceGroup portTypes
  • A grid service can belong to more than one ServiceGroup
  • Members of a ServiceGroup can be heterogeneous or homogenous
  • Each entry in a service group can be represented as its own service
  • Service group portTypes are optional OGSI interfaces
ogsi specification handle resolver

Service locator

  • Includes 0 or more Grid Service Handles (GSHs)
  • Includes 0 or more Grid Service References (GSRs)
OGSI Specification: Handle Resolver

HandleResolver portType

  • Defines a means for resolving a GSH (Grid Service Handle) to a GSR (Grid Service Reference)
    • A GSH points to a Grid Service

(GT3 uses a hostname-based GSH scheme)

    • A GSR specifies how to communicate with the Grid Service

(GT3 currently supports SOAP over HTTP, so GSRs are in WSDL format)

a service creation scenario
A Service Creation Scenario*

Registry

1. From a

known registry,

the client

discovers a

factory by

querying the

service data

of the

registry

Client

* All scenarios are offered as examples and are not prescriptive

a service creation scenario1
A Service Creation Scenario

Registry

Factory

2. The client calls

the createService

operation on the

factory

1. From a

known registry,

the client

discovers a

factory by

querying the

service data

of the

registry

Client

a service creation scenario2
A Service Creation Scenario

Registry

Factory

2. The client calls

the createService

operation on the

factory

1. From a

known registry,

the client

discovers a

factory by

querying the

service data

of the

registry

3. The factory

creates a

service

Client

Service

a service creation scenario3
A Service Creation Scenario

Registry

Factory

2. The client calls

the createService

operation on the

factory

1. From a

known registry,

the client

discovers a

factory by

querying the

service data

of the

registry

3. The factory

creates a

service

4. The factory

returns a locator to

the newly created

service

Client

Service

a service creation scenario4
A Service Creation Scenario

Registry

Factory

2. The client calls

the createService

operation on the

factory

1. From a

known registry,

the client

discovers a

factory by

querying the

service data

of the

registry

3. The factory

creates a

service

4. The factory

returns a locator to

the newly created

service

Client

Service

5. The client and

service interact

a notification scenario
A Notification Scenario

NotificationSource

1. NotificationSink calls

the subscribe operation

on NotificationSource

NotificationSink

a notification scenario1
A Notification Scenario

NotificationSource

1. NotificationSink calls

the subscribe operation

on NotificationSource

2.NotificationSource creates a subscription

service

NotificationSubscription

NotificationSink

a notification scenario2
A Notification Scenario

NotificationSource

1. NotificationSink calls

the subscribe operation

on NotificationSource

2.NotificationSource creates a subscription

service

3. Notification

Source returns a

locator to the

subscription service

NotificationSubscription

NotificationSink

a notification scenario3
A Notification Scenario

4.a deliverNotification

stream continues

for the lifetime of

NotificationSubscription

NotificationSource

1. NotificationSink calls

the subscribe operation

on NotificationSource

2.NotificationSource creates a subscription

service

3. Notification

Source returns a

locator to the

subscription service

NotificationSubscription

NotificationSink

4.b The NotificationSink and

Subscription service interact

to perform lifetime

management

a notification scenario4
A Notification Scenario

4.a deliverNotification

stream continues

for the lifetime of

NotificationSubscription

The sole mandated

cardinality: 1 to 1

NotificationSource

1. NotificationSink calls

the subscribe operation

on NotificationSource

subscribe

2.NotificationSource creates a subscription

service

3. Notification

Source returns a

locator to the

subscription service

NotificationSubscription

NotificationSink

4.b The NotificationSink and

Subscription service interact

to perform lifetime

management

slide76

GT-OGSA Grid Service Infrastructure

Grid Service Container

User-Defined Services

Base Services

System-Level Services

OGSI Spec Implementation

Security Infrastructure

Web Service Engine

Hosting Environment

security infrastructure
Security Infrastructure
  • Transport Layer Security/Secure Socket Layer (TLS/SSL)
    • To be deprecated
  • SOAP Layer Security
    • Based on WS-Security, XML Encryption, XML Signature
  • GT3 uses X.509 identity certificates for authentication
  • It also uses X.509 Proxy certificates to support delegation and single sign-on, updated to conform to latest IETF/GGF draft
slide78

GT-OGSA Grid Service Infrastructure

Grid Service Container

User-Defined Services

Base Services

System-Level Services

OGSI Spec Implementation

Security Infrastructure

Web Service Engine

Hosting Environment

system level services
System-Level Services
  • General-purpose services that facilitate the use of Grid services in production environments
  • The 3.0 distribution includes three system-level services
    • An administration service
    • A logging service
    • A management service
slide80

GT-OGSA Grid Service Infrastructure

Grid Service Container

User-Defined Services

Base Services

System-Level Services

OGSI Spec Implementation

Security Infrastructure

Web Service Engine

Hosting Environment

grid service container
Grid Service Container

Includes the OGSI Implementation, security infrastructure and system-level services, plus:

  • Service activation, deactivation, construction, destruction, etc.
  • Service data element placeholders that allow you to fetch service data values dynamically, at query time
  • Evaluator framework (supporting ByXPath and ByName notifications and queries)
  • Interceptor/callback framework (allows one to intercept certain service lifecycle events)
grid service container cont
Grid Service Container (cont.)

Layers in the Web Services Model

Interface Layer

OGSI Spec is here

Transport/binding

layer (e.g., GT3: SOAP over HTTP)

Transport Layer

Implementation Layer

Container is here

slide83

GT-OGSA Grid Service Infrastructure

Grid Service Container

User-Defined Services

Base Services

System-Level Services

OGSI Spec Implementation

Security Infrastructure

Web Service Engine

Hosting Environment

hosting environment
Hosting Environment

GT3 currently offers support for four Java hosting environments:

  • Embedded
  • Standalone
  • Servlet
  • EJB
virtual hosting environment framework
Virtual Hosting Environment Framework
  • Virtual hosting allows Grid services to be distributed across several remote containers
  • Useful in implementing solutions for problems common to distributed computing
    • Load balancing
    • User account sandboxing
slide86

GT-OGSA Grid Service Infrastructure

Grid Service Container

User-Defined Services

Base Services

System-Level Services

OGSI Spec Implementation

Security Infrastructure

Web Service Engine

Hosting Environment

resource management
Resource Management
  • GRAM Architecture rendered in OGSA
  • The MMJFS runs as an unprivileged user, with a small highly-constrained setuid executable behind it
  • Individual user environments are created using virtual hosting

MMJFS: Master Managed Job FactoryService

MJS

MJS

User 1

MJS

Master User

MJS: Managed JobService

MMJFS

MJS

User 2

MJS

User 3

User Hosting

Environment

MJS

gram job submission scenario
GRAM Job Submission Scenario

Index

Service

1. From an

index service,

the client

chooses

an MMJFS

Client

gram job submission scenario1
GRAM Job Submission Scenario

Index

Service

MMJFS

2. The client calls the

createService

operation on the factory

and supplies RSL

1. From an

index service,

the client

chooses

an MMJFS

Client

gram job submission scenario2
GRAM Job Submission Scenario

Index

Service

MMJFS

2. The client calls the

createService

operation on the factory

and supplies RSL

1. From an

index service,

the client

chooses

an MMJFS

3. The factory

creates a

Managed Job

Service

Client

MJS

gram job submission scenario3
GRAM Job Submission Scenario

Index

Service

MMJFS

2. The client calls the

createService

operation on the factory

and supplies RSL

1. From an

index service,

the client

chooses

an MMJFS

3. The factory

creates a

Managed Job

Service

4. The factory

returns a locator

Client

MJS

gram job submission scenario4
GRAM Job Submission Scenario

Index

Service

MMJFS

2. The client calls the

createService

operation on the factory

and supplies RSL

1. From an

index service,

the client

chooses

an MMJFS

3. The factory

creates a

Managed Job

Service

4. The factory

returns a locator

Client

MJS

5. The client subscribes tothe MJS’ status SDE and

retrieves output

information services
Information Services
  • Index service as caching aggregator
    • Caches service data from other Grid services
  • Index service as provider framework
    • Serves as a host for service data providers that live outside of a Grid service to publish data
reliable file transfer
Reliable File Transfer
  • OGSI-compliant service exposing GridFTP control channel functionality
    • 3rd-party transfer between GridFTP servers
  • Recoverable Grid service
    • Automatically restarts interrupted transfers from the last checkpoint
  • Progress and restart monitoring

GridFTP

Server 1

RFT

GridFTP

Server 2

JDBC

reliable file transfer service

Client

Client

Client

Request and manage file transfer operations

Notf’n

Source

Policy

Grid

Service

Fault

Monitor

Pending

interfaces

Query &/or

subscribe

to service data

Performance

service

data

elements

Policy

Perf.

Monitor

Faults

Reliable File Transfer Service

File

Transfer

Internal

State

Data transfer operations

slide96

GT-OGSA Grid Service Infrastructure

Grid Service Container

User-Defined Services

Base Services

System-Level Services

OGSI Spec Implementation

Security Infrastructure

Web Service Engine

Hosting Environment

gt3 user defined services
GT3 User-Defined Services
  • GT3 can be viewed as a Grid service development kit that includes:
    • Primitives, APIs, tools and runtime services designed to ease the task of building OGSI-compliant services
    • Primitives for provisioning security
    • Base services that provide an infrastructure with which to build higher-level services
gt3 user defined services cont
GT3 User-Defined Services (cont.)

GT3 build files

User source files

Grid service

executable files

ANT

(Diagram inspired by Borja Sotomayor’sexcellent GT3 tutorial)

User build file

overview3
Overview
  • Grid background
  • Open Grid Services Architecture
  • Open Grid Services Infrastructure
  • Beyond OGSI: other OGSA services
  • Globus Toolkit v3 implementation
  • Early GT3 performance results
  • Scientific and commercial perspectives
  • Summary
early gt3 performance analysis
Early GT3 Performance Analysis
  • Goal: explore GT3 under heavy load
    • Maximal throughput/rate of GT3 services
    • Identify limiting factors
  • Three GT3 Grid services measured
    • GRAM
    • DummyService
    • IndexService
  • Note: Preliminary results, performance continues to improve rapidly
  • Work performed by Massimo Lamanna, David Foster, et al., at CERN
gt3 gram performance setup
GT3 GRAM Performance: Setup
  • GRAM in GT3 standalone container
  • Managed-job-globusrun clients started simultaneously on up to 32 client nodes (lxplus) in non-batch mode used to submit jobs to GT3 GRAM
  • GRAM hardware: 2 * Intel Pentium III 600MHz processors, 256MB RAM
  • Note: 1 managed-job-globusrun client is able to submit 1 job
gt3 gram performance
GT3 GRAM performance
  • Results: service node
    • Saturation throughput for job submission on the service node: 3.8 jobs/minute with an average CPU user+system usage of 62%
  • Comments:
    • Scalability issue for heavily used servers
gt3 gram performance1
GT3 GRAM performance
  • Results: client node
    • Using a 2 * Intel Pentium III 600MHz processors, 256MB RAM client node, a managed-job-globusrun client consumes at average 16 seconds CPU user+system time (on both CPU’s) for 1 job
  • Comment:
    • Lightweight clients (e.g., written in "C") needed
dummyservice performance
DummyService performance
  • Setup (1)
    • Each DummyService client executes these steps:
      • Calls DummyServiceFactory to create a DummyService instance
      • Executes 2 simple methods (echo and getTime) on the DummyService instance
      • Calls DummyService instance to destroy itself
    • Up to 1000 clients talking to the DummyService from up to 45 client nodes (lxplus)
    • With & without GSI message level authentication
    • Grid service node hardware: 2 * Intel Pentium III 600MHz processors, 256MB RAM
  • Setup (2)
    • Same, but repeats step #2 one hundred times
dummyservice performance1
DummyService performance
  • Preliminary Results
  • Security overhead needs further investigation
  • Cross check our implementation/setup with Globus team foreseen
dummyservice performance2
DummyService Performance
  • Conclusions
    • Security overhead needs further investigation
    • More tests on more powerful machines
    • Container: depending on the setup the Tomcat container may be a bit slower or up to 50% faster, compared to the standalone container
  • Notes:
    • Results table shows top saturation rates
    • With varying number of clients throughput goes down by up to 30% and the average CPU u+s usage varies accordingly
slide108
The first time the client contacts the DummyServiceFactory, creates a DummyService instance, and calls the first method, it takes about 10s to accomplish it
  • These actions take about 1s on subsequent occasions
indexservice performance
IndexService performance
  • Setup (1): IndexService acting as a notification source (pushing data)
    • Multiple notification sinks subscribe to the IndexService "Host" Service Data Element (SDE), and are notified about each update of "Host" SDE, happening at a fixed rate
    • No security
    • Grid service node hardware: 2 * Intel Pentium III 600MHz processors, 256MB RAM
  • Setup (2): IndexService responding to findServiceData requests (pulling data)
    • Multiple ogsi-find-service-data clients are run sequentially and in parallel asking for IndexService "Host" Service Data Element
    • No security
    • Grid service node hardware: 2 * Intel Pentium III 600MHz processors, 256MB RAM
indexservice performance2
IndexService Performance
  • Saturation throughput with findServiceData is ~13-20 times higher than with notifications
  • Setup (1) measurement using Tomcat failed due to a thread bug, is fixed, fix announced to appear in next (4.1.28) Tomcat version
  • Preliminary measurement with a faster service node (2 * Intel(R) Xeon(TM) 2.40GHz processors, 1GB RAM):
    • Saturation throughput for setup (1) was about 32 notifications/s for 800 listeners compared to 10 notifications/s with 400 listeners on the 2 * 600MHz machines – not quite 4 times faster
overview4
Overview
  • Grid background
  • Open Grid Services Architecture
  • Open Grid Services Infrastructure
  • Beyond OGSI: other OGSA services
  • Globus Toolkit v3 implementation
  • Early GT3 performance results
  • Scientific and commercial perspectives
  • Summary
ad