DISTRIBUTED SERVICES
This presentation is the property of its rightful owner.
Sponsored Links
1 / 45

Iosif Legrand California Institute of Technology PowerPoint PPT Presentation


  • 75 Views
  • Uploaded on
  • Presentation posted in: General

DISTRIBUTED SERVICES. Iosif Legrand California Institute of Technology. Distributed Dynamic Services Architecture. Hierarchical structure of loosely coupled services which are independent & autonomous entities able to cooperate using a dynamic set of proxies or self describing protocols.

Download Presentation

Iosif Legrand California Institute of Technology

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Iosif legrand california institute of technology

DISTRIBUTED SERVICES

Iosif Legrand

California Institute of Technology

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

Distributed Dynamic Services Architecture

  • Hierarchical structure of loosely coupled services which are independent & autonomous entities able to cooperate using a dynamic set of proxies or self describing protocols.

  • They need a dynamic registration and discovery & subscription mechanism

  • For an effective use of distributed resources, these services should provide adaptability and self-organization (aggregation and hierarchical orchestration)

  • Reliable on a large scale network distributed environment

    • Avoid single points of failure

    • Automatic re-activation of components and services

  • Scalable & Flexible for adding dynamically new services and automatically replicate existing ones to cope with time dependent load

LISHEP 2004 Iosif Legrand


Distributed object systems corba rmi dcom

Lookup

Stub

Service

Skeleton

Lookup

Service

Distributed Object Systems CORBA , RMI, DCOM

“Traditional” Distributed Object Models

(CORBA, DCOM)

The Stub is linked to the Client.

The Client must know about the

service from the beginning and needs the right stub for it

Server

CLIENT

“IDL” Compiler

The Server and the client code must be created together !!

LISHEP 2004 Iosif Legrand


Web services wsdl soap

Interface

Lookup

Service

WSDL

Lookup

Service

Web Services WSDL/SOAP

SOAP

Server

CLIENT

The client can dynamically generate the data structures and

the interfaces for using remote objects based on WSDL

Platform independent

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

Lookup

Proxy

Proxy

Lookup

Service

Service

Any well suited protocol for the application

Service

CLIENT

Dynamic Code Loading

Services can be used dynamically

  • Remote Services Proxy == RMI Stub

  • Mobile Agents Proxy == Entire Service

  • “Smart Proxies” Proxy adjusts to the client

Distributed Services

Act as a true dynamic service and provide the necessary functionally to be used by any other services that require such information (Jini, interface to WSDL / SOAP)

  • mechanism to dynamically discover all the “Service Units"

  • remote event notification for changes in the any system

  • lease mechanism for each registered unit

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

CLIENT

Lookup

Lookup

Service

Service

Lookup

Lookup

Service

Service

Register

Service ID

Register with ID

Service

Ask for a lease

Get a lease for DT

JINI – Network Services

A Service Registers with at least one Lookup Service using the same ID.

It provides information about its functionality and the URL addressed from where interested clients may get the dynamic code to use it.

The Service must ask each Lookup Service for a lease and periodically renew it.

If a Service fails to renew the lease, it is removed form the Lookup Service Directory. When problems are solved, it can re-register.

jar

jar

Web

Web

Server

Server

Publish the

Publish the

“Interface” jar

“Interface” jar

The lease mechanism allows the Lookup Service to keep an up to date directory of services and correctly handle network problems.

LISHEP 2004 Iosif Legrand


Mobile agents

Mobile Agents

  • Mobile Agents are automatons entities able to migrate between AgentStations.

  • Each AgentStation offers the runtime environment, priority and controls the access permission for the hosted agents.

  • Mobile Agents interact with the AgentStations network to get access to the necessary information to migrate and to perform a certain task.

  • The agent migration is done as a weak migration ( e.g. the agent prepare itself before it is moved and save its state into structured which a serialized)

  • Mobile Agents should remain small and simple entities. To achieve this, and provide complex functionality they must use and access “static” services hosted by the AgentStation network.

LISHEP 2004 Iosif Legrand


The framework for mobile agents

The Framework for Mobile Agents

LISHEP 2004 Iosif Legrand


The components of the agent station

AgentStation provides the platform on which agent services can be hosted and allowed to execute. Agents are authenticated by the station using the security mechanism of the framework before they are allowed to execute. Once an agent has been authenticated, it is allocated a new thread for execution.

The Components of the Agent Station

LISHEP 2004 Iosif Legrand


Effective file replication

Effective File Replication

  • Identify the efficient way to replicate a data base file to all the interested Regional Centers. Find the right pass and order based on currently available bandwidth and additional constrains of the producer.

  • Use Remote Events for Global Notification

  • Allow interested parties to “cooperate” and improve the resource utilization and the global performance and the response time.

  • e.g. Avoid sending the same file twice over the same transatlantic internet connection if two Regional Centers in U.S. want the same file.

  • Based on currently available resources to find the optimal way to distribute the file from a global perspective.

LISHEP 2004 Iosif Legrand


Effective file replication using mobile agents

Java Space

Tier1

B

Remote Event

Notification

Tier2

A

Tier0

Tier1

A

Tier2

B

Effective File Replication using Mobile Agents

  • Example:

  • Remote Notification for creating a new file

  • Interested Centers deploy an agent in a time window.

  • Agents can get from the home Station Sever connectivity parameters

  • The agent objective is to bring the file home cooperating with other agents

  • Interested parties may join later and get the information from a JavaSpace

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

LISHEP 2004 Iosif Legrand


Monitoring data collection

Monitoring: Data Collection

Dynamic

Thread Pool

Other tools

(Ganglia, MRT…)

PULL

SNMP get & walk

rsh | ssh remote scripts

End-To-End measurements

Farm

Monitor

  • Configuration Control

Trap

Listener

PUSH

snmp trap

WEB Server

Dynamic loading of

modules or agents

Trap Agent

(ucd – snmp) perl

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

The Muti-Threaded Execution Architecture

  • Each request is done in an independent thread

  • A slow agent / busy node does not perturb the measurements of an entire system

  • Ex: Monitor 300 nodes @ 30 seconds interval 10-15 Threads are running in parallel

LISHEP 2004 Iosif Legrand


Data handling

Data Handling

Data Model

  • Configuration Farm , Function (Cluster), Node, Module

  • Monitored Values{Parameter, Value}

  • (Automatic) Mapping of the Data Model in :

    XML, SQL, SOAP, …

  • Configuration & Results objects are store in a DB

    (dynamically configurable for InstanDB, Postgres, MySQl, Oracle … )

  • Subscription to results objects matching a template / predicate

  • Clients can load filter objects into the Data Cache service and generate any derived (or aggregate) data structures and register to receive them.

  • Monitored parameters have a life time

TIME

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

MonALISA

Service

MonALISA

Service

MonALISA

Service

Registration / Discovery / Remote Notification

Registration

Discovery

Client

(other service)

Lookup

Service

Services

Proxy

Multiplexer

Data

Filters & Agents

Services

Proxy

Multiplexer

Lookup

Service

Client

(other service)

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

Service Monitor UNIT & Data Handling

Lookup

Service

Monitor

Data Stores

Lookup

Service

Client

(other service)

Web client

WEB

Service

WSDL

SOAP

Farm

Monitor

Data Cache

Service & DB

Discovery

Registration

Client

(other service)

Java

data

McKoi DB

MySQL

Predicates & Agents

Other tools

  • Configuration Control (SSL)

UDP

MySQL

  • Predicates & Agents

User defined loadable

Modules to write /sent data

MDS

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

Key store

Key store

Key

Secure – Automatic Update Mechanismfor Services and Clients

Lookup

Service

MonaLisa

Service

Web Server

Sign Distribution

Discovery

  • download

  • Update Signal SSL

  • download

Admin Client

  • Update Signal SSL

MonaLisa

Service

  • All running services are update using the discovery mechanism

  • At startup each service check if it an update is done at a set of Web Servers

  • Clients use the Web Start mechanism

Lookup

Service

LISHEP 2004 Iosif Legrand


Global client for farms and network connectivity

@ CALTECH

Global Client for Farms and Network Connectivity

DataTAG

-

-

LISHEP 2004 Iosif Legrand


Real time data for large systems lxshare cluster at cern 600 ndoes

Real-time Data for Large Systems“lxshare” cluster at cern ~ 600 ndoes

LISHEP 2004 Iosif Legrand


Access to historical and real time values

Access to historical and real-time values

Past values are presented and the GUI

remains a registered listener and

the new vales are added

Real Time Histograms for various parameters

LISHEP 2004 Iosif Legrand


Mobile agents and filters

Mobile Agents and Filters

Simple “Global Load” filter agent

From FNAL to all

Maximum Flow Data Replication Path Agent Deployed to each RC and evaluates the best path for real-time data replication

From CERN to all

LISHEP 2004 Iosif Legrand


Pseudo clients dedicated repositories

Lookup

Service

MonaLisa

Service

Discovery

WAP

  • Filter Agents / Data

TOMCAT

JSP/servelts

Pseudo Client

WEB

MySQL

IDB

MySQL

IDB

MySQL

  • Filter Agents / Data

MonaLisa

Service

Lookup

Service

Pseudo – Clients & Dedicated Repositories

LISHEP 2004 Iosif Legrand


Monalisa repositories

MonALISA repositories

LISHEP 2004 Iosif Legrand


Pda 3 d monalisa client

PDA 3 D MonALISA Client

Developed by Research Lab

LISHEP 2004 Iosif Legrand


Vrvs architecture

VRVS Architecture

pub

cal-

tech

cor-

nell

funet

Reflectors are hosts that interconnect users by permanent IP tunnels.

vrvs

5

star-

light

vrvs

us

vrvs

eu

The active IP tunnels must be selected so that there is no cycle formed.

usf

sinica

inet

2

Tree

usp

The selection is made according to the assumed network links performance.

kek

triumf

LISHEP 2004 Iosif Legrand


Minimum spanning tree algorithms

Minimum Spanning Tree Algorithms

Finding a tree T that contains all the vertices

of a graph G spanning tree

and has the least total weight over all

such trees minimum-spanning tree(MST)

Input: A weighted connected graph G = (V,E) with n vertices

and m edges

Output: A minimum- spanning tree T

LISHEP 2004 Iosif Legrand


Kruskal s algorithm

Kruskal‘s Algorithm

  • Each vertex is in its own cluster

    2. Take the edge e with the smallest weight

    - if e connects two vertices in different clusters,

    then e is added to the MST and the two clusters,

    which are connected by e, are merged into a single cluster

    - if e connects two vertices, which are already in the same

    cluster, ignore it

    3. Continue until n-1 edges were selected

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

Kruskal‘s Algorithm

5

A

B

4

6

2

2

D

C

3

1

2

3

E

F

4

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

Kruskal‘s Algorithm

5

A

B

4

6

2

2

D

C

3

1

2

3

E

F

4

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

Kruskal‘s Algorithm

5

A

B

4

6

2

2

D

C

3

1

2

3

E

F

4

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

Kruskal‘s Algorithm

5

A

B

4

6

2

2

D

C

3

1

2

3

E

F

4

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

Kruskal‘s Algorithm

5

A

B

4

6

2

2

D

C

3

1

2

3

E

F

4

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

Kruskal‘s Algorithm

5

A

B

4

6

2

2

D

cycle!!

C

3

1

2

3

E

F

4

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

Kruskal‘s Algorithm

5

A

B

4

6

2

2

D

C

3

1

2

3

E

F

4

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

Kruskal‘s Algorithm

5

A

B

4

6

2

2

D

C

3

1

2

3

E

F

4

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

Kruskal‘s Algorithm

minimum- spanning tree

A

B

2

2

D

C

1

2

3

E

F

LISHEP 2004 Iosif Legrand


Bar vka s algorithm

Barůvka‘s Algorithm

  • For all vertices search the edge with the smallest weight

    of this vertex and mark these edges

  • Search connected vertices (clusters) and replace them by

    a “new“ vertex (cluster)

  • Remove the cycles and, if two vertices are connected by

    more than one edge, delete all edges except the “cheapest“

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

Barůvka‘s Algorithm

5

A

B

4

6

2

2

D

C

3

1

2

3

E

F

4

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

Barůvka‘s Algorithm

5

A

B

4

6

2

2

D

C

3

1

2

3

E

F

4

LISHEP 2004 Iosif Legrand


Iosif legrand california institute of technology

Barůvka‘s Algorithm

5

A

B

4

6

2

2

D

C

3

1

2

3

E

F

4

LISHEP 2004 Iosif Legrand


Monitoring vrvs reflectors

Monitoring VRVS Reflectors

LISHEP 2004 Iosif Legrand


Monitoring vrvs reflectors1

Monitoring VRVS Reflectors

LISHEP 2004 Iosif Legrand


Global client dynamic discovery

Global Client / Dynamic Discovery

LISHEP 2004 Iosif Legrand


Summary

SUMMARY

  • MonALISA is able to dynamically discover all the “Service Units" used by a community and through the remote event notification mechanism keeps an update state for the entire system

  • Automatic & secure code update (services and clients) .

  • Dynamic configuration for services. Secure Admin interface.

  • Access to aggregate farm values and all the details for each node

  • Selected real time / historical data for any subscribed listeners

  • Active filter agents to process the data and provided dedicated / customized information to other services or clients.

  • Mobile Agents for decision support and global optimization.

  • Dynamic proxies and WSDL & WAP pages for services.

  • Embedded SNMP support and interfaces with other tools ( LSF, PBS, Ganglia, Hawkeye, IEPM-BW…)

  • Dedicate pseudo-clients for repository, WAP access or decision making units

  • It proved to be a stable and reliable distributed service system. It is currently running at ~ 100 sites

  • http://monalisa.cacr.caltech.edu

LISHEP 2004 Iosif Legrand


  • Login