The anatomy and physiology of the grid revisited
1 / 33

The Anatomy and Physiology of the Grid Revisited - PowerPoint PPT Presentation

  • Uploaded on

The Anatomy and Physiology of the Grid Revisited. Nenad Medvidovic USC-CSSE and Computer Science Department University of Southern California [email protected] http:// / Collaborative work with Joshua Garcia, Ivo Krka , Chris Mattmann , and Daniel Popescu. What is the grid?.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' The Anatomy and Physiology of the Grid Revisited' - nellie

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
The anatomy and physiology of the grid revisited

The Anatomy and Physiologyof the Grid Revisited

Nenad MedvidovicUSC-CSSE and Computer Science DepartmentUniversity of Southern California

[email protected]

Collaborative work with Joshua Garcia, IvoKrka, Chris Mattmann, and Daniel Popescu

What is the grid
What is the grid?

  • A distributed systems technology that enables the sharing of resources across organizations scalably, efficiently, reliably, and securely

  • Analogous to the electric grid

Why study the grid
Why Study the Grid?

  • A highly successful technology

  • Deficiencies in the existing guidance for building grids

    • More to come

  • Grids are not easy to build

    • See CERN’s Large Hadron Collider

  • Their architecture was published very early

    • “anatomy” and “physiology”

  • Yet “What is (not) a grid?” is still a subject of debate

The architectural perspective
The Architectural Perspective

  • Grids are large, complex systems

    • Thousands of nodes or more

    • Span many agency boundaries

  • Qualities of Service (QoS) are critical

    • Scalability

    • Security

    • Performance

    • Reliability ...

  • Software architecture is just what the doctor ordered

    • The set of principal design decisions about a software system [Taylor, Medvidovic, Dashofy 2009]

So what did we set out to do
So, What Did We Set out to Do?

  • Study grid’s reference requirements and architecture

  • Study the architectures of existing grid technologies

  • Compare the two

    • Knowing that there will likely be very few straightforward answers

  • Suggest how to fix any discrepancies

    • Knowing that there will likely be very few straightforward answers

Architecture recovery technique focus
Architecture Recovery Technique- Focus -

  • Establish idealized architecture and candidate architectural style(s)

  • Identify data and processing components

    • Groups implementation modules according to a set of rules

  • Map identified data and processing components onto an idealized architecture

  • Examine

    • Source code

    • Documentation

    • Runtime behavior

    • Tie to requirements satisfied by component

Rules of focus
Rules of Focus

  • Group based on isolated classes

  • Group based on generalization

  • Group based on aggregation

  • Group based on composition

  • Group based on two-way association

  • Identify domain classes

  • Merge classes with a single originating domain class association into domain class

  • Group classes along a domain class circular dependency path

  • Group classes along a path with a start node and end node thatreference a domain class

  • Group classes along paths with the same end node, and whose start node references the same domain class

Some refinements to the rules
Some Refinements to the Rules

  • Domain class rules

    • Class with large majority of outgoing calls

  • Exclusion rules

    • Class with large majority of incoming calls

    • Utility classes

    • Heavily passed data-structures

    • Benchmarking and test classes

  • Additional groupings

    • By exception

    • By interface

    • By package if idealized architecture matches first-class component

Focus rules for distributed systems
Focus Rules for Distributed Systems

  • Infer distributor connectors from idealized architecture

  • Classes with methods and names similar to first-class components are domain classes

  • Classes importing network communication libraries are domain classes

  • main() functions often identify first-class components

  • Classes deployed onto different hosts must be grouped separately

Discovered discrepancies
Discovered discrepancies

  • Empty layers

  • Skipped Layers

  • Up-calls

  • Multi-layer components

Empty layers wings
Empty Layers- Wings -

Skipped layers pegasus
Skipped Layers - Pegasus -

Upcalls hadoop
Upcalls- Hadoop-

What about globus1

Two layer boundary AND



Couldn’t determine right “layer”


Two layer boundary AND



Two layer boundary AND


What about Globus?

Revised grid architecture
Revised Grid Architecture

  • The connectivity layer is eliminated

  • Explicitly addressing deployment view

  • Subsystem types rather than layer-oriented

  • Four architectural styles comprise the grid

    • Client/server

    • Peer-to-peer

    • Layered

    • Event-based

  • An improved classification of grid technologies

Grid styles c s
Grid Styles – C/S

  • Applicationcomponents are clients to Collectivecomponents

    • e.g., application components query for resource component locations from collective components

  • Applicationcomponents are clients to Resourcecomponents

    • e.g., direct job submission from application components to resource components

  • Resourcecomponents can act as clients to Collectivecomponents

    • e.g., resource components may obtain locations of other resource components through collective components

Grid styles p2p
Grid Styles – p2p

  • Resource componentsare peers

    • e.g., Grid Datafarm Filesystem Daemon (gfsd) instance makes requests for file data from other gfsds

  • Collective components are peers

    • e.g., iRODS agents communicate with each other to exchange data to create replicas

Grid styles event based
Grid Styles – Event-Based

  • Resource components notify Collective components that monitor them

    • e.g., executors send heartbeats to managers

Grid architectural styles layered
Grid Architectural Styles – Layered

  • Collective or Resource components request services from Fabric components

    • e.g., iRODS agent accesses a DBMS with metadata

Grid technology classification
Grid Technology Classification

  • Computational grid

    • Implementing all Collective components

    • e.g., Alchemi and Sun Grid Engine

Grid technology classification1
Grid Technology Classification

  • Data grid

    • Job scheduling components in Collective subsystem are not required

    • e.g., Grid Datafarm and Hadoop

Grid technology classification2
Grid Technology Classification

  • Hybrid

    • Resource components providing services either to perform operations on a storage repository or to execute a job or task

    • e.g. Gridbus Broker and iRODS

Computational Resource

File Resource

Correcting violations in the reference architecture
Correcting Violations in the Reference Architecture

  • Why were there originally so many upcalls?

    • Legitimate client-server and event-based communication

  • Why so many skipped layer calls?

    • The Fabric layer was at the wrong level of abstraction

    • Mostly utility classes that should be abstracted away

  • Why so many multi-layer components?

    • Connectivity layer was at the wrong level of abstraction

    • Not a layer, but utility libraries to enable connector functionality

    • Also accounts for skipped layer calls

  • Benefit of the deployment view

    • Essential for distributed systems

    • Helped to identify that the Fabric layer was not abstracted properly

Where are we currently
Where Are We Currently?

  • There are remaining violations

    • Are they legitimate or a result of an improperly recast reference architecture?

  • Original Focus is not ideal for recovering systems of these types

    • Distributed systems realized by a middleware

  • A more automated approach that combines static and dynamic analysis would be preferable

  • Use the recast reference architecture to build a new grid

  • What are the overarching grid principles?

Evolving grid principles
Evolving Grid Principles

  • A grid is a collection of logical resources (computing and data) distributed across a wide-area network of physical resources (hosts).

  • In a single grid-based application, the logical resources are owned by a single agency, while the physical resources are owned by multiple agencies.

  • All resources in a grid are described using a common meta-resource language.

  • Atomic-level logical resources are defined independently of the atomic-level physical resources.

  • The allocation of the atomic-level logical resources to the atomic-level physical resources can be N:M.

  • All computation in a grid is initiated by a client, which is a physical resource. The client sends the logical resources to the servers, which are also physical resources. A server can, in turn, delegate the requested computation to other physical resources.

  • All agencies that own physical resources in a grid must be able to specify policies that enforce the manner in and extent to which their physical resources can be used in grid applications.