The legion project
Download
1 / 70

The Legion Project - PowerPoint PPT Presentation


  • 138 Views
  • Uploaded on

The Legion Project. Adam Belloum Computer Architecture & Parallel Systems group University of Amsterdam [email protected] Legion requirements. Security Global name space Fault tolerance Accommodating heterogeneity Binary management and application provisioning Multilanguage support

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' The Legion Project' - kamala


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
The legion project

The Legion Project

Adam Belloum

Computer Architecture & Parallel Systems group

University of Amsterdam

[email protected]


Legion requirements
Legion requirements

  • Security

  • Global name space

  • Fault tolerance

  • Accommodating heterogeneity

  • Binary management and application provisioning

  • Multilanguage support

  • Scalability

  • Persistence

  • Extensibility

  • Site autonomy

  • Complexity management


Legion design principles
Legion design principles

  • Provide a single-system view

  • Provide transparency as a means of hiding detail

  • Provide flexible semantics

  • Reduce user effort

  • Reduce “activation energy”

  • Do no harm

  • Do not change host operating systems


Legion architectural details
Legion architectural details

  • Legion started out with the top-down premise

     “ that a strong architecture is necessary to build an infrastructure of this scope.”

  • This architecture is based on communicating services, which are implemented as objects.

  • An object is a stateful component in a container process.


Legion is a system
Legion is a System

  • Object based meta-system, providing a single integrated infrastructure

  • All components are objects (unlike GT)

    • Data abstraction, encapsulation, inheritance, polymorphism

  • API to core services

  • Core object types

    • Classes/metaclasses: managers and policy makers

    • Host objects: abstractions of processing resources (one or many)

    • Vault objects: persistent storage

    • Implementation objects and caches: “exectuables”

    • Binding agents: maps objects to physical addresses

    • Context objects: naming of objects


Tasks involved in the design
Tasks involved in the design

  • 3-level naming, binding, and communication scheme.

  • Remote invocation based on dataflow graphs.

  • Security layer to every instance of communication between any services.

  • Fault-tolerance layers.

  • Layers for resource discovery through

    • naming and mechanisms for caching name bindings


  • Much of the later development in Legion has been in terms of adding new kinds of services

    • two dimensional files,

    • specialized schedulers,

    • firewall proxies,

    • file export services,

    • etc.

    • or tools that invoke methods of services (running an application, running parameter space applications, using a Web portal [41], etc.).


Architecture
Architecture adding new kinds of services

  • Object-based Integrated System

  • What Legion does not do:

    • require that Legion run as “root”

    • replace host operating systems

    • legislate changes to Internet


Legion objects
Legion Objects adding new kinds of services

  • Address-space disjoint

  • Member functions

  • Non-blocking calls

  • Arbitrary metadata

  • Active or Inert


Some object types
Some Object types adding new kinds of services

  • Host

    • Abstraction of a processing Unit

  • Vault

    • Abstraction of a persistent storage

  • Class Manager

    • Responsible for object creation, Responsible for object creation, monitoring, and lifetime management monitoring, and lifetime management


Legion object identifier loid
Legion Object Identifier (LOID) adding new kinds of services

  • Context Name Context Name

    • Human Human-readable readable

  • Legion Object ID (LOID)

    • Location Location-independent independent

  • Object Address Object Address

    • Machine Machine-and and-communication communication-specific


Legion object identifier loid1
Legion Object Identifier (LOID) adding new kinds of services

  • Object name

  • Location-independent

  • String of bits

  • Fields

    • Type

    • Domain

    • Class ID

    • Instance ID

    • Public Key


What s a name
What’s a name adding new kinds of services

date

  • LOIDs are necessary

    • immutable identifier of an object

  • Context names are convenient

    • human-readable

  • OAs are actual

    • machine name, port number

LOID


Object naming
Object Naming adding new kinds of services

  • Human-readable

    • translation

  • Legion Object ID

    • Biding

  • Location-specific

  • Communication-Specific

  • Context Name

    Legion Object ID

    Obbject Address


    Object address
    Object Address adding new kinds of services

    • Incorporates physical addresses (IP, XTP)

    • Contains multiple addresses for various forms of multicast and group communication

    • Can find “best” address dynamically

    • Bindings are <LOID, OA, timeout> tuples


    Unified console
    Unified Console adding new kinds of services

    TTY

    File

    Program produces

    stdout, stderr

    User creates

    tty object

    Prog.

    User shares

    tty LOID

    User starts

    running program

    Legion passes tty

    LOID to program

    User shares

    tty LOID


    Tty object
    TTY Object adding new kinds of services

    • Redirect run-time output to central (or multiple) consoles

    • Connect and disconnect dynamically

    • Debug quickly and simply

    • Monitor status, errors, easily

    • Share console with others

      legion_tty <ttyobj>


    Context space
    Context Space adding new kinds of services

    /

    • Unix-like

      legion_ls

      legion_pwd

      legion_cd

      legion_cat

      ...

    hosts

    home

    users

    mach1

    mach2

    mydir

    you

    me

    subdir

    prog

    file1

    tty


    The legion architectural view
    The Legion Architectural view adding new kinds of services


    Legions two lowers layers
    Legions Two lowers layers adding new kinds of services

    • consist of

      • location-independent abstract names called Legion object identifiers (LOIDs)

      • and object addresses specific to communication protocols,e.g., an IP address and a port number.

    • The binding between a LOID and an object address can and does change over time.


    Security layer
    Security Layer adding new kinds of services

    • The security layer implements the Legion security model for authentication, access control, and data integrity (e.g., mutual authentication and encryption on the wire).


    The core object layer
    The core object layer adding new kinds of services

    • The core object layer addresses:

      • method invocation,

      • event processing (including exception and error propagation on a per-object basis ),

      • interface discovery,

      • and the management of metadata.

    • Objects can have arbitrary metadata, such as the load on a host object or the parameters that were used to generate a particular data file.


    The core services
    The core services adding new kinds of services

    • Implement object instance

      • management (class managers)

      • abstract processing resources (hosts),

      • and storage resources (vaults).

    • Core services are represented by base classes

      • that can be extended to provide different or enhanced implementations.


    Higher level system service
    Higher-Level System Service adding new kinds of services

    • Collection of higher-level system service types and enhancements to the base service classes.

    • These include classes for object

      • replication for availability

      • Message logging for accounting or postmortem debugging

      • firewall proxy servers for securely transiting firewalls,

      • enhanced schedulers

      • databases,

      • job proxy managers that “wrap” legacy codes for remote execution.


    Application support layer
    Application support layer adding new kinds of services

    • Application support layer

      • contains user centered tools for parallel and high-throughput computing, data access and sharing, and system management.

    • High-performance tool set for wrapping legacy codes

      • legion_register_program

      • legion_run

      • Legion MPI tools support cross-platform, cross-site

      • Fortran support tools (Fortran programs for running on a grid).


    Legion basic file type
    Legion basic file type adding new kinds of services

    • a basic file type (BasicFile) supports the usual functions: read, write, stat, seek, etc.

    • All other file types are derived from this type.

      • for example, piped into tools that expect sequential files.


    Legion basic file type1
    Legion basic file type adding new kinds of services

    • There are two-dimensional files that support

      • read/write operations on columns, rows, and rectangular patches of data (both primitive types as well as “structs”).

    • There are file types to support

      • unstructured sparse data,

      • parallel files in which the file has been broken up and decomposed across several storage systems.


    Data movement on the grid
    Data movement on the Grid adding new kinds of services

    • Data can be copied into the grid

      • in which case Legion manages the data

      • decides where to place it,

      • how many copies to generate for higher availability,

      • where to place those copies.

    • Data can be exported into the grid.

      • When a local directory structure is exported into the grid, it is mapped to a chosen path name in the global name space (directory structure).


    Data movement on the grid1
    Data movement on the Grid adding new kinds of services

    • For example, a user can map

      • data/sequences in his local file system into /home/grimshaw/sequences using the legion_export_dir command,

      • legion_export_dir data/sequences /home/grimshaw/sequences.

      • Subsequent access from anywhere in the grid (read/write) is done directly against the files in the users file system (subject to access control).


    • data grid can be accessed via a daemon that implements the Network File System (NFS) protocol.

    • The entire Legion namespace, including files, hosts, and so forth, can be mapped into local OS file systems.

    • Shell scripts, Perl scripts, an user applications can run unmodified on the Legion data grid.

    • The usual UNIX commands work, as does Microsoft Windows Exploring


    Security in legion

    Security in Legion Network File System (NFS) protocol.

    Overview of Security Standards in the Grid, CSE 225 High Performance and Computational Grids Spring 2000 [email protected]


    Security in legion 1 design principals
    Security in Legion (1) Network File System (NFS) protocol. Design Principals

    • As in the Hippocratic Oath, do no harm!

    • Caveat emptor - let the buyer beware.

    • Small is beautiful.


    Security in legion standards
    Security in Legion Network File System (NFS) protocol. Standards

    • X.509 ?

    • Keberos ?


    Security in legion legion security model
    Security in Legion Network File System (NFS) protocol. Legion Security Model

    • responsible for securing the communications between Legion objects

    • responsible for access control

      • determines what objects/ users are allowed to call a particular object’s methods.

      • relies on the message layer for some services.


    Security in legion 2 basic concepts
    Security in Legion (2) Network File System (NFS) protocol. Basic Concepts

    • Every object provides certain known member functions

      • MayI, CanI, Iam, and Delegate.

    • Two objects associated with each operation:

      • a responsible agent (RA)

      • and a calling agent (CA)

    • Every invocation of member function is performed in the context of

      • a certificate which contains the LOID.

      • Certificate digitally signed by maker


    Security in legion1
    Security in Legion Network File System (NFS) protocol.

    • Object A invokes method B.foo

    • call is passed to B, and the Legion run-time system automatically invokes B.MayI

      • If B.MayI is true, then foo is invoked with the arguments

      • If not, then an exception is raised and passed back the information necessary to make such a decision


    Security in legion2
    Security in Legion Network File System (NFS) protocol.

    • Legion users are responsible for own security.

    • Object might trust that the CA is correct.

    • Policies defined by objects themselves.

    • Every class defines a special member function, MayI.

    • MayI defines the security objects for a class.

    • Every member function invocation permitted only if MayI sanctions it.


    Security in legion automatic invocation of outgoing calls
    Security in Legion Network File System (NFS) protocol. Automatic invocation of outgoing calls

    • Object A invokes B.foo,

    • the compiler has interposed code so that A.CanI is automatically invoked before the call leaves A.

      • If CanI returns true, then the method call proceeds.

      • If CanI returns false, an exception is raised.


    Security in legion3
    Security in Legion Network File System (NFS) protocol.

    • Authentication aided by use of Legion certificates

      • - based on public-key cryptography by default. Must know private key to authenticate.

    • MayI functions can code their own authentication protocols

    • Every Legion object required to supply special member function Iam for authentication purposes.


    Security in legion4
    Security in Legion Network File System (NFS) protocol.

    • Login establishes user identity and creates responsibility agent for user.

    • Login is building block for authentication and delegation.

    • Object can delegate new certificate to delegate rights.

    • Delegation policy defined by object.


    Security in legion future work
    Security in Legion Network File System (NFS) protocol. Future Work

    • Legion does not specify any particular encryption. Future standardization?

    • Legion eschews distinguished trusted objects - centralized key management server

    • Composition of a security policy


    Web based portal interface

    Web-based portal interface Network File System (NFS) protocol.


    User portal and system management tools
    User portal and system management tools Network File System (NFS) protocol.

    • the user portal and system management tools

      • add and remove users,

      • add and remove hosts,

      • and join two separate Legion grids together.

    • Web-based portal interface

      • for access to Legion

      • a system status display tool that gathers information from system-wide metadata collections and makes it available via a browser


    Web based portal
    Web-based portal Network File System (NFS) protocol.

    • Web-based portal allows an alternative, graphical interface to Legion.

    • Using this interface a user can submit

      • An Amber job (a 3D molecular modeling code) to NPACI-net (a University of Virginia Legion network)

      • and not care at all where it executes.


    Portal view
    Portal view Network File System (NFS) protocol.

    • user can copy files out of the running simulation

    • and in which a three dimensional molecular visualization is being used to display the intermediate results.


    Legion job status tools
    Legion job status tools Network File System (NFS) protocol.

    • Using these tools users can determine the status

      • of all of the jobs they have started from the Legion portal

      • and access their results as needed.


    Legion accounting system
    Legion accounting system Network File System (NFS) protocol.

    • The portal interface to the underlying Legion accounting system.

    • Legion keeps track of who used resource (CPU, application, etc.),

      • with exit status

      • with how much resource consumption.

    • Data is loaded into a relational database, and various reports can be generated


    • Originally a research project at U. Virginia Network File System (NFS) protocol.

      • commercialised via Applied Meta (now Avaki Corp)

    • Some successful applications

      • used to perform a metacomputing run of an ocean model

      • ran a task-farm of computational chemistry calculations

    • Grid Object Model not really found favour

      • in development phase, Grid must run existing applications

      • hard to do this within an integrated environment

      • major applications are not themselves object oriented

      • however cf. OGSA

    Rob Baxter

    A Complete History of the Grid (abridged)


    Context space1

    Context space Network File System (NFS) protocol.


    Context space2
    Context Space Network File System (NFS) protocol.

    /

    • Unix-like

      legion_ls

      legion_pwd

      legion_cd

      legion_cat

      ...

    hosts

    home

    users

    mach1

    mach2

    mydir

    you

    me

    subdir

    prog

    file1

    tty


    Context space3
    Context Space Network File System (NFS) protocol.

    • Network-wide, transparent file system

    • Location-independent read/write of files

    • Convenient transfer of files between context space and local file system

    • I/O libraries for access

    • Unix-like utilities


    Context example
    Context Example Network File System (NFS) protocol.

    legion_ls /


    Another context
    Another Context Network File System (NFS) protocol.

    legion_ls /hosts


    Yet another context
    Yet Another Context Network File System (NFS) protocol.

    legion_ls /users


    More context fun
    More Context Fun Network File System (NFS) protocol.


    Other context commands
    Other Context Commands Network File System (NFS) protocol.

    • Locate a LOID in context space

      legion_list_names

    • Locate an object on a machine

      legion_whereis

    • Find status of an object

      legion_object_info

    • List metadata of an object

      legion_list_attributes


    Status of an object
    Status Of An Object Network File System (NFS) protocol.

    legion_object_info -c work


    Physical location of object
    Physical Location Of Object Network File System (NFS) protocol.

    legion_whereis -c work


    Context space vs local space
    Context Space vs. Local Space Network File System (NFS) protocol.

    • Local space = your machine’s directory structure

      • OS-specific, Machine-specific

      • Use cp, copy, etc.

      • e.g., C:\Program Files\, /usr/bin, /mnt/disk1

    • Context space = Legion’s directory structure

      • OS-independent, Machine-independent

      • Uselegion_cp, etc.


    Context space and local space
    Context Space and Local Space Network File System (NFS) protocol.

    • Transfer one file from local space to context space

      legion_cp -localsrc <localfile> <contextfile>

    • Transfer one file from context space to local space

      legion_cp -localdest <contextfile> <localfile>


    Context space and local space1
    Context Space and Local Space Network File System (NFS) protocol.

    • Copying local directory to context space

      legion_cp -r -localsrc <localdir> <contextdir>

      OR

      legion_import_tree <localdir> <contextdir>

    • Copying context directory to local space

      legion_cp -r -localdest <contextdir> <localdir>


    Context space and local space2
    Context Space and Local Space Network File System (NFS) protocol.

    • Map (not copy!) local directory to context space temporarily

      legion_export_dir <localdir> <contextdir>

    • Does NOT make copy of local directory

    • Merely provides Legion-like access to local directory

      • Use legion_cat on local files


    Making context space
    Making Context Space… Network File System (NFS) protocol.

    • Local sub-directory with Legion NFS daemon

      • Use cat on context files

    • FTP directory with FTP interface

    • Windows directory with Samba interface

    • URL tree with HTTP interface


    I o performance
    I/O Performance Network File System (NFS) protocol.

    • X-Axis = number of clients simultaneously performing 1MB reads on 10MB files

    • Y-Axis = total read bandwidth

    • Each point = average of multiple runs

    • Clients = 400MHz Intels, NFS Server = 800MHz Intel


    Making context space1
    Making Context Space… Network File System (NFS) protocol.

    • Local sub-directory with Legion NFS daemon

      • Use cat on context files

    • FTP directory with FTP interface

    • Windows directory with Samba interface

    • URL tree with HTTP interface


    Flexible context space
    Flexible Context Space Network File System (NFS) protocol.

    Disk

    e

    ftp

    Directory

    NFS

    HTTP

    Samba

    FTP

    Context

    Context

    Context

    legion_export_dir

    legion_import_tree

    Disk

    Directory

    Context

    Directory

    Directory


    Access control
    Access Control Network File System (NFS) protocol.

    • MayI for each object implements access control on a per-function basis

    • Users named by login object

    • Sets of users grouped by contexts

      legion_change_permissions [+-rwx] [-v] <group/user context> <target context>

      legion_change_permissions +r /users/fred /home/grimshaw/myfile


    Access control example
    Access Control Example Network File System (NFS) protocol.


    Unified console1
    Unified Console Network File System (NFS) protocol.

    TTY

    File

    Program produces

    stdout, stderr

    User creates

    tty object

    Prog.

    User shares

    tty LOID

    User starts

    running program

    Legion passes tty

    LOID to program

    User shares

    tty LOID


    Tty object1
    TTY Object Network File System (NFS) protocol.

    • Redirect run-time output to central (or multiple) consoles

    • Connect and disconnect dynamically

    • Debug quickly and simply

    • Monitor status, errors, easily

    • Share console with others

      legion_tty <ttyobj>


    ad