the legion project
Download
Skip this Video
Download Presentation
The Legion Project

Loading in 2 Seconds...

play fullscreen
1 / 70

The Legion Project - PowerPoint PPT Presentation


  • 139 Views
  • Uploaded on

The Legion Project. Adam Belloum Computer Architecture & Parallel Systems group University of Amsterdam [email protected] Legion requirements. Security Global name space Fault tolerance Accommodating heterogeneity Binary management and application provisioning Multilanguage support

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' The Legion Project' - kamala


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
the legion project

The Legion Project

Adam Belloum

Computer Architecture & Parallel Systems group

University of Amsterdam

[email protected]

legion requirements
Legion requirements
  • Security
  • Global name space
  • Fault tolerance
  • Accommodating heterogeneity
  • Binary management and application provisioning
  • Multilanguage support
  • Scalability
  • Persistence
  • Extensibility
  • Site autonomy
  • Complexity management
legion design principles
Legion design principles
  • Provide a single-system view
  • Provide transparency as a means of hiding detail
  • Provide flexible semantics
  • Reduce user effort
  • Reduce “activation energy”
  • Do no harm
  • Do not change host operating systems
legion architectural details
Legion architectural details
  • Legion started out with the top-down premise

 “ that a strong architecture is necessary to build an infrastructure of this scope.”

  • This architecture is based on communicating services, which are implemented as objects.
  • An object is a stateful component in a container process.
legion is a system
Legion is a System
  • Object based meta-system, providing a single integrated infrastructure
  • All components are objects (unlike GT)
    • Data abstraction, encapsulation, inheritance, polymorphism
  • API to core services
  • Core object types
    • Classes/metaclasses: managers and policy makers
    • Host objects: abstractions of processing resources (one or many)
    • Vault objects: persistent storage
    • Implementation objects and caches: “exectuables”
    • Binding agents: maps objects to physical addresses
    • Context objects: naming of objects
tasks involved in the design
Tasks involved in the design
  • 3-level naming, binding, and communication scheme.
  • Remote invocation based on dataflow graphs.
  • Security layer to every instance of communication between any services.
  • Fault-tolerance layers.
  • Layers for resource discovery through
    • naming and mechanisms for caching name bindings
slide7
Much of the later development in Legion has been in terms of adding new kinds of services
    • two dimensional files,
    • specialized schedulers,
    • firewall proxies,
    • file export services,
    • etc.
    • or tools that invoke methods of services (running an application, running parameter space applications, using a Web portal [41], etc.).
architecture
Architecture
  • Object-based Integrated System
  • What Legion does not do:
    • require that Legion run as “root”
    • replace host operating systems
    • legislate changes to Internet
legion objects
Legion Objects
  • Address-space disjoint
  • Member functions
  • Non-blocking calls
  • Arbitrary metadata
  • Active or Inert
some object types
Some Object types
  • Host
    • Abstraction of a processing Unit
  • Vault
    • Abstraction of a persistent storage
  • Class Manager
    • Responsible for object creation, Responsible for object creation, monitoring, and lifetime management monitoring, and lifetime management
legion object identifier loid
Legion Object Identifier (LOID)
  • Context Name Context Name
    • Human Human-readable readable
  • Legion Object ID (LOID)
    • Location Location-independent independent
  • Object Address Object Address
    • Machine Machine-and and-communication communication-specific
legion object identifier loid1
Legion Object Identifier (LOID)
  • Object name
  • Location-independent
  • String of bits
  • Fields
    • Type
    • Domain
    • Class ID
    • Instance ID
    • Public Key
what s a name
What’s a name

date

  • LOIDs are necessary
    • immutable identifier of an object
  • Context names are convenient
    • human-readable
  • OAs are actual
    • machine name, port number

LOID

object naming
Object Naming
  • Human-readable
          • translation
  • Legion Object ID
          • Biding
  • Location-specific
  • Communication-Specific

Context Name

Legion Object ID

Obbject Address

object address
Object Address
  • Incorporates physical addresses (IP, XTP)
  • Contains multiple addresses for various forms of multicast and group communication
  • Can find “best” address dynamically
  • Bindings are <LOID, OA, timeout> tuples
unified console
Unified Console

TTY

File

Program produces

stdout, stderr

User creates

tty object

Prog.

User shares

tty LOID

User starts

running program

Legion passes tty

LOID to program

User shares

tty LOID

tty object
TTY Object
  • Redirect run-time output to central (or multiple) consoles
  • Connect and disconnect dynamically
  • Debug quickly and simply
  • Monitor status, errors, easily
  • Share console with others

legion_tty <ttyobj>

context space
Context Space

/

  • Unix-like

legion_ls

legion_pwd

legion_cd

legion_cat

...

hosts

home

users

mach1

mach2

mydir

you

me

subdir

prog

file1

tty

legions two lowers layers
Legions Two lowers layers
  • consist of
    • location-independent abstract names called Legion object identifiers (LOIDs)
    • and object addresses specific to communication protocols,e.g., an IP address and a port number.
  • The binding between a LOID and an object address can and does change over time.
security layer
Security Layer
  • The security layer implements the Legion security model for authentication, access control, and data integrity (e.g., mutual authentication and encryption on the wire).
the core object layer
The core object layer
  • The core object layer addresses:
    • method invocation,
    • event processing (including exception and error propagation on a per-object basis ),
    • interface discovery,
    • and the management of metadata.
  • Objects can have arbitrary metadata, such as the load on a host object or the parameters that were used to generate a particular data file.
the core services
The core services
  • Implement object instance
    • management (class managers)
    • abstract processing resources (hosts),
    • and storage resources (vaults).
  • Core services are represented by base classes
    • that can be extended to provide different or enhanced implementations.
higher level system service
Higher-Level System Service
  • Collection of higher-level system service types and enhancements to the base service classes.
  • These include classes for object
    • replication for availability
    • Message logging for accounting or postmortem debugging
    • firewall proxy servers for securely transiting firewalls,
    • enhanced schedulers
    • databases,
    • job proxy managers that “wrap” legacy codes for remote execution.
application support layer
Application support layer
  • Application support layer
    • contains user centered tools for parallel and high-throughput computing, data access and sharing, and system management.
  • High-performance tool set for wrapping legacy codes
    • legion_register_program
    • legion_run
    • Legion MPI tools support cross-platform, cross-site
    • Fortran support tools (Fortran programs for running on a grid).
legion basic file type
Legion basic file type
  • a basic file type (BasicFile) supports the usual functions: read, write, stat, seek, etc.
  • All other file types are derived from this type.
    • for example, piped into tools that expect sequential files.
legion basic file type1
Legion basic file type
  • There are two-dimensional files that support
    • read/write operations on columns, rows, and rectangular patches of data (both primitive types as well as “structs”).
  • There are file types to support
    • unstructured sparse data,
    • parallel files in which the file has been broken up and decomposed across several storage systems.
data movement on the grid
Data movement on the Grid
  • Data can be copied into the grid
    • in which case Legion manages the data
    • decides where to place it,
    • how many copies to generate for higher availability,
    • where to place those copies.
  • Data can be exported into the grid.
    • When a local directory structure is exported into the grid, it is mapped to a chosen path name in the global name space (directory structure).
data movement on the grid1
Data movement on the Grid
  • For example, a user can map
    • data/sequences in his local file system into /home/grimshaw/sequences using the legion_export_dir command,
    • legion_export_dir data/sequences /home/grimshaw/sequences.
    • Subsequent access from anywhere in the grid (read/write) is done directly against the files in the users file system (subject to access control).
slide30
data grid can be accessed via a daemon that implements the Network File System (NFS) protocol.
  • The entire Legion namespace, including files, hosts, and so forth, can be mapped into local OS file systems.
  • Shell scripts, Perl scripts, an user applications can run unmodified on the Legion data grid.
  • The usual UNIX commands work, as does Microsoft Windows Exploring
security in legion

Security in Legion

Overview of Security Standards in the Grid, CSE 225 High Performance and Computational Grids Spring 2000 [email protected]

security in legion 1 design principals
Security in Legion (1)Design Principals
  • As in the Hippocratic Oath, do no harm!
  • Caveat emptor - let the buyer beware.
  • Small is beautiful.
security in legion legion security model
Security in LegionLegion Security Model
  • responsible for securing the communications between Legion objects
  • responsible for access control
    • determines what objects/ users are allowed to call a particular object’s methods.
    • relies on the message layer for some services.
security in legion 2 basic concepts
Security in Legion (2)Basic Concepts
  • Every object provides certain known member functions
    • MayI, CanI, Iam, and Delegate.
  • Two objects associated with each operation:
    • a responsible agent (RA)
    • and a calling agent (CA)
  • Every invocation of member function is performed in the context of
    • a certificate which contains the LOID.
    • Certificate digitally signed by maker
security in legion1
Security in Legion
  • Object A invokes method B.foo
  • call is passed to B, and the Legion run-time system automatically invokes B.MayI
    • If B.MayI is true, then foo is invoked with the arguments
    • If not, then an exception is raised and passed back the information necessary to make such a decision
security in legion2
Security in Legion
  • Legion users are responsible for own security.
  • Object might trust that the CA is correct.
  • Policies defined by objects themselves.
  • Every class defines a special member function, MayI.
  • MayI defines the security objects for a class.
  • Every member function invocation permitted only if MayI sanctions it.
security in legion automatic invocation of outgoing calls
Security in LegionAutomatic invocation of outgoing calls
  • Object A invokes B.foo,
  • the compiler has interposed code so that A.CanI is automatically invoked before the call leaves A.
    • If CanI returns true, then the method call proceeds.
    • If CanI returns false, an exception is raised.
security in legion3
Security in Legion
  • Authentication aided by use of Legion certificates
    • - based on public-key cryptography by default. Must know private key to authenticate.
  • MayI functions can code their own authentication protocols
  • Every Legion object required to supply special member function Iam for authentication purposes.
security in legion4
Security in Legion
  • Login establishes user identity and creates responsibility agent for user.
  • Login is building block for authentication and delegation.
  • Object can delegate new certificate to delegate rights.
  • Delegation policy defined by object.
security in legion future work
Security in LegionFuture Work
  • Legion does not specify any particular encryption. Future standardization?
  • Legion eschews distinguished trusted objects - centralized key management server
  • Composition of a security policy
user portal and system management tools
User portal and system management tools
  • the user portal and system management tools
    • add and remove users,
    • add and remove hosts,
    • and join two separate Legion grids together.
  • Web-based portal interface
    • for access to Legion
    • a system status display tool that gathers information from system-wide metadata collections and makes it available via a browser
web based portal
Web-based portal
  • Web-based portal allows an alternative, graphical interface to Legion.
  • Using this interface a user can submit
    • An Amber job (a 3D molecular modeling code) to NPACI-net (a University of Virginia Legion network)
    • and not care at all where it executes.
portal view
Portal view
  • user can copy files out of the running simulation
  • and in which a three dimensional molecular visualization is being used to display the intermediate results.
legion job status tools
Legion job status tools
  • Using these tools users can determine the status
    • of all of the jobs they have started from the Legion portal
    • and access their results as needed.
legion accounting system
Legion accounting system
  • The portal interface to the underlying Legion accounting system.
  • Legion keeps track of who used resource (CPU, application, etc.),
    • with exit status
    • with how much resource consumption.
  • Data is loaded into a relational database, and various reports can be generated
slide48
Originally a research project at U. Virginia
    • commercialised via Applied Meta (now Avaki Corp)
  • Some successful applications
    • used to perform a metacomputing run of an ocean model
    • ran a task-farm of computational chemistry calculations
  • Grid Object Model not really found favour
    • in development phase, Grid must run existing applications
    • hard to do this within an integrated environment
    • major applications are not themselves object oriented
    • however cf. OGSA

Rob Baxter

A Complete History of the Grid (abridged)

context space2
Context Space

/

  • Unix-like

legion_ls

legion_pwd

legion_cd

legion_cat

...

hosts

home

users

mach1

mach2

mydir

you

me

subdir

prog

file1

tty

context space3
Context Space
  • Network-wide, transparent file system
  • Location-independent read/write of files
  • Convenient transfer of files between context space and local file system
  • I/O libraries for access
  • Unix-like utilities
context example
Context Example

legion_ls /

another context
Another Context

legion_ls /hosts

yet another context
Yet Another Context

legion_ls /users

other context commands
Other Context Commands
  • Locate a LOID in context space

legion_list_names

  • Locate an object on a machine

legion_whereis

  • Find status of an object

legion_object_info

  • List metadata of an object

legion_list_attributes

status of an object
Status Of An Object

legion_object_info -c work

physical location of object
Physical Location Of Object

legion_whereis -c work

context space vs local space
Context Space vs. Local Space
  • Local space = your machine’s directory structure
    • OS-specific, Machine-specific
    • Use cp, copy, etc.
    • e.g., C:\Program Files\, /usr/bin, /mnt/disk1
  • Context space = Legion’s directory structure
    • OS-independent, Machine-independent
    • Uselegion_cp, etc.
context space and local space
Context Space and Local Space
  • Transfer one file from local space to context space

legion_cp -localsrc <localfile> <contextfile>

  • Transfer one file from context space to local space

legion_cp -localdest <contextfile> <localfile>

context space and local space1
Context Space and Local Space
  • Copying local directory to context space

legion_cp -r -localsrc <localdir> <contextdir>

OR

legion_import_tree <localdir> <contextdir>

  • Copying context directory to local space

legion_cp -r -localdest <contextdir> <localdir>

context space and local space2
Context Space and Local Space
  • Map (not copy!) local directory to context space temporarily

legion_export_dir <localdir> <contextdir>

  • Does NOT make copy of local directory
  • Merely provides Legion-like access to local directory
    • Use legion_cat on local files
making context space
Making Context Space…
  • Local sub-directory with Legion NFS daemon
    • Use cat on context files
  • FTP directory with FTP interface
  • Windows directory with Samba interface
  • URL tree with HTTP interface
i o performance
I/O Performance
  • X-Axis = number of clients simultaneously performing 1MB reads on 10MB files
  • Y-Axis = total read bandwidth
  • Each point = average of multiple runs
  • Clients = 400MHz Intels, NFS Server = 800MHz Intel
making context space1
Making Context Space…
  • Local sub-directory with Legion NFS daemon
    • Use cat on context files
  • FTP directory with FTP interface
  • Windows directory with Samba interface
  • URL tree with HTTP interface
flexible context space
Flexible Context Space

Disk

e

ftp

Directory

NFS

HTTP

Samba

FTP

Context

Context

Context

legion_export_dir

legion_import_tree

Disk

Directory

Context

Directory

Directory

access control
Access Control
  • MayI for each object implements access control on a per-function basis
  • Users named by login object
  • Sets of users grouped by contexts

legion_change_permissions [+-rwx] [-v] <group/user context> <target context>

legion_change_permissions +r /users/fred /home/grimshaw/myfile

unified console1
Unified Console

TTY

File

Program produces

stdout, stderr

User creates

tty object

Prog.

User shares

tty LOID

User starts

running program

Legion passes tty

LOID to program

User shares

tty LOID

tty object1
TTY Object
  • Redirect run-time output to central (or multiple) consoles
  • Connect and disconnect dynamically
  • Debug quickly and simply
  • Monitor status, errors, easily
  • Share console with others

legion_tty <ttyobj>

ad