1 / 80

Legion: The Grid OS Architecture and User View

Legion: The Grid OS Architecture and User View. Anand Natrajan ( anand@virginia.edu ) Marty Humphrey ( humphrey@cs.virginia.edu ) The Legion Project, University of Virginia ( http://legion.virginia.edu ). Grid Environment. Disjoint file systems Disjoint namespaces

hye
Download Presentation

Legion: The Grid OS Architecture and User View

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Legion: The Grid OSArchitecture and User View Anand Natrajan (anand@virginia.edu) Marty Humphrey (humphrey@cs.virginia.edu) The Legion Project, University of Virginia (http://legion.virginia.edu)

  2. Grid Environment • Disjoint file systems • Disjoint namespaces • Multiple administration domains • Unpredictable load, availability, failures • Security problems • Computers • Networks • People • Data • Devices

  3. Wide-area High Performance Complexity Management Extensibility Security Site Autonomy Input / Output Heterogeneity Fault-tolerance Scalability Simplicity Single Namespace Resource Management Platform Independence Multi-language Legacy Support Grid OS Requirements

  4. Legion - A Grid OS

  5. MPI / PVM P-space studies - multi-run Parallel C++ Parallel object-based Fortran CORBA binding Object migration Accounting Remote builds and compilations Fault-tolerant MPI libraries Post-mortemdebugger Console objects Parallel 2D file objects Collections Licence support Tools

  6. Mentat Avaki Legion Commercial Support - Avaki Corp. Web • Venture funded • Headquartered in Boston • Growing number of employees • Multi-tiered support offering

  7. Protein Folding with CHARMM Molecular Dynamics Simulations 100-200 structures to sample (r,Rgyr ) space r Rgyr

  8. Resources Available IBM SP3 UMich 375MHz Power3 24/24 HP V-class CalTech 440 MHz PA-8700 128/128 DEC Alpha UVa 533MHz EV56 32/128 IBM Blue Horizon SDSC 375MHz Power3 512/1184 Sun HPC 10000 SDSC 400MHz SMP 32/64 IBM Azure UTexas 160MHz Power2 32/64

  9. Transparent Remote Execution • User initiates “run” • User/Legion selects site • Legion copies binaries • Legion copies input files • Legion starts job(s) • Legion monitors progress • Legion copies output files

  10. Mechanics of CHARMM Runs Register binaries Legion Create task directories & specification Dispatch runs Dispatch more runs

  11. Types Of Applications • Legacy applications • Legion-aware applications • I/O library • 2D file object • Applications Using Stdgrid • Parameter Space Studies • Parallel Programs • MPI, PVM, MPL, Basic Fortran Support (BFS)

  12. Grid Application Requirements • Security • Fault-tolerance • Heterogeneity • Collaboration • … • Legion supports these and other needs

  13. Heterogeneous Runs BT-Med Ocean Model

  14. Cross-Organisation Collaboration • Different companies • Proprietary simulations and data • Each needs the other • Form virtual partnership

  15. Windows NT, 2K, 98, 95 Sun (Solaris) SGI (Irix, Origin) Intel (Linux, Free BSD) DEC (Unix, Linux) Cray (T90, T3E) IBM (AIX, SP-2) HP (HPUX) Codine LoadLeveler Maui PBS NQS LSF Platforms

  16. Applications • Biochemistry and Molecular Science • Information Retrieval • Materials Science • Climate Modelling • Neuroscience • Aerospace • Astronomy • Graphics NPACI - SDSC, UCSD, Caltech, UTexas, Umich, UCB, UVa. DoD MSRCs - NAVO & ARL, NASA Ames

  17. User View Command-Line Interface

  18. Setup • Setup shell environment variables . ~legion/setup.sh OR export LEGION=/home/legion/Legion export LEGION_OPR=/home/maya/OPR . $LEGION/bin/legion_env.sh • Specifies where binaries and configuration files can be found • Sets root context

  19. Login • Authentication to system legion_login /users/stephen • Currently uses password - other mechanisms, e.g., Kerberos ticket possible • Login object (a.k.a. Authentication object) - /users/stephen - is user’s proxy to world • Login object generates certificate identifying user

  20. Context Space / • Unix-like legion_ls legion_pwd legion_cd legion_cat ... hosts home users mach1 mach2 mydir you me subdir prog file1 tty

  21. Context Space • Network-wide, transparent file system • Location-independent read/write of files • Convenient transfer of files between context space and local file system • I/O libraries for access • Unix-like utilities

  22. Context Example legion_ls /

  23. Another Context legion_ls /hosts

  24. Yet Another Context legion_ls /users

  25. More Context Fun

  26. Other Context Commands • Locate a LOID in context space legion_list_names • Locate an object on a machine legion_whereis • Find status of an object legion_object_info • List metadata of an object legion_list_attributes

  27. Status Of An Object legion_object_info -c work

  28. Physical Location Of Object legion_whereis -c work

  29. Context Space vs. Local Space • Local space = your machine’s directory structure • OS-specific, Machine-specific • Use cp, copy, etc. • e.g., C:\Program Files\, /usr/bin, /mnt/disk1 • Context space = Legion’s directory structure • OS-independent, Machine-independent • Uselegion_cp, etc.

  30. Context Space and Local Space • Transfer one file from local space to context space legion_cp -localsrc <localfile> <contextfile> • Transfer one file from context space to local space legion_cp -localdest <contextfile> <localfile>

  31. Context Space and Local Space • Copying local directory to context space legion_cp -r -localsrc <localdir> <contextdir> OR legion_import_tree <localdir> <contextdir> • Copying context directory to local space legion_cp -r -localdest <contextdir> <localdir>

  32. Context Space and Local Space • Map (not copy!) local directory to context space temporarily legion_export_dir <localdir> <contextdir> • Does NOT make copy of local directory • Merely provides Legion-like access to local directory • Use legion_cat on local files

  33. Making Context Space… • Local sub-directory with Legion NFS daemon • Use cat on context files • FTP directory with FTP interface • Windows directory with Samba interface • URL tree with HTTP interface

  34. I/O Performance • X-Axis = number of clients simultaneously performing 1MB reads on 10MB files • Y-Axis = total read bandwidth • Each point = average of multiple runs • Clients = 400MHz Intels, NFS Server = 800MHz Intel

  35. Making Context Space… • Local sub-directory with Legion NFS daemon • Use cat on context files • FTP directory with FTP interface • Windows directory with Samba interface • URL tree with HTTP interface

  36. Flexible Context Space Disk e ftp Directory NFS HTTP Samba FTP Context Context Context legion_export_dir legion_import_tree Disk Directory Context Directory Directory

  37. Access Control • MayI for each object implements access control on a per-function basis • Users named by login object • Sets of users grouped by contexts legion_change_permissions [+-rwx] [-v] <group/user context> <target context> legion_change_permissions +r /users/fred /home/grimshaw/myfile

  38. Access Control Example

  39. Unified Console TTY File Program produces stdout, stderr User creates tty object Prog. User shares tty LOID User starts running program Legion passes tty LOID to program User shares tty LOID

  40. TTY Object • Redirect run-time output to central (or multiple) consoles • Connect and disconnect dynamically • Debug quickly and simply • Monitor status, errors, easily • Share console with others legion_tty <ttyobj>

  41. User View Web Interface

  42. Logging In

  43. Listing Contents Of A Context

  44. Control Window

  45. Status Window

  46. StdOut Window

  47. StdErr Window

  48. Listing Classes (Contents of /class)

  49. Listing Hosts (Contents of /hosts)

  50. List Attributes Of An Object

More Related