1 / 20

gLite Middleware Status

gLite Middleware Status. Frédéric Hemmer, JRA1 Manager, CERN On behalf of JRA1 EGEE 3 rd Conference April 18-22, 2005 Athens. Outline. Contents of Release 1.0 Processes, Tools and Statistics Next steps Summary. Contents of Release 1.0. JRA3. UK. Access Services.

bcaves
Download Presentation

gLite Middleware Status

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. gLite Middleware Status Frédéric Hemmer, JRA1 Manager, CERN On behalf of JRA1 EGEE 3rd Conference April 18-22, 2005 Athens

  2. Outline • Contents of Release 1.0 • Processes, Tools and Statistics • Next steps • Summary 3rd EGEE Conference, Athens - gLite Middleware Status

  3. Contents of Release 1.0

  4. JRA3 UK Access Services Grid AccessService API CERN IT/CZ Security Services Authorization Information & Monitoring Services Application Monitoring Information &Monitoring Auditing Authentication Data Services Job Management Services MetadataCatalog File & ReplicaCatalog JobProvenance PackageManager Accounting StorageElement DataManagement WorkloadManagement ComputingElement Site Proxy gLite Release 1.0 • Job management Services • Workload Management • Computing Element • Logging and Bookkeeping • Data management Services • File and Replica catalog • File Transfer and Placement Services • gLite I/O • Information Services • R-GMA • Service Discovery • Security • Deployment Modules • Distribution available as RPM’s, Binary Tarballs, Source Tarballs and APT cache 3rd EGEE Conference, Athens - gLite Middleware Status

  5. Job Management Services • Efficient and reliable scheduling of computational tasks on the available infrastructure • Started with LCG-2 Workload Management System (WMS) • Inherited from EDG • Support partitioned jobs and jobs with dependencies • Support for different replica catalogs for data based scheduling • Modification of internal structure of WMS • Task queue: persistent queue of pending submission requests • Information supermarket: repository of information on resources • Better reliability, better performance, better interoperability, support push and pull mode • Computing Element (CE) • Service representing a computing resource • CE moving towards a VO based local scheduler • Incorporates new technologies provided by Condor and Globus • Policy based notifications on changes of the CE (pull model) • Logging and Bookkeeping • Tracks jobs during their lifetime (in terms of events) • Backward compatibility(to facilitate migration from current SA1 infrastructure) 3rd EGEE Conference, Athens - gLite Middleware Status

  6. Catalog Catalog Catalog MOM MOM MOM LocalCat LocalCat LocalCat Data Management Services • Efficient and reliable data storage, movement, and retrieval on the infrastructure • Storage Element • Reliable file storage (SRM based storage systems) • Posix-like file access (gLite I/O) • Transfer (gridFTP) • File and Replica Catalog • Resolves logical filenames (LFN) to physical location of files (URL understood by SRM) and Storage Elements • Hierarchical File system like view in LFN space • Single catalog for Release 1.0 • File Transfer and Placement Service • Reliable file transfer and transactional interactions with catalogs • Technology Preview only • Metadata Catalog • Limited metadata can be attached to the File and Replica Catalog • Interface to application specific catalogs have been defined • Web Services • WSDL available • Command line interface available Data Scheduler VOs Catalog Site boundary VOs FPS FPS FPS FPS TransferAgent SRM GridFTP I/O Storage Element 3rd EGEE Conference, Athens - gLite Middleware Status

  7. Information and Monitoring Services • Efficient and reliable provision of Grid Information and Grid and Application monitoring data • R-GMA (Relational Grid Monitoring Architecture) • Implements GGF GMA standard • Development started in EDG, deployed on the production infrastructure for accounting • Producer, Consumer, Registry and Schema services with supporting tools • Registry replication • Simpler API – matching the next (WS) release • Provides smooth transition between old API and WS • coping with life on the Grid: poorly configured networks, firewalls, MySQL corruptions etc • Generic Service Discovery API 3rd EGEE Conference, Athens - gLite Middleware Status

  8. Security • VOMS • Virtual Organization Membership Service • Job Management Services • WMS, LB, and CE implement authorization based on VOMS VO, groups, and user information • Data Services • Authorization: ACL and (Unix) permissions • Fine-grained ACL on data enforced through gLite-IO and Catalogs • Catalog data itself is authorized through ACLs • Currently supported through DNs • VOMS integration being developed • Information Services • Authentication based on VOMS certificates (being developed) 3rd EGEE Conference, Athens - gLite Middleware Status

  9. gLite Services for Release 1.0Components Summary and Origin • Computing Element • Gatekeeper, WSS (Globus) • Condor-C (Condor) • CE Monitor (EGEE) • Local batch system (PBS, LSF, Condor) • Workload Management • WMS (EDG) • Logging and bookkeeping (EDG) • Condor-C (Condor) • Storage Element • File Transfer/Placement (EGEE) • glite-I/O (AliEn) • GridFTP (Globus) • SRM: Castor (CERN), dCache (FNAL, DESY), other SRMs • Catalog • File and Replica Catalog (EGEE) • Metadata Catalog (EGEE) • Information and Monitoring • R-GMA (EDG) • Service Discovery (EGEE) • Security • VOMS (DataTAG, EDG) • GSI (Globus) • Authentication for C and Java based (web) services (EDG) 3rd EGEE Conference, Athens - gLite Middleware Status

  10. Processes, Tools and Statistics

  11. Architecture & Design • Design team including representatives from Middleware providers (AliEn, Condor, EDG, Globus,…) including US partners produced middleware architecture and design. • Now includes Operations (SA1) • Takes into account input and experiences from applications, operations, and related projects • Focus on medium term (few months) and commonalities with other projects (e.g. OSG) • Effective exchange of ideas, requirements, solutions and technologies • Coordinated development of new capabilities • Open communication channels • Joint deployment and testing of middleware • Early detection of differences and disagreements • DJRA1.1 – EGEE Middleware Architecture (June 2004) • https://edms.cern.ch/document/476451/ • DJRA1.2 – EGEE Middleware Design (August 2004) • https://edms.cern.ch/document/487871/ gLite is not “just” a software stack, it is a “new” framework for international collaborative middleware development. Much has been accomplished in the first year. However, this is “just” the first step. 3rd EGEE Conference, Athens - gLite Middleware Status

  12. LCG-2 (=EGEE-0) 2004 prototyping prototyping product 2005 product From Development to Product • Fast prototyping approach allowing end users for rapid feedback • Provide individual components to SA1 for deployment on the pre-production service • See Markus Schulz talk [32] on Deployment/Migration plans • These components need to go through integration and testing • To ensure they are deployable and basically work 3rd EGEE Conference, Athens - gLite Middleware Status

  13. Software Process • JRA1 Software Process is based on an iterative method • It comprises two main 12-month development cycles divided in shorter development-integration-test-release cycles lasting 1 to 4 weeks • The two main cycles start with full Architecture and Design phases, but the architecture and design are periodically reviewed and verified. • The process is documented in a number of standard documents: • Software Configuration Management (SCM) Plan • Test Plan • Quality Assurance Plan • Developer’s Guide 3rd EGEE Conference, Athens - gLite Middleware Status

  14. Functional Tests Release Process Development Integration Testing Deployment Packages Software Code Fail Pass Testbed Deployment Integration Tests Fix Fail Pass Installation Guide, Release Notes, etc 3rd EGEE Conference, Athens - gLite Middleware Status

  15. QA and SCM Metrics • Several QA and SCM Metrics are mandated by the SCM and QA Plans • Metrics are calculated periodically and published on the gLite web site: • Total complete builds done: 208 • Number of subsystems: 12 • Number of CVS modules: 343(development, integration modules, test suites, documentation and tools) • Total Physical Source Lines of Code (SLOC) • SLOC = 632,478 (as of 5 April 2005) • Total SLOC by language (dominant language first) • C++ 193996 (30.67%)Java 183782 (29.06%)Ansi C 149411 (23.62%)Perl 62627 ( 9.90%)Python 24967 ( 3.95%)sh 12634 ( 2.00%)Yacc 3635 ( 0.57%) 3rd EGEE Conference, Athens - gLite Middleware Status

  16. Code Size and Stability The Code Size chart shows the changes in total number of SLOCs during the life of the project The Code Stability chart shows the change rate of code size during the life of the project.As the project nears completion the rate should approach 0 3rd EGEE Conference, Athens - gLite Middleware Status

  17. Bug Counts and Trends 3rd EGEE Conference, Athens - gLite Middleware Status

  18. Next Steps • Certification by SA1 • Defect handling • First priority • Issued as Quick Fixes (QF’s) when necessary • Monthly Release Cycles • Including contents of QF’s • May include new functionality as agreed with SA1/NA4 • Example File Transfer Service • Draft Work Plan available • Describing possible improvements • Will be discussed as of tomorrow • Revised Architecture and Designed • Scheduled for May and June’2005 3rd EGEE Conference, Athens - gLite Middleware Status

  19. Summary • gLite architecture, design and release plan have been produced • Will be revised in May and June • gLite release 1.0 with key functionalities implemented have been produced • Incorporating Web Services • Now ready to be exercised on the SA1 Pre-Production Service • Will be demoed within GILDA Wednesday • A software process is in place which is being exercised by SA1 on the Pre-Production Service • Including QA metrics • And defect tracking portal • Next steps based on the JRA1 workplan will now be discussed and prioritized with NA4/SA1 starting from tomorrow 3rd EGEE Conference, Athens - gLite Middleware Status

  20. More information http://www.glite.org glite-announce@cern.ch http://cern.ch/egee-jra1 3rd EGEE Conference, Athens - gLite Middleware Status

More Related