ARC Roadmap Plans and dreams

1. ARC RoadmapPlans and dreams Jon Kerr Nilsen, ARC Release Manager

2. Outline • A bit on ARC releases and version numbers • https://wiki.nordugrid.org/wiki/Release_management#Release_categories • Roadmap • Short term plans • Long term dreams • https://wiki.nordugrid.org/wiki/Roadmap www.nordugrid.org

3. ARC releases • A NorduGrid ARC release consists of a set of source packages • A certain source package is identified by its version number, <major>.<minor>.<bugfix> • E.g., NorduGrid ARC 4.1.0 or NorduGrid ARC Documents 1.3.4 www.nordugrid.org

4. Release types • major release • longer term planning • 3-6 months preparation • alpha, beta, rcx test releases • introduces new components, features • obsoletes components • may break backward compatibility • scheduled release • this release bumps the major number in the version number of the package causing the major release www.nordugrid.org

5. Release types • minor release • planned bugfix release • max one month preparation • no new components • only minor new features and/or significant bugfixes • scheduled release • this release bumps the minor number in the version number of the package causing the minor release www.nordugrid.org

6. Release types • minor release • planned bugfix release • max one month preparation • no new components • only minor new features and/or significant bugfixes • scheduled release • this release bumps the minor number in the version number of the package causing the minor release www.nordugrid.org

7. Release types • bugfix release • planned bugfix release • max one month preparation • no new features, no new components • only a limited number of less significant fixes • scheduled release • this release bumps the bugfix number in the version number of the package causing the bugfix release • emergency release (rarely used) • unplanned urgent release to fix a security issue or a critical bug • effectively the same as bugfix release, but may or may not require bump of bugfix number www.nordugrid.org

8. ARC roadmap • ARC enables a number of national Grid/e-Science infrastructures • one of the solutions used by the European Grid Infrastructure • ARC Roadmap is based on the necessity to continuously support such infrastructures, and evolve following user requirements and technology developments • Implemented features and new functionality from the Roadmap are rolled out with the ARC releases • NorduGrid aims to deliver one major release per year around every spring and a few minor releases per year www.nordugrid.org

9. Short term plans On the one-year perspective the following tasks serve as main objective for ARC development: • arc.conf restructuring and major clean-up • implement consistent server-side logging • revise the way Virtual Organizations are treated within ARC • documentation review including necessary updates due to configuration changes • provide a solution for automatic server-side performance data collection, including the definition of performance metrics • improve scalability of the data-transfer subsystem • improve overall ARC performance by various reorganization within the "controldir" • testing and fine tuning the GLUE2-based information system integration with Top-BDII information tree • continued roll-out of the WS-based job submission and management interface of A-REX: the EMI-ES channel. • provide virtual images and containers of various ARC services, provision A-REX as a cloud appliance, enter into the IaaS and PaaS domain. • provide out-of-the box support integration with pilot job frameworks, the dominant job management platform of LHC community • short term migration to python-based LRMS for some of the batch systems. Start the work of re-implementation of the LRMS backend layer of A-REX, try to move away from perl www.nordugrid.org

10. Configuration restructuring and documentation review • The current configuration of ARC CE is sometimes less than intuitive and has gathered quite a bit of old, unused artifacts from the last 15 years • Documentation of configuration parameters is suffering from fragmentation • Some parameters are documented in several places (arc.conf.reference, sysadmin guide, component specific manuals, wiki, technical documents…) • Next major release will contain a complete clean-up of the configuration and documentation of config parameters • arc.conf.reference will be the main reference • More details provided in sysadmin guide • No other documents should be needed to configure ARC www.nordugrid.org

11. Configuration changes • Description of each and every blocks and parameters have been reviewed, refined, improved • Every config value will have explicit information on • default value • multiplicity • if mandatory or not • Consistent parameter naming across all blocks • E.g., certificate-related parameters, logfileand loglevel • Introducing clear functionality based block structure • If a block is enabled the service/interface is enabled • Removes many implicit surprises • E.g., BES interface was enabled implicitly with setting arex_mountpoint-> most people enabling EMI-ES got surprised that they implicitly enabled BES as well • Hierarchical structure with blocks/sub-blocks • If a certain functionality is not needed, just skip configuring the corresponding block • E.g., all argus-related stuff is grouped in a dedicated block, if you don’t need argus, just skip that block • Most radical change is the way accounting is configured • as that used to get the most complaints • Grid-manager block will look quite different due to new interface sub-blocks • [grid-manager/wsinterfaces/…] • Config changes in numbers: • From 399 to 366 variables • 52 paramenters deleted (and some new ones added) • Old config contained 437 config objects (block headers and config parameters), out of which 157 will be changed www.nordugrid.org

12. Performance/scalability improvements • Starting with automating collection of performance metrics (planned for minor release in March) • Based on the gathered metrics, identify bottlenecks and prioritize possible improvements • Heavy use of file based control directory and scalability of data transfer system are likely candidates for improvement www.nordugrid.org

13. Code cleanup • While not visible to the users, some parts of the ARC code are hard to debug and maintain • Current batch system backends are written in a mix of Perl and BASH • Information system implementation is quite messy, with some pretty scary Perl incantations • First step: Introducing Python based backends • See tutorial after this talk www.nordugrid.org

14. Interfaces • Current job interface relies heavily on gridftp • Pulls in a large set of dependencies from Globus • Will Continue roll-out of WS-based job submission and management interface of A-REX: EMI-ES • HTTPS based, just as fast as gridftp, but more standard (outside grid world) and far fewer build dependencies • Complete move from GLUE1.x to GLUE2 • Now also pushed by EGI and WLCG • Requires testing and fine tuning of the GLUE2-based information system integration with Top-BDII information tree • Some infosys related bugs can only be resolved by moving to GLUE2 www.nordugrid.org

15. Longer term plans On the two or three year perspective we are considering the following development activities: • Replacement of the EGIIS-based information indexing layer of ARC. A GOCDB-based system may be a candidate. • Interface simplification: our dream is to arrive at the "one set of golden ARC interface" status. • New intermediate layer between ARC components and the batch systems • Containerized Runtime Environments. Complete re-design of the ARC RTE concept. • Support for Federated Identities, Open up ARC for Federated Identity Management solutions • Opening up A-REX for better interactive access, interactive analysis is highly requested missing feature • Comply with the data access protocols and metadata/file catalogue directions of WLCG • Develop a prototype ARC client on one of the mobile platforms • Experiment with intelligent auto-parallelization mechanisms built into A-REX • Consider moving away from LDAP as an information system technology (risky) www.nordugrid.org

16. Phase out plans • Short term • CANL security libraries • Longer term • GridFTPjobplugin interface of the CE to be completely replaced by EMI-ES • glue1.3 and nordugrid schema support to be replaced by glue2 www.nordugrid.org

17. Conclusion www.nordugrid.org