1 / 32

Europeean Condor Week 2006 Milano, June 26-29, 2006

Condor for building with NMI and ETICS Marian ZUREK. Europeean Condor Week 2006 Milano, June 26-29, 2006. Overview. What is ETICS What is NMI Where is Condor NMI architecture NMI job HelloWorld example ETICS Testbed What we do using NMI & Condor gLite testing requirements Conclusions

jory
Download Presentation

Europeean Condor Week 2006 Milano, June 26-29, 2006

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Condor for building with NMI and ETICS Marian ZUREK Europeean Condor Week 2006 Milano, June 26-29, 2006

  2. Overview • What is ETICS • What is NMI • Where is Condor • NMI architecture • NMI job • HelloWorld example • ETICS Testbed • What we do using NMI & Condor • gLite testing requirements • Conclusions • Wish list • Q & A CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  3. What is ETICS • ETICS stands for e-Infrastructure for Testing, Integration and Configuration of Software • It’s an SSA • Vision: A dependable, reliable, stable grid infrastructure requires high-quality, thoroughly tested, interoperable software middleware and applications • Mission: Provide a generic service that other projects can use to efficiently and easily build and test their grid and distributed software. • Set up the foundations for a certification process to help increasing the quality and interoperability of such software CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  4. Build system, software configuration, service infrastructure, dissemination, EGEE, gLite, project coord. Software configuration, service infrastructure, dissemination Web portals and tools, quality process, dissemination, DILIGENT The Condor batch system, distributed testing tools, service infrastructure, NMI Test methods and metrics, unit testing tools, EBIT The ETICS Consortium CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  5. Web Application NMI Scheduler Web Service ETICS Service Overview Via browser Build/Test Artefacts Report DB Project DB Command- Line tools Clients NMI Client Wrapper WNs ETICS Infrastructure Continuous Builds CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  6. Module Details Module Configurations Project Structure Browsing a project CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  7. Configuration Browser Browsing configurations Module Details Conf Details Sub-Configurations Sub-Configurations Commands Properties Environment Dependencies Build Schedules CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  8. Command line tools • Command Line Interface (CLI) is developped along with the WebApplication and offers the same functionality as WebApplication (they use the same underlying web service libraries) • Written in Python • Using ZSI • CVS-style command syntax • etics-* (etics-get-project, etics-checkout, …) • The CLIs are directly used for NMI/Condor job/tests submission • etics-get-project CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  9. List of Commands: Submit • etics-remote-build [options] –c <configuration-name> -p <platform-name-list> -m <project-name> • It performs a remote build for an ETICS project • The remote build consists on submitting a request to an external build system that uses the ETICS infrastructure to build • By default, the command remotely builds the current project by using the current configuration and platform. • It returns an ID that will be used to get the status build • -p, --platforms • It specifies the list of platforms where to build • -m, --project • It specifies the project name to build • -c --configuration • It specifies the configuration name to be used to build • --all-platforms • It means that the remote build is performed on all platforms CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  10. CLI - example What is under the hood ? • cvs co org.etics (etics-get-project, etics-checkout) • ./configure; make dist | test | install (etics-build) CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  11. What is Condor and NMI • NMI is a multi-platform facility designed to provide (automated) software building and testing services for a variety of (grid) computing projects. • NMI is a layer on the top of Condor to abstract the typical complexity of the Build and Test process • Condor is offeringmechanisms and policies that support High Throughput Computing (HTC) on large collections of distributed computing resources CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  12. NMI architecture CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  13. NMI job a set of user commands/scripts representing the workflow of the build or test producing the output (optional) CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  14. NMI job CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  15. HelloWorld.nmi CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  16. HelloWorld.cvs CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  17. Dedicated UI WN CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  18. ETICS testbed • etics.cern.ch • Official ETICS submission node - production host • 2000+ jobs (as of 24th June 2006) • etics-test.cern.ch, etics-01.cnaf.infn.it • test submission nodes • etics-dev.cern.ch • development node • non-stable • etics-db.cern.ch • MySQL database host CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  19. etics.cern.ch pool • ia32 • 6x SLC3 • 2x SLC4 • 1x RHES v3.0 • 1x Debian v3.1 • 1x FC3 • 1x FC4 • 1x WinXP • ia64/x86_64 • 2 x SLC3 • 1 x SLC4 • 1x PPC CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  20. Production • Operational setup • WNs status page http://etics.cern.ch/nmi/?page=pool/index http://etics-test.cern.ch/nmi/?page=pool/index • Job submission status page http://etics.cern.ch/nmi/?page=results/overview http://etics-test.cern.ch/nmi/?page=results/overview CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  21. CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  22. CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  23. What we use NMI & Condor for ? CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  24. What we want to do • our final goal is to test the current gLite components with ETICS. We’d like to : • automatically deploy the full set of gLite services on different platforms • automatically have our full set of tests executed periodically • set explicit dependencies for the tests on other tests or specific middleware components: • «  execute test B only is test A was successful  » • « install component C1 and C2 if required by test A   » • easily access the test results • easily debug test failures • easily reproduce a test result from a specific configuration ==> for that we need reliably scheduling and executing infrastructure CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  25. gLite testing requirements • Dependency on UI (User Interface) • UI installation required for each test • Time impact --> dedicated UI Worker Node • append_requirements = ( Machine == “lxb1115.cern.ch”) • Proxy forwarding • Once proxy exists on the submit node NMI/Condor can forward it to the target WN • Freeze option • Shoud the test fail I want this node to leave condor pool and I want to debug the failing component(s) ==> we need very flexible environment CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  26. Root-enabled installations/tests • Some gLite tests require root environment • System test require services persistency • Some services after installation should remain operational for the predefined period of time • Configuration of the Condor running in the priviliged mode • sudo, STARTd configuration • ==> not the preferred operational mode, but possible (very close to the production quality) CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  27. Public/private pools • A project wants to use ETICS and brings in its private nodes and wants its full power to be private • Steering the jobs to this node, preventing from others landing there • ==> already implemented and operational CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  28. EGEE2 and Other Collaborations • The ETICS services are intended to provide from the start the integration, testing and QA tools for EGEE/EGEE2, Diligent and other partner projects • Part of the infrastructure and tools, the NMI infrastructure, are the same already used by project like Condor and VDTat the UoW. It also provides a direct bridge between EGEE2 and OSG • Other projects developing software and in particular applications providers are more than welcome to use the services and tests their products with ETICS • One of the major goals of ETICS is to strengthen the interoperability and quality of middleware and applications developed for the Grid and other complex distributed environments. Everybody’s collaboration is welcome and needed. CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  29. Conclusions • We need robust components • We need reliable scheduling and execution environment for the complex workflows • We have many spread resources (CERN, INFN, Madison, …) and want to access them transparently • Collaboration with UW • Very good support from the Condor Team (Peter COUVARES and Anatholy KARP) • Bi-weekly technical meetings • We are on the track CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  30. Wish list … • Co-scheduling/parallel system tests • Persistency • Virtualisation • Light dynamic ClassAd publish mechanism • Easy mechanism for match-making investigation • condor_q -better-analyse • Quill++ ? • How about your wishes for ETICS? CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  31. Want to learn more ? http://www.eu-etics.org etics-support@cern.ch CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

  32. Certification question Standards, QA, interoperability, … ETICS ETICS PAB User communities In what sense software is “good”? The software adheres to agreed standards, it executes in the expected way, satisfied a number of qualitative and quantitative requirements, etc, etc, etc CondorWEEK’2006, Milano, Marian.ZUREK@cern.ch

More Related