1 / 20

Quality Assurance and Testing in LCG

Quality Assurance and Testing in LCG. CHEP 2004 Interlaken, Switzerland 30 September 2004 Manuel Gallas , Jakub MOSCICKI CERN http://spi.cern.ch mgallas@mail.cern.ch jakub.moscicki@cern.ch. Quality Assurance and Testing in LCG. LCG SPI Context SPI Services Overview

maggy-schultz
Download Presentation

Quality Assurance and Testing in LCG

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Quality Assurance and Testing in LCG CHEP 2004 Interlaken, Switzerland 30 September 2004 Manuel Gallas, Jakub MOSCICKI CERN http://spi.cern.ch mgallas@mail.cern.ch jakub.moscicki@cern.ch

  2. Quality Assurance and Testing in LCG • LCG SPI Context • SPI Services Overview • Sw-Testing: • Test Frameworks • Test Documents • User support • Quality Assurance: • QA activities • Bug tracker statistics • Test coverage • LCG App.Area policies • Summary Outline: SPI - Software Process & Infrastructure

  3. LCG Application Area SIMULATION LCG SPI project LCG SPI context LHC grid software applications (LHC experiments, projects, etc) • LCG Application Area software projects • POOL: Persistency • SEAL: Core common software • PI: Physics Interfaces • SIMU: Simulation • …etc… LCG Infrastructure • Common services • Similar ways of working (process) • Tools, templates, training • General QA, tests, integration, release SPI - Software Process & Infrastructure

  4. Software Development Development Coding Analysis and Design Release Quality Assurance Testing Specifications Deployment and Installation Documentation Build systems SPI project General Services CVS service External Software Web Portal Collaborative Facilities Task Management Mailing Lists SPI Services Overview Provide General Services needed by each project • CVS repository, Web Site, Software Library • Mailing Lists, Bug Reports, Task Management, Collaborative Facilities Provide solutions specific to the Software Development phases • Tools, Templates, Policies, Support, Documentation, Examples • Software Testing and QA  The rest of the talk describes QA and Sw-Testing services SPI Web Site - http://spi.cern.ch SPI - Software Process & Infrastructure

  5. Nightly Building System SPI SW-Testing Automated testing Use in Exp. Software testing Acceptance test Examples LHC experiments System test System Tests Sw-testing team Integration test Integration Tests Software developer Work Package Test Unit test CODE SW- testing support SW-testing polices Test FrameWorks SW-testing doc Code Documentation SW-Testing Overview - http://spi.cern.ch/testing • Software testing should be an integral part of the software development in the LCG App Area. • All level of software testing should be run as part of an automatic process. SPI - Software Process & Infrastructure

  6. to produce test code suitable to be run in automatic way • to document their tests • to plan their test phase • to run all their test in automatic way • to browse and scan easily the test results and failures Aim:to help developers: to help projects: • Our constraints: • Avoid commercial software and • licensing problems. • Avoid “do it yourself solutions” • Try to adopt commonly used • open-source software. • Our inputs: • Contacts within HEP-community. • What is available as free open source code. LCG-SPI SW-Testing • Requirements from the users : • Work with different languages (C++, Python, …) and in different environments (platforms & compilers) • Allow two ways of testing: (1) “check the output of the test-code” • (2) “check the test results inside the test-code”. • Provide a easy way to integrate existing tests. • Allow the creation of dependencies among tests. • Easily establish a criteria if a test should pass or fail in a given release • Organize the tests by components or packages / integration / system /acceptance . • Regression testing and check respect to the previous sw-release. • Easy integration with the Nightly Building System. • Provide also a graphical interface to run the tests and examine the test results. SPI - Software Process & Infrastructure

  7. End User Installation checks QA activity Project Release Nightly Building Sw-Testing Top layer QMTest • Uses a GUI for creating and running tests (also in batch). • Can run tests in parallel, supports execution of a single • test or many at once (test-cases & test-suites) • Organizes tests hierarchically • Records dependencies among tests - Integrates different ways to test. - Common environment to run the tests and to access the test results. Oval CppUnit PyUnit QtUnit JUnit Bottom layer Acceptance testing SW Product Examples - Adaptable to the programming language and developer - Prepared to be run in automatic way Old tests Test Scripts Unit testing X-Unit family Test Frameworks: Global picture SPI - Software Process & Infrastructure

  8. A simple test: • Subclass the TestCase CppUnit or PyUnit class • Override the method runTest(). • When you want to check a value, call the (CppUnit or PyUnit) ASSERT(bool) and pass inan expresion that is true if the test succeeds CppUnit PyUnit QtUnit JUnit X-Unit family Test Frameworks: X-Unit family • Main Scope: Unit–testing • The same “assertion style” in different languages. • Provides:- The name of the test case that failed. • -The name of the source file that contains the test. • -The line number where the failure occurred • Different platforms/compilers: (Linux/Solaris/Windows) SPI - Software Process & Infrastructure

  9. Oval build Test Source Code • Oval: • validation and regression • developed & used within • CMS experiment Executable Oval run Ovalfile (configuration) Log file Reference file Oval diff Test Frameworks: Oval • Can be use for Unit-test to Validation tests. • Compare the output log file with a given reference file (Smart comparison of those lines which start with “[Oval]”) • It is possible to set different run environments. • Can run external scripts and external binaries. SPI - Software Process & Infrastructure

  10. QMTest Test Frameworks: QMTest • Uses a graphical interface for creating and running tests • The configuration files are in XML and can be created from the GUI. We provide also script to do it • Runs tests in parallel • Organizes tests hierarchically • Supports execution of a single test or many at once • Records dependencies among tests • Can be run in batch mode -> easy integration with the Nightly-Building systems SPI - Software Process & Infrastructure

  11. Test Case template collects information about the existing tests and how they cover the expected functionality of the software. Test Documents • For all project is needed a Test Plan in order to have a clear idea of the type of tests, schedule, environment and responsibilities. • We use a template SPI - Software Process & Infrastructure

  12. Policies Test Frameworks HowTo Test doc User support: Web information http://spi.cern.ch/ http://spi.cern.ch/testing SPI - Software Process & Infrastructure

  13. Follow sw-testing policies Automatic testing requires some standardization 1 3 2 User support: How-To * * * * * SPI - Software Process & Infrastructure

  14. Quality Assurance Service - http://spi.cern.ch/qa • The main goal of QA activity is help LCG projects • assess and improve the quality of the software • provide tools to collect useful metrics/statistics which help toasses quality; • generate reports; • verify if project setup is correct with LCG policies. • QA Tools and Focus • Automatic reports • Development/integration of automatic tools • LCG Policies • agreed and defined by AF • SPI supports them in the tools and procedures and only helps to work them out SPI - Software Process & Infrastructure

  15. Automatic QA Reports • Generated at every release • Published on the SPI web site • External tools • Test coverage • SLOC, Valgrind, ignominy Quality Assurance Activities QA Checklist on each Release • Build the release • Run automatic tests • Metrics • Test Inventory & Metrics • Documentation/Examples Inventory • Bug tracker Statistics • Code Inventory • Rule Checker , Logiscope • LCG Policies • Configuration of a build system • CVS directory structure • Software testing polices • QA procedure: well-defined, transparent, open • clear rules and checklist of assessed items • anybody at anytime may see statistics • anybody may create reports themselves • anybody may contribute SPI - Software Process & Infrastructure

  16. Bug tracker statistics • Testing inventory by package • Results of the automatic testing • Test coverage • Source code statistic (n files, n lines, sloc, .. ) • CVS structure QA Reports SPI - Software Process & Infrastructure

  17. QA reports: Bugtracker Statistics • Defect Analysis in Time • Defect Reparation Trends • Activity of the project / users • Flexible category selection subsystems, severity,... • Caveat: • not all bug reports are“real” bugs • Correlation of defects and tests • Traceability • Every bug fix should be accompanied with a test case • Test case should contain a bug reference # • Avoiding reappearing bugs SPI - Software Process & Infrastructure

  18. QA reports: Test Coverage • Coverage of • Unit tests • Integration tests • Validation tests • At the level of • Project • Package • File • Source Lines • Tools • gcc3.2 –gcov • LTP scripts SPI - Software Process & Infrastructure

  19. LCG Policies • LCG Policies • CVS and Build Directory Policy • Software Testing Policies • Version Numbers, Tagging and Release Procedure • Installation Directory Structure • Platform string, binary names, debug flags and more • They are a needed by the LCG • They are defined by the LCG projects (AF), collected by SPI • SPI supports them in the tools and procedures and only helps to work them out • If everything is different is too difficult to use and to automate • compromising on our habits, for project needs • tell when they are not followed • First time that this exists at this extent, and that is checked QA tools verify the compliance of software to the Policies SPI - Software Process & Infrastructure

  20. Summary Quality Assurance and Testing in LCG • LCG SPI Context • SPI Services Overview • Sw-Testing: • Test Frameworks • Test Documents • User support • Quality Assurance: • QA activities • Bug tracker statistics • Test coverage • LCG App.Area policies • Summary • QA activities • designed to match the HEP environment • focused more on testing and bug tracking • less on sophisticated source code metrics • integrated and automated into the LCG development environment • QA infrastructure will be reused in EGEE • Software testing activities • a complete and coherent automatic testing infrastructure was deployed based on existing tools in the open source community. It covers covers from unit to validation tests. • a unique entry point (QMTest) can be use to run all the tests at the release time, in the nightly building system, for the QA reports or by the developer. SPI - Software Process & Infrastructure

More Related