WP 4 : Testing Tools and Methodologies

1. WP4: Testing Tools and Methodologies Status Report ETICS Review – 15February 2008 Éva Takács (4D SOFT)

2. Content • WP4 in numbers • Goals • Challenges • Distributed testing • Achievments • Case study • Future plans • Conclusion

3. WP in numbers • 4D Soft: PM – 24 ? • UW: PM –18 To be completed following the template

4. Staffing and Resources/ 4D Soft ????? István Forgács Tasks: Coordination Etics, Diligent Eva Takács – full time Etics Tasks: Implementation of plugins Implementation and testing of installation modules Diligent system testing – modelling in ETICS Andor Dirner – 30% Etics, 60% Diligent, 10% system administration Tasks: Implementation and testing of installation modules Diligent testing 4

5. Deliverables Status ETICS-D4.1-Requirements for unit, functional and regression testing – M6 ETICS-D4.2-Distributed testing – M12 ETICS-D4.3-Distributed testing – draft M18 ETICS-D4.3-Distributed testing – final M22 ETICS 3rd All-Hands Meeting - University of Wisconsin, Madison - 23-25/05/2007 5

6. Goals The major goal of WP4 is to realize distributed testing by: • Identification and analysis of test metrics • Identification and selection of software testing tools and metric calculation utilities • Adopting of the selected testing tools and metric calculation utilities to the ETICS build and test system (Test and Metrics Plugin Framework) by plugin implementation • Implementation of distributed testing mechanisms

7. Challenges • For the WP4 as technical work • Realising distributed testing – a quite new aspect in software engineering • Customising existing ‘traditional’ testing technics for cutting-edge new technologies • Identifying the most suitable open source test tools that fit te best in the ETICS system and could help the users with testing facilities after performing builds and before releasing software packages • For 4D Soft as company • Organisational challenge – being a WP leader as an SME in a project having mostly large academic institutions • Technical challenge - working with open source operation systems/tools widely adopted by academic institutions in contrast to the industry working mostly with commercially available software

8. Distributed testing in ETICS The aim of ETICS is to provide a distributed build/test framework where the functionalities of the framework and the testing tools are clearly separated. Having these principles in mind in ETICS distributed testing environment consists of four major elements: • Test and Metrics Plugin framework – as part of the ETICS client (command line interface) • Testing tools and Tools for Metrics collection as plugins of the Test and Metrics Plugin framework • NMI/Metronome Build&Test system – as execution framework • co-scheduling mechanism – feature that allows automated multi-site deployment and test execution

9. Test and Metrics Plugin Framework In order to host the wide range of testing tools (client-side), without breaking the ETICS’s core functionality, a plugin-framework was developed The goal of the plugin framework can be summarised as follows: • Provide an extendible framework for common actions to be performed during build and test execution • Provide natural separation between specific build and test execution and analysis and metrics collection • Provide open interface for users to register their own plugins (comercially available tools) to be executed during build and test • Provide raw data for build and test and repository services especially for tracking trends, having statistics

10. Why using Plugin Framework ? ETICS ships with some built-in plugins but the rational behind using „Eclipse style” plug-in architecture is • that most of present and/or future ETICS users will come with some tests and testing tools already in place and of course they would like to use them when working with ETICS • Users can plug comercially available testing tools • Commercial providers can extend the sytem with their products

11. Examples of metrics provided • Static analysis • Single line of code count (SLOC) • Cyclomatic complexity • Depth of inheritance • IPv6 compliance • Check style • Nr of possible bugs • Dynamic analysis • Code coverage • Memory leaks • Test execution • Java: JUnit • C++: CPPUNIT (CPPTEST) • Python: PyUnit • Shell: shutil • Script/Executable

12. Implemented Test and Metrics plugins • PyUnitPlugin.py (pyton unit tests) • PyCoveragePlugin.py (pyton coverage) • PyComplexityPlugin.py • SLOCCountPlugin • IPv6Plugin • WSInteroperabilityPlugin • JUnitPlugin.py (junit) • JUnitreportsPlugin.py • JUnitemmaPlugin.py (emma coverage for junit) • JCoberturaPlugin.py (cobertura coverage for junit)

13. Implemented Test and Metrics plugins JCnnPlugin.py (javaccn) JDependPlugin.py (jdepend – dependency analyser) JPmdPlugin.py (pmd – source code quality) JFindbugsPlugin.py (findbugs – bug pattern analysis) JRat.py (java profiler) CFlawfinderPlugin.py (flawfinder - static analyser for C/C++) CPyPHPRatsPlugin.py (Rats – static analyser for C/C++, Perl, python, php) CValgrindPlugin.py (Valgrind – C profiler) 13

14. Metrics and their tool support in ETICS

15. Installation modules • Additionally to plugins so-called installation modules have been implemented • The aim of installation modules is to assist users in their installation and testing procedure by providing some general purpose installation modules, for example, tomcat, mysql or wscore. • The installation modules are components with appropriate test commands to install, start and stop the services on which other test components can set dependencies in order to get those services running before the test is executed.

16. Major Tasks Performed/Achievments Distributed testing facilities are in place Particularly, • distributed testing environment is in place and co-schedulig mechanism is implemented • metrics and testing tools have been identified • Test and Metrics Plugins Framework as part of ETICS client is in place • approximately 15 test and metrics plugins have been implemented • general purpose deployment scripts to assist testing have been implemented

17. Case study - Diligent system testing • Remote regression testing using the Diligent system testing environment as a private resource • grid7.4dsoft.hu/grid5.4dsoft.hu are attached to the etics pool • Role of regression tests: • Initially, to assure the testers that the test environment system is testable. Successfull running of regression tests means for testers that they can start working on the implementing new test cases. • Later on, the initial set of test cases have been augmented with the new test cases implemented by testers.

18. Case study - Diligent system testing • Applying Java static analysis plugins for core Diligent services Applied plugins: JFindbugsPlugin, JDependPlugin JPmdPlugin, JCcnPlugin • The principal aim here has been to model a system testing environment in ETICS for Diligent components where the java static analysis plugins can run easily in an automatic way on a periodic manner.

19. Future plan • As the test execution mechanism assuaring test process reproducibility is in place in ETICS the future work in ETICS2 is • to implement a modelling environment for creating advance test cases/ test workflows

20. Conclusion 20