Test Suite Reduction for Regression Testing of Simple Interactions between Two Software Modules

1. Test Suite Reduction for Regression Testing of Simple Interactions between Two Software Modules Dmitry Kichigin

2. Software regression integration testing Software integration testing – testing of interactions between software elements being integrated Software regression testing – retesting of software to verify that modifications have not caused unintended effects and that the software still complies with its specified requirements Problem: high cost of regression testing Solutions: • Automation of testing • Test suite reduction

3. Test suite reduction “ seeks to reduce the number of test cases in a test suite while retaining a high percentage of the original suite’s fault detection effectiveness “ This reduces: • The costs associated with test suite maintenance • The costs of software execution on the tests Result: reduction of overall costs associated with regression testing

4. Traditional approach Involves a static analysis and/or instrumentation of a source code and consists of 3 main steps: • Instrumentation of software source code • Execution of software on a test suite and collecting coverage information • Using coverage information to reducing the test suite Existing methods: - Interface Mutation (Mutation Analysis) - Structural methods: • Control-flow based • Data-flow based

5. Drawbacks Recent trends in software development may limit their applicability: • Component-based software development and use of commercial off-the-shelf (COTS) components: • the source code is unavailable • Increasing size and complexity of the software: • source-code based coverage analysis is expensive and hard to conduct for large software

6. Proposed solution Test Suite Reduction Method for Regression Testing of Simple Interactions between Two Software Modules: • Based on interactions behaviour monitoring • Uses sequences of interface function calls to analyse interactions behaviour • Takes into account function parameters • Does not require access or instrumentation of a source code

7. Definitions … FILE* fp = fopen(…); for (int i=0; i<3; i++) { fread(…); } fclose(fp); … … fopen fread fread fread fclose … Simple interaction- interaction via interface of functions which can take only scalar parameters Interface functions are the functions, which form a part of module’s programming interface. Interaction trace of interaction between module A and module B is a sequence of interface functions of module B, which are invoked by module A during software execution on a test case t.

8. Algorithm t := next(T) Mt := build_model(t) yes Mt empty? no ? Mt  MT’ MtMT’ MT’ := MT’Mt T’ := T’ t MtMT’ no ? T ==  yes finish T – original test suite T’ – reduced test suite, T’T t – next test case from the original test suite, tT Mt– model of interaction behaviour on the test case t MT’– set of models of interaction behaviour on test cases from T’ Main steps: • Pick the next test case from the original test suite and execute the program • Build modules interactions model Mt • If MtMT’, then add Mt to MT’and add test case t to T’ • Run until T is exhausted

9. Interaction Behaviour Model fopen fwrite fwrite fopen fwrite fclose fwrite fclose fopen fwrite fwrite fopen fwrite fwrite fopen fwrite fwrite fopen fwrite fclose fopen fwrite fclose fwrite fwritefclosefwritefclose K-length sequenceof interface functions is an arbitrary continuous sequence of length K, which can be located in an interaction trace. Model of interaction between module A and module B on a test t - set of all possible K-length sequences, which can be located in the interaction trace for the interaction between A and B on t. Use “sliding window” technique to build the model:

10. Equality relation between models Two given models are equal <=> their sequence sets are equal Two sequences of functions are equal <=> their elements are equal pairwise Function calls are equal when: • the functions have the same names, • the values of nominal parameters are the same, and • the values of numerical parameters belong to the same intervals.

11. Equality relation between modelswith taking into account parameter values Function scalar parameters Nominal Numerical Nominal: • definition area is a finite set • no order relation and distance defined • equality relation: (x,y),where - Kronecker symbol Numerical: • definition area is R • both order relation and distance are defined • equality relation:

12. Results of the experiment:level of test suit reduction Subject programs: GNU Assembler & the C standard library

13. Results of the experiment:fault detection ability Subject programs: GNU Assembler & the C standard library

14. Summary The new test suite reduction method was presented, which: • does not require access or instrumentation of a source code • uses sequences of interface function calls as behaviour characteristic • takes into account values of function parameters The experimental evaluation demonstrated: • can successfully reduce the size of a test suite while retaining its fault detection ability • the method outperforms the random test suite reduction method

15. Thank you

16. Questions ?