Regression Testing

1. Department of Computer Science Regression Testing

2. Regression testing • Testing changed software • Retest all • Selective regression testing • Regression test suite • Types of changes • Defect fixes • Enhancements • Adaptations • Perfective maintenance

3. Retest all • Assumption: • Changes could have introduced errors anywhere in the code • Expensive, prohibitive for large systems • Reuse existing test suite • Add new tests as needed • Remove obsolete tests (discrepancies between expected and actual output)

4. Selective regression testing • Impact analysis • Only code impacted by change needs to be retested • Select tests that exercise such code • Add new tests if needed • Remove obsolete tests

5. Regeneration strategy • Often for defect fixes when unsure whether defect is really fixed • Develop new tests • Probe around defect fix • Test affected functionality • Comes in retest all, and selective regression testing flavors

6. Regression testing process • Identify changes • Determine which tests will remain valid: T’ • Test modified software with T’ • If T’ does not meet test criterion, generate new test cases T’’ • Execute modified software with T’’

7. Regression testing process (2) • T’: • test whether modifications have broken anything • Determined either by retest all, or selective strategy • T’’: • test whether modified code works • Use scope appropriate criteria

8. Classification of tests in T • Reusable: • does not test software modification • Should produce same result • Need not be rerun • Retestable: • Tests software modification • Must be rerun • Obsolete: no longer applies

9. Practical regression testing

10. Approaches • Full new test cycle • Redesign entire test plan • Rebuild whole test suite • Minimum regression test • Reuse test plan as much as possible • Rerun minimum number of retestable test cases • Generate minimum number of new test cases for changes

11. Approaches (2) • Expanded scope of regression testing • Reuse part of test plan • Rerun all retestable test cases • Generate new test cases on the full scope of changes • Full reuse of existing test suite • Reuse test plan and test suite

12. Attributes of regression tests • Safe: • All tests that could possibly exhibit different outputs on the modified software • Is 100% inclusive • Precision: • Percentage of tests in a reduced test suite that cannot reveal regression faults, and • Are not selected for the reduced test suite

13. Attributes of regression tests (2) • Efficiency: cost of identifying a reduced regression test suite • Generality: range of application for the selection strategy

14. Regression test patterns • Retest changed code • Class, cluster, subsystem scope • Identify segment coverage by test case • Use version control tool to identify changed segments • Select tests covering changed/deleted segments • Develop new tests for new segments • Safe

15. Retest by profile • Assumes operational profile • Unsafe • Include critical use cases • Include other use cases by frequency in operational profile • No impact analysis • Better: determine impact on use case

16. Retest risky use cases • Suspicious use cases • Use cases that depend on components, objects, middleware, resources that are • Unstable, unproven • Have not been shown to work together before • Implement complex business rules • Are complex • Were fault-prone during development • Critical use cases • Unsafe

17. Retest within Firewall • Firewall: set of components whose test cases will be included in a regression test • External interface, contract change • Internal implementation change