Validating and Improving Test-Case Effectiveness

1. Validating and Improving Test-Case Effectiveness Author: Yuri Chernak Presenter: Lam, Man Tat

2. Agenda • Some definition that need to be understand • How verification and validation is done • Overview of how in process validation of test cases is done • Case-study • Conclusion

3. Functional testing • By focusing on a system's functionality • Looking for as many defects as possible • Test plan documents and test case specifications are important deliverables

4. Test Cases • Critical for effective testing • Testers use test-case specifications does not guarantee that systems are sufficiently tested

5. Verification • First verify test case specifications at the end of the test design phase • Reviews or inspections – evaluate test case specifications for their correctness and completeness • Test cases that passed verification could have weak failure-detecting ability

6. Validation • Can process as soon as testers have executed all test cases • Determines whether the test cases were sufficiently effective in finding defects • If test cases effectiveness has not proved satisfactory, it is not too late to analyze the reasons and correct the test process • Test case effectiveness metric introduced

7. Test case effectiveness metric • Mainly use for online client server systems • Certain number of defects are always found as a side effect • Assume that the more defects test cases find, the more effective they are • Define as the ratio of defects found by test cases (Ntc) to the total number of defects (Ntot) reportes during the test cycle

8. Test case effectiveness metric Cont’ • TCE = Ntc / Ntot *100% • TCE metric evaluates test cases from the test cycle perspective, which provides in process feedback to the project team on how well a test suite has worked for testers • Compare the actual TCE for a given project with a baseline value (such as previous successful project, author experience suggests 75 %)

9. Test case effectiveness metric Cont’ • If the TCE value is at baseline or above, we can conclude that the test cases have been sufficiently effective in test cycle • If the TCE value falls below the baseline, the higher is the risk of user dissatisfaction

10. Improving test case effectiveness • If in process validation finds test case effectiveness to be less than acceptable, the project team should analyze the cause and identify areas that need to be improve • Author’s proposed improvement framework was based on IBM’s “Test case escape”

11. IBM’s “Test case escape” • Define as “product defects that a particular test failed to find, but which were found in a later test, or by a customer [in production]

12. Author’s proposed improvement framework • Define as software defects that a given suite of test cases failed to find but that were found as a side effect in the same test cycle • Rather than compare the Ntot with the whole process

13. Author’s proposed improvement framework • 1- Understand and document the test process used by the project team • 2- Make assumptions about factors that affect test cases • 3- Gather defect data • 4- Identify main factors • 5- Implement corrective actions

14. Understand, document the test process • Test planning – definition of the scope, objectives, and approach to testing • Test design – the design of the test cases • Test preparation and execution – preparation of the test environment executing test cases and finding defects • Test evaluation and improvement – analyzing the results of testing

15. Make assumption • Once understands and documents the test process, the project team should analyze each phase and identify factors that can affect test case effectiveness • Lets look at some example

16. Example of making assumption • Test planning – if the functional specifications do not completely define functional features, the test plan document will not be complete either. So, the test cases will not completely cover a system’s functionality

17. Example of making assumption 2 • Test design – a common example could be lack of negative test cases (such as abnormal conditions by using either invalid data input or the wrong user action • Test design – test case specifications could be simply incorrect

18. Factors affecting test case effectiveness

19. Gather defect data and perform analysis • Defect tracking system • Able to identify which defects were found as result of test cases and which were found as a side effect • Once selected, defects should be classified according to one of the factors based on the analysis

20. Test Case escape classification logic

21. Gather defect data and perform analysis cont’ • Analysis is used to evaluate each test case and understand why the test suite missed the corresponding defect during the test execution • Example: A lack of negative test cases in is a common cause of missed defects. Can you see which category it will be under?

22. Identify the main factors • At this point, all test case escapes have been classified according to their respective causes • Build a “Pareto chart” which displays frequency bars in descending order, convenient for analyzing (will be an example later)

23. Implement corrective actions • After identification of the main causes of test case escapes, the project team should implement corrective actions and repeat the test execution

24. Example of corrective actions • Rework functonal specifications and test case specification • Use a traceability matrix to ensure complete coverage of business rules • Use checklists to design test case specification • Implement training of testers on test execution

25. Case study • Banking application intended for external clients • Project team consisted 10 developers and 3 testers • 183 defects reported • 71 found by test case escapes • 112 found by test case execution

26. Case study cont’ • TCE = 61% • Lower than the acceptable level of 75% • Project team performed the test process improvement according to the framework • Pareto chart created for analysis

27. Example of Pareto chart

28. Case study cont’ • Analysis of the distribution showed incomplete test design and incomplete functional specification to be the main factors causing missed defects. • Project team correcting and completing the test design and functional specification

29. Case study cont’ • As a result, 48 additional defects found before releasing the product • By the end of the second month the number of production, only 23 defects was found • Indicated sufficient effectiveness of test progress, user should be satisfied with the system quality

30. Conclusion • This technique intended to give project teams better visibility into test process effectiveness before their system are released into production • Help evaluate each test case and understand why the test suite missed the corresponding defect during test execution

31. Conclusion • Help testers revise and improve the test suite • In other word, it helps testers find additional defects and deliver a better software product

32. If my presentation did not put you into sleep and you have a question… Now its time!!!