Coverage Criteria

1. Coverage Criteria Drawn mostly from Ammann&Offutt and Pezze&Yooung

2. IEEE definition of V&V • Validation: The process of evaluating software at the end of software development to ensure compliance with intended usage • Verification: The process of determining whether the products of a given phase of the software development process fulfill the requirements established during the previous phase

3. Static vs dynamic testing • Static testing : Testing without executing the program • Include software inspections and some forms of analyses • Very effective at finding certain kinds of problems – especially “potential” faults, that is, problems that could lead to faults when the program is modified • Dynamic testing : Testing by executing the program with real inputs

4. Testing vs debugging • Testing : Finding inputs that cause the software to fail • Debugging : The process of finding a fault given a failure

5. A test case • Test case values: The values that directly satisfy one test requirement • Expected results: The result that will be produced when executing the test if the program satisfies it intended behavior

6. Top-down vs bottom up testing • Top-down testing : Test the main procedure, then go down through procedures it calls, and so on • Bottom-up testing : Test the leaves in the tree (procedures that make no calls), and move up to the root. • Each procedure is not tested until all of its children have been tested

7. Test requirement • Test requirements: Specific things (software artifacts) that must be satisfied or covered during testing • Non-software example: test a bag of jelly beans • Come up with ways to test • Suppose the following flavors: Lemon (yellow), pistachio (green), cantaloupe (orange), pear (white), tangerine (orange), apricot (yellow) • One requirement: test each flavor (six test requirements)

8. Test criterion • Test criterion: A collection of rules and a process that define test requirements Flavor criterion: TR = {flavor=lemon, flavor=pistachio, flavor=cantaloupe, flavor=pear, flavor=tangerine, flavor=apricot}

9. Test coverage • Given a set of test requirements TR for coverage criterion C, a test set T satisfies C coverage if and only if for every test requirement tr in TR, there is at least one test t in T such that t satisfies tr • A test case with 12 beans: 3 lemons, 1 pistachio, 2 cantaloupe, 1 pear, 1 tangerine, 4 apricot • OK to satisfy a test requirement with more than one test

10. Coverage level • Given a set of test requirements TR and a test set T, the coverage level is simply the ratio of the number of test requirements satisfied by T to the size of TR • Why? Sometime test requirements may be infeasible • Example: suppose tangerine jelly beans are rare and some bags may not contain any • Flavor criteria cannot be 100% satisfied • Maximum coverage level: 5/6 or 83%

11. Criteria vs subsumption • Criteria subsumption : A test criterion C1 subsumes C2 if and only if every set of test cases that satisfies criterion C1 also satisfies C2 • Must be true for every set of test cases • Example: color criteria for the jelly bean: {yellow, green, orange, white} • If we satisfy flavor criteria, we’ll satisfy color criteria • Example : If a test set has covered every branch in a program (satisfied the branch criterion), then the test set is guaranteed to also have covered every statement

12. Coverage is not size • Coverage does not depend on the number of test cases • T0 , T1 : T1 >coverage T0 T1 <cardinality T0 • T1 , T2 : T2 =coverage T1 T2 >cardinality T1 • Small test cases make failure diagnosis easier • A failing test case in T2 gives more information for fault localization than a failing test case in T1

13. Question 1 • Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose that test set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P. • Does T1 necessarily satisfy C2? Explain.

14. Question 2 • Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose that test set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P. • Does T1 necessarily satisfy C2? Explain. Yes. This follows directly from the definition of subsumption.

15. Question 2 • Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose that test set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P. • Does T2 necessarily satisfy C1? Explain.

16. Question 2 • Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose that test set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P. • Does T2 necessarily satisfy C1? Explain. • No. There is no reason to expect test requirements generated by C1 to be satisfied by T2.

17. Question 3 • Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose that test set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P. • If P contains a fault, and T2 reveals the fault, T1 does not necessarily also reveal the fault. Explain.

18. Question 3 • Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose that test set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P. • If P contains a fault, and T2 reveals the fault, T1 does not necessarily also reveal the fault. Explain. No. This is the hard question. Testers often think that test sets for strong criteria are at least as good at finding faults as test sets for weaker criteria. But subsumption is about criteria, not about test sets. In particular, there is no requirement that test set T2 be a subset of test set T1. So, it could happen that T2 contains that one test that reveals the fault, and T1 doesn't.

19. Statements vs branches • Traversing all edges of a graph causes all nodes to be visited • So test suites that satisfy the branch adequacy criterion for a program P also satisfy the statement adequacy criterion for the same program • The converse is not true: A statement-adequate (or node-adequate) test suite may not be branch-adequate (edge-adequate)

20. “All branches” can still miss conditions • Sample fault: missing operator (negation) digit_high == -1 || digit_low == -1 • Branch adequacy criterion can be satisfied by varying only digit_low • The faulty sub-expression might never determine the result • We might never really test the faulty condition, even though we tested both outcomes of the branch

21. Condition testing • Branch coverage exposes faults in how a computation has been decomposed into cases • Intuitively attractive: check the programmer’s case analysis • But only roughly: groups cases with the same outcome • Condition coverage considers case analysis in more detail • Also individual conditions in a compound Boolean expression • e.g., both parts of digit_high == 1 || digit_low == -1

22. Basic conditions vs branches • Basic condition adequacy criterion can be satisfied without satisfying branch coverage • Branch and basic condition are not comparable (neither implies the other)

23. Covering branches and conditions • Branch and condition adequacy: cover all conditions and all decisions • Compound condition adequacy • Cover all possible evaluations of compound conditions • Cover all branches of a decision tree

24. Compound conditions: Exponential complexity (((a || b) && c) || d) && e Test a b c d e Case (1) T — T — T (2) F T T — T (3) T — F T T (4) F T F T T (5) F F — T T (6) T — T — F (7) F T T — F (8) T — F T F (9) F T F T F (10) F F — T F (11) T — F F — (12) F T F F — (13) F F — F — Short-circuit evaluation often reduces this to a more manageable number, but not always

25. Multiple (compound) condition coverage • The multiple condition coverage of T is computed as Cc/(Ce -Ci) , where Cc is the number of combinations covered, Ci is the number of infeasible simple combinations, and Ce is the total number of combinations in the program • Potentially a large number of test cases

26. Modified condition/decision (MC/DC) • Motivation: Effectively test importantcombinations of conditions, without exponential blowup in test suite size • “Important” combinations means: Each basic condition shown to independently affect the outcome of each decision • Compound condition as a whole evaluates to true for one and false for the other

27. MC/DC: linear complexity • N+1 test cases for N basic conditions (((a || b) && c) || d) && e Test a b c d e outcome Case (1) true -- true -- true true (2) false true true -- true true (3) true -- false true true true (6) true -- true -- false false (11) true -- falsefalse -- false (13) falsefalse -- false -- false • Underlined values independently affect the output of the decision • Required by the RTCA/DO-178B standard

28. Comments on MC/DC • MC/DC is • Basic condition coverage (C) • Branch coverage (DC) • Plus one additional condition (M): every condition must independently affect the decision’s output • It is subsumed by compound conditions and subsumes all other criteria discussed so far • Stronger than statement and branch coverage • A good balance of thoroughness and test size (and therefore widely used) • Federal Aviation Administration’s requirement that test suites be MC/DC adequate

29. Paths? (Beyond individual branches) • Should we explore sequences of branches (paths) in the control flow? • Many more paths than branches • A pragmatic compromise will be needed

30. Path adequacy • Decision and condition adequacy criteria consider individual program decisions • Path testing focuses consider combinations of decisions along paths • Adequacy criterion: each path must be executed at least once • Coverage: # executed paths # paths (c) 2007 Mauro Pezzè & Michal Young

31. Practical path coverage criteria • The number of paths in a program with loops is unbounded • the simple criterion is usually impossible to satisfy • For a feasible criterion: Partition infinite set of paths into a finite number of classes • Useful criteria can be obtained by limiting • the number of traversals of loops • the length of the paths to be traversed • the dependencies among selected paths (c) 2007 Mauro Pezzè & Michal Young

32. Cyclomatic adequacy • Cyclomatic number: number of independent paths in the CFG • If e = #edges, n = #nodes, c = #connected components of a graph, it is: • e - n + c for an arbitrary graph • e - n + 2 for a CFG • Cyclomatic coverage counts the number of independent paths that have been exercised, relative to cyclomatic complexity (c) 2007 Mauro Pezzè & Michal Young