COVERAGE CRITERIA FOR TESTING

1. COVERAGE CRITERIA FOR TESTING

2. ill-defined terms in Testing • complete testing • exhaustivetesting • full coverage Arepoorly defined termsbecause of theoretical limitation of software. • The number of potentialinputs for most programs is so large as to be effectively infinite. • The only limitationis the size of the file that can be read by the parser. • the number ofinputs is effectively infinite

3. FormalCoverageCriteria • We cannot test with all input • Coverage criteria are used to decide which test inputs to use. • Effectiveuse of coverage criteria makes it more likely • Testengineers will find faults in a program • Thisprovides informal assurance Result : Highquality and Reliabilityof the Software

4. Advantage of CoverageCriteria Accordingtothepracticalperspective Coveragecriteriaproviderules for when to stop testing.

5. Integration testing : Test how modules interact with each other main Class P Class A Class B • System testing : Test the overall functionality of the system method mA1() method mB1() method mA2() method mB2() • Module testing (developer testing) : Test each class, file, module or component • Unit testing (developer testing) : Test each unit (method) individually Traditional Testing Levels • Acceptance testing : Is the software acceptable to the user? This view obscuresunderlying similarities

6. Object-Oriented Testing Levels • Inter-class testing : Test multiple classes together Class A Class B • Intra-class testing : Test an entire class as sequences of calls method mA1() method mB1() method mA2() method mB2() • Inter-method testing : Test pairs of methods in the same class • Intra-method testing : Test each method individually

7. Coverage Criteria • Even small programs have too many inputs to fully test them all • private static computeAverage (int A, int B, int C) • On a 32-bit machine, each variable has over 4 billion possible values • Over 80 octillion possible tests!! • Input space might as well be infinite • Testers search a huge input space • Trying to find the fewest inputs that will find the most problems • Coverage criteria give structured, practical ways to search the input space • Search the input space thoroughly • Not much overlap in the tests

8. Advantages of Coverage Criteria • Provide traceability from software artifacts to tests • Source, requirements, design models, … • Make regression testing easier • Gives testers a “stopping rule” … when testing is finished • Can be well supported with powerful tools

9. Test Requirements and Criteria • Test Criterion : A collection of rules and a process that define test requirements • Cover every statement • Cover every functional requirement • Test Requirements : Specific things that must be satisfied or covered during testing • Each statement is a test requirement • Each functional requirement is a test requirement Testing researchers have defined dozens of criteria, but they are all really just a few criteria on four types of structures … Graphs Logic expressions Input domains Syntax descriptions

10. GraphCoverageCriteria Structural Coverage Criteria : Defined on a graph just in terms of nodes and edges Data Flow Coverage Criteria : Requires a graph to be annotated with references to variables

11. Coverage Criterion A coverage criterion is simply a recipe for generating test requirements in a systematic way: Coverage Criterion: A coverage criterion is a rule or collectionof rules that impose test requirements on a test set.

12. CoverageCriterion A coverage criterion is a rule or collectionof rules that impose test requirements on a test set. That is, the criterion describes the test requirements in a complete and unambiguousmanner. The “flavor criterion” yields a simple strategy for selecting jelly beans. In this case, the set of test requirements, TR, can be formally written out as TR = {flavor = Lemon, flavor = Pistachio, flavor = Cantaloupe, flavor = Pear, flavor = Tangerine, flavor = Apricot}

13. Coverage Given a set of test requirements TRfor a coveragecriterionC, a test set T satisfies C if and only if for every test requirement tr in TR, at least one test t in T exists such that t satisfies tr .

14. CoverageLevel Given a set of test requirements TR and a test set T, the coverage level is simply the ratio of the number of test requirements satisfied by T to the size of TR.

15. Graphs Logic Input Space Syntax Applied to Applied to Appliedto FSMs Source DNF Specs Source Models Source Specs Integ Input Design Use cases Graph Coverage Four Structures for Modeling Software

16. 1 0 2 Structural Coverage Criteria -I Node Coverage (Formal Definition): For each noden ∈ reachG(N0), TR containsthepredicate “visit n.” Node Coverage (NC): TR contains each reachable node in G. • Node Coverage (NC) (Standard Definition) Test set T satisfiesnode coverage on graph Gif and only if for every syntactically reachable node n in N, there is some path p in path(T) such that p visits n. • TR = { 0, 1, 2 } • path(t1) = [0, 1 , 2] T1={t1} • T1 satisfiesnodecoverage on thegraph

17. 1 0 2 StructuralCoverageCriteria - II Edge coverage is slightly stronger than node coverage • Edge Coverage (EC) : TR contains each reachable path of length up to 1, inclusive, in G. • The “length up to 1” allows for graphs with one node and no edges Edge Coverage : TR = { (0,1), (0, 2), (1, 2) } path (t1) = [ 0,1,2] path (t2 ) = [ 0, 2 ] T2={t1, t2} Test Paths = [ 0, 1, 2 ] [ 0, 2 ] T2satisfiesedgecoverage on thegraph

18. 0 0 1 Paths of Length 1 and 0 A graph with only one node will not have any edges • It may be boring, but formally, Edge Coverage needs to require Node Coverage on this graph EdgeCoverage needs to require Node Coverage on this graph Otherwise, Edge Coverage will not subsume Node Coverage So we define “length up to 1” instead of simply “length 1” We have the same issue with graphs that only have one edge for Edge Pair Coverage …

19. 0 3 1 4 5 2 6 StructuralCoverageExample Node Coverage TR = { 0, 1, 2, 3, 4, 5, 6 } Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ] Edge Coverage TR = { (0,1), (0,2), (1,2), (2,3), (2,4), (3,6), (4,5), (4,6), (5,4) } Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 2, 4, 5, 4, 6 ] path (t1)= [ 0, 1, 2, 3, 6 ] path (t2)=[ 0, 2, 4, 5, 4, 6 ] T={t1 ,t2}