Flowgraphs and Path Testing

1. Flowgraphs and Path Testing Presented By Dr. ShazzadHosain Asst. Prof., EECS, NSU

2. Path Testing Strategies • Path Testing (P∞) • Execute all possible paths i.e. 100% path coverage • Impossible to achieve, strongest criteria • Statement Testing (P1) • Execute all statements at least once • 100% statement or node coverage, denoted as C1 • Weakest criteria • Branch Testing (P2) • 100% branch or link coverage, denoted as C2 There are infinite number of strategies P1, P2, … P∞ But even that’s insufficient to exhaust testing C1 and C2 are the minimum requirement

3. Which paths to select? • Pick up paths to achieve C1 + C2 • Start at the beginning and take the most obvious path (1, 3, 4, 5, 6, 2) or (abcde) • Take the next most obvious path, abhkgde • Take a simple loop, preferably with previous path, such as abhlibcde • Then take another loop, abcdfjgde • Finally, abcdfmibcde

4. Which paths to select? • Continue selecting/choosing paths you appear to have achieved C1 + C2 • We say, appear, because some of the path you selected might not be achievable. • As you trace the paths, create a table as the following • Check the following • Does every decision have a YES and a NO in its column? (C2) • Has every case of all case statements been marked? (C2) • Is every three-way branch (less, equal, greater) covered? (C2) • Is every link (process) covered at least once? (C1)

5. Predicates, Path Predicates and Achievable Paths • Selecting path does not mean achievable path • If all decision variables are independent to processing and of one another • Then 2n paths and all are achievable • But reality is not so. There are dependent paths • Every selected paths leads to an associated Boolean expression, called path predicate expression.

6. Predicates • The logical function evaluated at a decision is called predicate. Example • A is greater than zero • The fifth character has numerical value of 31 • X + Y = 3Z2 – 44 • A predicate associated with a path is called path predicate

7. Inputs • Input • all data object i.e. number, string, values left in registers etc. that are referenced by the routine whose values are fixed prior to entering it • Input vector • Set of inputs to the routine as if it is a one-dimensional array

8. Predicate Interpretation • The simplest predicate depends on input variables e.g. X1 + X2 >= 0 • Assume the predicate: X1 + Y >= 0 • And along the path we had Y := X2 + 7 and nothing else affected Y on that path • Although process dependent, equivalent expression will be: X1 + X2 + 7 >= 0 • The act of substitution, solely in terms of input vector is called predicate interpretation.

9. Predicate Interpretation • The interpretation may depend on path • The predicate interpretation at HEN (the IF stmt) depends on path i.e. • IF Y + 7 > 0 GOTO …. • IF Y – 7 > 0 GOTO …. • IF Y > 0 GOTO

10. Path Predicate • Path predicates are specific form of the predicates of the decisions along the selected path after interpretation. • We assume, unless stated otherwise, all predicates have been interpreted

11. Independence and Correlation of Variables and Predicates • The path predicate take on truth values based on input variables, either directly or indirectly (interpretation required) • Process independent variable • Process dependent variable • Process independent predicate • Process dependent predicate

12. Independence and Correlation of Variables and Predicates • Process dependent variable does not mean process dependent predicate, Example • X + Y = 10 • Process increments X by 1 and decrements Y by 1 • The predicate is process independent

13. Correlated Predicates • Decision 1: X = Y • Decision 2: X + Y = 8 • Let we like to take path YN • Choose X = Y = 4 • Correlated predicates • Every path through a routine is achievable only if all predicates are uncorrelated

14. Path Predicate Expressions • A conceptual exercise • Aid to understanding testing issues • Not the way you design test cases • Select and entry/exit path • Write down the uniterpreted predicates • Interpret predicates only for input variables • The result is a set of Boolean expressions • This set is called path predicate expression • Any set of input variable that satisfies the above expression will force the path to take. • Otherwise the path is not achievable.

15. Path Predicate Expressions • More complex when OR condition • IF X5 > 0 OR X6 < 0 THEN ….. • Simplified notation • If opposite branch at fourth predicate, then • Resulting predicate would be

16. Compound Predicate • Compound predicate example • A .OR. B • A .AND. B • Negation can produce compound predicate • X = 17, the opposite is X .NE. 17 • Equivalent to • X > 17 .OR. X < 17

17. Predicate Coverage • A+B+C+D, any true will stop other evaluation • We do not know the compiler evaluation order • Let’s evaluate A+B, where A is correct, B is buggy • Choose test case A to be true, execute the branch for coverage, but bug remains • A+B+C, you try with B, evaluates false, but executes for C, even nastier. B’s failure is masked by B • C2 could still hide control flow bugs • Stronger notion, predicate coverage • Mean, all possible combination of truth values

18. Testing Blindness • SAM predicate (X) • Has two alternatives • Two tests would be sufficient • But SAM is reached Via BOB or COB • So four tests are necessary • Because of testing blindness • Testing blindness • Desired path is achieved for wrong reason • Interaction of two or more statements makes the buggy predicate work • Three kinds: assignment, equality and self-blindness

19. Assignment Blindness • Occurs when buggy predicate appears to work correctly because of specific value chosen for an assignment statement work both for correct and buggy predicate • If Y = 1 for the test case • Both works • But there is still a bug • Some other path leading to the same predicate might have different assignment for X • So wrong path would be taken

20. Equality Blindness • Occurs when the path selected by a prior predicate results in a value that works both for correct and buggy version. • For any positive value of X • The first predicate forces the rest of the path

21. Self-Blindness • Occurs when a buggy predicate is a multiple of the correct predicate, thus indistinguishable along the path. • The direction taken for both path is same • A path with another assignment could have reveal the bug

22. Review: Achievable and Unachievable paths • Want select enough paths for C1 and/or C2 • Draw control flowgraph, select a set of path • Any path of that set, interpret the predicates • Trace the path, multiply the individual compound predicates • Find boolean expressions like • The terms in parenthesis are compound predicate • Each letter (A, B, …) stands for simple predicate • Multiply out the expression to find sum-of-products forms • Each product term denotes a set of inequalities • If any one is solved, you found a set of input values for the path • In that case it is achievable, otherwise not • The act of finding a set of solutions from path predicate expression is called path sensitization.

23. Path Sensitization • There is no general algorithm for sensitization • In practice do you really want to do all these? • There are heuristics

24. Simple, Independent, Uncorrelated Predicates • Uppercase letter represents decision • Capital letters in link represents true/false • Four uncorrelated and independent predicates • So, 24 = 16 possible values • But number of possible paths is far less (8) • The number of covering paths is smaller still.

25. Simple, Independent, Uncorrelated Predicates • Every link has been covered, every decision two ways • But two changes on the second path • Instead use few more but simpler paths • You can determine input values easily

26. Correlated, Independent Predicates • The following two decisions are correlated • Path abdeg and acdfg seem to provide coverage, but unachievable, why? • Two decisions, four potential paths, but two are there • In general, correlated predicates mean, some paths are not achievable.

27. Correlated, Independent Predicates • Reproduce the common code • Let, you selected paths from flowgraph, and find a path unachievable, though seems meaningful case, the following must be true: • You found a bug • The design can be simplified by removing some decisions • You have better understanding of how the decisions are interrelated.

28. Correlated, Independent Predicates • Correlation appears because of urge of “saving code” • If the second predicate is based on a flag that was set based the previous predicate • Then the correlation is not obvious • If two pieces are widely separated, then it is subtle and obscure

29. Path Instrumentation • Test design procedure • Predict the outcome of test • Run the test, observe actual outcome • Compare actual outcome with expected one • If both matched, we say test passed • But outcome may match for wrong reason • Known as coincidental correctness.

30. Path Instrumentation • For X = 16, all outcome will be 2 • Path instrumentation is what we have to do to confirm that the outcome was achieved by the intended path.

31. Path Instrumentation: Link Markers • Name every link by a lowercase letter. • Instrument the links for interpretive trace program so that link’s name is recorded when executed • The succession of letters show the path e.g. abcde Single link marker

32. Path Instrumentation: Link Markers • Single link marker is not enough • Links can be chewed open by bugs • You want ikm path, but GOTO statement will take different path • If coincidental correctness is there, then bug is not detected Single link marker

33. Path Instrumentation: Link Markers Implement double link marker Double link marker Link counters, less informative. Instead of labeling link, count the Number of link traversed. Other methods are there too.

34. Path Instrumentation • For unit testing, path instrumentation requires comprehensive test tool • In reality you may not find of your chosen language • It is more important at higher level of program structure, say transaction flow testing, rather than unit testing

35. Application of Path Testing • Path testing mainly used in unit testing • Consider the process of component integration Component 1 { process.. return 20,00 } Stub 1 { return 20,00 } Component under test Stub1 Stub2 • Integration process • Test Component with stubs, test interfaces • Replace stubs by components, retest the integrated • The component is ready for next level integration Component 1 Component 2

36. Application of Path Testing What does it mean to achieve C1 or C2 coverage at this level when lower–level or corequisite components could affect our control flow? Component under test • Predicate interpretation could require us to treat the subroutine as if it were in-line code • Will add complexity and processing the problem • Sensitization obviously becomes more difficult • A selected path may be unachievable • Assumption is: we can do effective testing one level at a time without being overly concerned about lower levels. Component 1 Component 2

37. Application of Path Testing • Although few hard statistics • We typically loose 15% coverage with each level • We achieve C2 at current level, but 85% one level down, 70% two levels down and so on. • When all testing done, C1 coverage at system level remains some 50% - 85%

38. Testability Tips • Testable software has • Fewer bugs • Easier to test • Easier to debug • Some Tips • Keep in mind three numbers • The total number of paths • Total number of achievable paths • Number of paths required to achieve C2 coverage • The closer these numbers are each other, the more testable the routine is

39. Why? • Few unachievable paths means • Less sensitizing problems • Fewer sensitizing dead ends • Fewer total paths means • Reduced opportunities for blindness and coincidental correctness

40. Testability Tips • Make your decision once, only once, and stick to them – no correlated decisions. • Design goal: n decisions means 2n paths – all achievable, all needed for C2 • Don’t squeeze the code • If you can’t test it, don’t build it • If you don’t test it, rip it out • Introduce no extras, unwanted generalizations, additional functionality • It requires more cases to cover

41. Testability Tips • If you can’t sensitize a path you need for coverage • Then you do not know what you’re doing • Easy cover beats elegance every time • Deeply nested and/or horrible loops aren’t a mark of genius but of a murky mind • Don’t squeeze the code. • Don’t squeeze the code! • DON’T SQUEEZE THE CODE!

42. Summary • Path testing based on structure is a powerful unit testing tool • With suitable interpretation, it can be used for system functional testing. • The objective of path testing is to achieve C1+C2 • Select simple normal path. Then add one more. • Cover extreme cases for loops: no looping, once, twice, one less than maximum, the maximum • Use instrumentation and tools to verify the path and to monitor coverage.

43. References • Chapter 3, Software Testing Techniques - Boris Beizer