Avoiding Coincidental Correctness in Boundary Value Analysis – by R.M. Hierons

1. Avoiding Coincidental Correctness in Boundary Value Analysis – by R.M. Hierons • Two popular Black Box Testing techniques are: • Partitioned analysis (PA) • Boundary Value analysis (BVA) • In PA, the potential input domain is subdivided into a finite set of sub-domains, S1, S2, ----, Sn such that, according to the specifications, the behavior of the system within each sub-domain of inputs, Si, will be uniform. Good for testing if a data input x from Si really performs the specified function, fi. - All inputs from domain Si will produce the same behavior. - So we can just use one sample input from each sub-domain to cut down on test cases. * S1 * S3 * S2 * S4 * is a sample input test data

2. PA Problem • Taking small sample from each of the specified sub-domains, S1, S2, - - - , Sn may not be able to detect potential problem caused by an incorrect implementation of a boundary among sub-domains. S2 S2 S1 S1 * S4 * S3 ** S3 ** S4 Input Domain boundaries as defined by the code implementation. (note the boundary shift between S3 and S4) Input Domain boundaries as defined by the Specification. The input test cases, as represented by * and ** will not be able to detect the “shift” in the boundary caused by possibly a coding error.

3. Boundary Value Analysis (BVA) may help • With BVA, input test cases close to the boundaries of the sub-domains are developed to gauge the system behavior close to the boundaries, with an aim to find boundary “shift.” Thus , if the boundary is wrong in the actual code, some input test case close to the boundary will have the wrong functionality applied and thus detect the shift. BVA is good for input domain test. • For picture below specifies that f1, f2, and f3 are applied to inputs in sub-domains S1, S2, S3. - Note that the input test case, *, in S3 should have functionality, f3, applied to it according to the specification. - But with the Implementation shift of the boundary, * will have functionality, f2, applied to it. - Thus with test case, *, close to the boundary, the shift can be detected. * S3 S1 S2 boundary shift

4. BVA Problem with Coincidental Correctness • With BVA’s goal of detecting errors at the boundary, there may still be coincidental correctness that will affect the test result. • Consider the case where the specification states that: • Function F1 is to be applied to input data, d1, in S1 • Function F2 is to be applied to input data, d2, on the boundary • Function F3 is to be applied to input data, d3, in S2. Even though the boundary has shifted considerably, the test data d1, d2, and d3 will still have the correct F1, F2, and F3 applied to it !!! Thus the error will not be detected. S1 d1 d3 d2 S2 Shift in boundary Original boundary

5. Scope of Coincidental Correctness • From the pictorial example, one can see that coincidental correctness can occur. • Choice of input test data: • Assume we have 2 adjacent sub-domains, S1 and S2 and two functions f1 and f2 applied to the input test data in each domain. Further assume that there is a shift and the implemented sub-domains are A1 and A2. • If we pick a test data, x, in S1 and also in A2. • But the test data x will not show an error if f1(x) = f2(x). Def.-1: For boundary value analysis (BVA), coincidental correctness occurs for input data x in Sj and in Ak, but fj (x) = fk (x). Thus we should look beyond the “geometrical” approach in picking the input data, x, but also need to consider the functions expected in each sub-domain.

6. Example 1: for Definition 1 • Assume that the Specification for charging customers water and electricity bill is as follows: • Let w = units of water consumed • Let e = units of electricity consumed • Let c1 = $/ unit of water consumed • Let c2 = $/unit of electricity consumed • The monthly billing formula is Bill = (c1 * w) + (c2 * e), except if the customer uses at least b units of water, then the customer gets a 20%discount on the electricity portion of the bill • So we can set up 2 sub-domains: • S2 = {(w, e) l w >=b and e>=0 } with f2(w, e) = (c1* w) + (.8* c2 * e) • S1 = {(w, e) l 0=< w < b and e >=0} with f1(w, e) = (c1 *w) + (c2* e)

7. Example 1 (continued) • Suppose in the specification b = 30 (for discount) water units, but in the implementation the coder’s design had the following (error): w >=40? Bill = (c1*w) + (.8* c2* e) Bill = (c1*w) + (c2* e) Suppose we choose our test data, x = (w, e) = (30, 0) because b=30 is specified boundary for our sub-domains S1 and S2. Then we get: f1(30, 0) = (c1 * 30) + (0 * e) = c1 * 30 f2(30, 0) = (c1 * 30) + (.8 * 0 * e) = c1 * 30 Def 1 stated that if we choose x such that f1 (x) = f2 (x) coincidental correctness will happen and we can not detect the “implementation error”. Note that all input data x where x = (z, 0), regardless of what z is, will give have f1(x) = f2 (x).

8. “Distinguishing” Input Test • To avoid the Example 1 situation where coincidental correctness occurred due to “poor” choice of input test data without considering the functions, the following definition is needed. Def-2 : Let Sj and Sk be adjacent sub-domains. Then a test case input, x, is a distinguishing testcase for Sj and Sk iff x is in Si and fi(x) ≠ fk (x) In Example 1, the test data x= (z,0) did not satisfy this definition of Distinguishing test case. Thus the error was not detected. Def-3 : Let Sj and Sk be adjacent sub-domains, inputs x be in Sj and x’ be in Sk. Then a test case input set, <x, x’>, is distinguishing for ( Sj ,Sk ) if x is a distinguishing input test for (Sj,Sk) and x’ is a distinguishing input test for (Sj, Sk).

9. For Deterministic Systems,Use “Distinguishing” Test Input • When applying BVA to detect error in the boundary of adjacent sub-domains, Sj and Sk, produce a pair of test input <x, x’> such that: • x is in Sj • x’ is in Sk and • <x, x’> is distinguishing for (Sj , Sk) • , x and x’ are close together (If possible, choose one of the <x, x’> to be on the boundary of Sj and Sk)

10. Example 2: for Distinguishing Input Test <x, x’> • Consider the specification of Example 1 for billing water and electricity. • S2 = {(w, e) l w >=b and e>=0 } with f2(w, e) = (c1* w) + (.8* c2 * e) • S1 = {(w, e) l 0=< w < b and e >=0} with f1(w, e) = (c1 *w) + (c2* e) • Generate a input test pair <x, x’> that is distinguishing as follows: • Let x = (w, e) and x’ = (w’, e’) • Since the boundary of S1 and S2 is at w=b, we can set e = e’ • Set x and x’ to be ε >0 apart. So we have w’ = w + ε , and w’ > w. • This gives us x = (w, e) and x’ = (w+ ε, e), and w <b and w’ >= b. • To satisfy Def-3, we need f1(x)≠ f2(x) and f1(x’)≠ f2(x’). Thus we must have : f1(w,e) = c1*w + c2*e ≠ c1*w + .8c2*e = f2(w,e) and f1(w’,e) = c1*w’ + c2*e ≠ c1*w’ + .8c2*e = f2(w’,e) hold. • The above will be true if e ≠ 0. • Thus pick pair, < x, x’>, such that x = ( b – ε, 5) and x’ = (b, 5), where b is the specified boundary. All input pairs <x, x’> of the form x= (b- ε, e) and x’ = (b, e) where e ≠ 0 will be distinguishing test cases for this example.

11. Implementing Real numbers with Floating Pt. • Discusses the fact that when we implement real numbers, we need to approximate it with floating point data type, through a function, t, truncation and mapping. • Define: x, y are real numbers, then x ≡ y if their converted floating point t(x) = t(y). • Program implementation correctness relative to the specification is defined as follows: • For an input x, let yp = fp(x) and ys = fs(x) to mean function output from executing program and program output as specified by specification. • Then the implementation program is considered “correct” if for input x, yp = t(ys) Now define “Strongly” part of the distinguishing test case definition. Def : Let Sj and Sk be adjacent sub-domains. Then a test case input, x, is a strongly distinguishing testcase for Sj and Sk iff x is in Si and fi(x) ≡ fk(x), for floating point approximation of real numbers.

12. Non-Deterministic Specification • When the specification is non-deterministic, there can be a boundary error detection problem. • Consider the following Example. • A system that takes a floating point input value z and returns a floating point value y according to the following rules: • If z >= 0 then return value = {y l 0 =< y < 100 } • If z < 0 the return value = { y l -100 < y =< 0 } • There are two sub- domains S1 = {z l z >=0} and S2 = {z l z<0} with f1 = { y l 0 =< y < 100} and f2 = {y l -100 < y =< 0 } • While for an input test z, f1(z) ≠ f2(z) in general, f1(z) ∩ f2(z) ≠ Ø because f1(z) = 0 = f2(z) can occur for some input test z. • In this case, we may not know whether f1 or f2 was really applied since an output 0 can come from either f1 or f2. A shift in boundary will not be detected.

13. Choosing Test Input for Non-Deterministic Case • Suppose we are using an input test x in Sj to check the boundary between specified sub-domains Sj and Sk. If we test x with an implementation, then the following may be the cases: • Fail: the output is not allowed by the specification, assuming there is no computation fault. • Pass: the output is one allowed by the specification, assuming no computation fault. • Inconclusive: the output is from fj(x) ∩ fk(x) and fj(x) ∩ fk(x) is non-empty . Def-5: Let Sj and Sk be adjacent sub-domains and x be in Sj. The (non-deterministic) specification gives “functions” fj and fk on the respective sub-domains. Then x is a distinguishing test input for (Sj, Sk) if fj(x) ∩ fk(x) = Ø

14. Heuristics or Guide for searching test input x • Suppose Sj and Sk are adjacent sub-domains. Further assume that there is no x in Sj that is close to the boundary and that it also satisfies fj(x) ∩ fk(x) = Ø. So we have no distinguishing test input for the specification. A possible heuristic goes as follows: 1. We are looking for x in Sj to check the boundary with Sk. This x will produce a “pass” result if the result is in fj(x) but not in fk(x), or it is in [ fj(x) \ fk(x) ]. 2. The above x will produce a “fail” verdict if it is in [ fk(x) \ fj(x) ]. 3. Assuming that all out put result from [ fj(x) U fk(x) ] is equally likely, then what the should be aiming at is to maximize: { l [ fj(x) \ fk(x) ] l + l [ fk(x) \ fj(x) ] l } / l [ fj(x) U fk(x) ] l

15. Example of Adaptive Heuristics • Consider the case where we have a specification which states that given a real number, z, display the execution time to display z. And the following rule is to be used for the value of execution time: • If z >= 0 execution time = { t l min+ < t < max+ } • If z < 0 execution time = { t l min- < t < max- } • And that min+ < min- < max+ < max- (note the overlap) Let’s look at this case: the boundary between S1 and S2 is at z = 0. So, we would test with small test input number, z, close to 0. Let that be a positive input z = ε, and the resulting execution time be denoted as tε. Further assume min- < tε < max+ , the overlapped area. Let value F(ε) = min { tε – min- , max+ - tε }. Note that when F(ε) becomes negative, then tε is in the “conclusive” area. So, the problem is to find the ε >0 that will minimize F(ε) and fall outside of the interval {min- , max+ }. z >= 0 Area between max+ and max- , z < 0. Area between min+ and min- , z >= 0. z < 0 Min+ Min- Max+ Max-

16. Heuristic Guide to Finding “distinguishing”Input Test • For either determinist or non-deterministic cases, in testing the boundary between two sub-domains S1 and S2, use the input x, where f1(x) ≠ f2(x). • If when input x gets close to the boundary, f1(x) ∩ f2(x) ≠ Ø, then we need to use heuristic and repeatedly adapt the input test case x.

17. Summary • BVA Test cases generated from only geometric basis may not be good enough. • When testing for boundary between sub-domains Si and Sk, pick an input test x such that fi (x) ≠ fk (x) • In case of non-deterministic specification, which is similar to the real number example, it is possible for fi (x) ≠ fk (x) but fi (x) ∩ fk (x) ≠ Ø. The we can have three possibilities: • Pass • Fail • Inconclusive • We would want to choose test input cases that would not lead us to inconclusive verdict. • If we can find such an input test case, then we may have to search through adaptive heuristics