Software Testing

Software Testing Sudipto Ghosh CS 406 Fall 99 November 9, 1999

Learning objectives • Functional testing • Equivalence class partitioning • Boundary value analysis • Structural testing • Control flow-based CS 406 Testing

Testing for correctness • Identify the input domain of P. • Execute P against each element of the input domain. • For each execution of P, check if P generates the correct output as per its specification S. CS 406 Testing

Input domain • The set of all valid inputs that a program P can expect is the input domain of P. • The size of an input domain is the number of elements in it. • An input domain could be finite or infinite. • Finite input domains may be large! CS 406 Testing

Identifying input domains • For the sort program: • N: size of the sequence, N < 3 • Sequence of N integers, such that for any integer K, 0 K (e –1), e = 3 • K: each element in the sequence • Some sequences are: • [ ] : An empty sequence (N = 0) • [0] : A sequence of size 1 (N = 1) • [2 1] : A sequence of size 2 (N = 2) CS 406 Testing

Size of input domain • Suppose that 0 N 106 • The size of the input domain is the number of all sequences of size 0, 1, 2… • The size is computed as: • To test for correctness, P needs to be tested on all inputs. • How many years would that take? CS 406 Testing

Exhaustive testing • Previous example • P is executed on all elements in the input domain. • For most programs, exhaustive testing is not feasible • What is the alternative? CS 406 Testing

Partition testing • Input domain is partitioned into a finite number of sub-domains. • P is then executed on a few elements in each sub-domain. • Four sub-domains for our example: [], [2], [2 0], [2 1 0] Reduced from 85 to 4 ! N=0 1 N=2 3 N=3 4 N=1 2 CS 406 Testing

Confidence in your program • Confidence is a measure of one’s belief in the correctness of the program • Correctness is not measured in binary terms: a correct or incorrect program • Measured as a probability of correct operation of a program when used in various scenarios • Reliability • Test completeness: Extent to which a program has been tested and errors have been found and removed CS 406 Testing

Types of module testing • “Informal testing” performed by programmer while developing module • Methodical testing performed by SQA group • Nonexecution-based testing • Execution-based testing • Haphazard data as input • Systematically constructed test cases • Testing to specifications • Testing to code CS 406 Testing

Testing to specifications • Black-box testing • Data-driven testing • Input/output-driven testing • Functional testing • Code is ignored, i.e. the internal structure of the code is not important • The specification document is the only information used for creating test cases CS 406 Testing

Testing to code • White-box testing • Glass-box testing • Logic-driven testing • Path-oriented testing • Specifications are not used to generate the test cases • Test “to code,” i.e. the internal structure of the code is important CS 406 Testing

What is functional testing? • Functional testing tests how well a program meets the functionality requirements. • When test inputs are generated using program specifications, we say that we are doing functional testing. CS 406 Testing

Feasibility of functional testing • Problem with exhaustive testing • Very large (possibly infinite) number of test cases. • Need to devise: • Small, manageable set of test cases. • Maximize the chances of detecting faults. • Minimize the chances of wasting test cases by having more than one test case detecting the same fault. CS 406 Testing

Equivalence class partitioning • Divide the domain of all inputs into a set of equivalence classes. • If any test in an equivalence class succeeds, then every test in that class will succeed. • How would you get ideal equivalence classes? • Difficult without looking at the internal structure • Difficult even with the internal structure CS 406 Testing

Equivalence class partitioning Specifications Set of all inputs Put those inputs together for which the behavior pattern of the module is specified to be different into similar groups Equivalence classes CS 406 Testing

Rationale behind equivalence class partitioning We assume that if the specifications require exactly the same behavior from each element in a class of values, then the program is likely to be constructed so that it either succeeds or fails for each of the values in that class. CS 406 Testing

Guidelines for partitioning • For robust software, we must test for incorrect inputs too. • For each equivalence class of valid inputs, we have equivalence classes of invalid inputs. • Input condition specifies a range a  X  b. • a  X  b (Valid case) • X < a, and X > b (invalid cases) • Input specifies a value • create one for the valid value • create two for incorrect (one above, one below) CS 406 Testing

Guidelines for partitioning • Input specifies a value (contd.) • For boolean, only one value • Input condition specifies a member of a set • Create one for the valid value • Create one for invalid value (not in the set) • Example: • Factorial (n), where n  0 • Valid classes {0}, {x | x  1} • Invalid class {x | x < 0} CS 406 Testing

Non-overlapping partitions • In the previous example, the equivalence classes were non-overlapping, i.e., the sub-domains were disjoint. • It is sufficient to pick one test from each equivalence class to test the program. • An equivalence class is considered covered if at least one test has been selected from it. • Our goal is to cover all equivalence classes. CS 406 Testing

Overlapping partitions • Suppose a program P takes three integers, X, Y and Z. It is known that: • X < Y • Z > Y X < Y X < Y Z > Y X  Y Z  Y Z > Y CS 406 Testing

Overlapping partition:test selection • Select 4 test sets as • X=4, Y=7, Z=1 (satisfies X < Y) • X=4, Y=2, Z=1 (satisfies X  Y) • X=1, Y=7, Z=9 (satisfies Z > Y) • X=1, Y=7, Z=2 (satisfies Z  Y) • Can also reduce the number of test cases to 2 • X=4, Y=7, Z=1 (satisfies X < Y and Z  Y) • X=4, Y=2, Z=3 (satisfies X  Y and Z > Y) CS 406 Testing

Boundary Value Analysis (BVA) • Observation: • Programs that work correctly for a set of values in an equivalence class, fail on some special values. • These values lie on the boundary of the equivalence class. • Choose an input from a test case from an equivalence class such that the input lies at the edge of the equivalence class. • These test cases are “extreme cases” CS 406 Testing

BVA examples • Suppose the range is 0.0  X  1.0. • 0.0 (valid input) • 1.0 (valid input) • -0.1 (invalid input) • 1.1 (invalid input) • For a list • first and last element of the list • A program takes a string S and integer X as input such that • a  X  b, and length(S)  100. Derive tests. CS 406 Testing

BVA analysis • A program takes two integers X and Y, such that a  X  b, c  Y  d. • How many test cases do we get? d c a b CS 406 Testing

Output variables • Equivalence class partitioning can be applied to output variables • BVA also can be applied to output data CS 406 Testing

Testing functions • Identify each function implemented in a module • Devise test data to test each function separately • Module may contain a hierarchy of lower level functions connected by program control structures • Perform functional testing recursively • Higher level: black-box • Lower level: glass-box CS 406 Testing

Problems with previous approach • Usually higher level functions are not as structured • Lower level functions are intertwined • Functionality does not coincide with module boundaries • Distinction between module testing and integration testing is blurred • Testing one module is not possible without simultaneously testing other modules whose functionality is used CS 406 Testing

Structural testing • Intent is to exercise the different programming structures and data structures used in the program • Intent is not to exercise all the different input and output conditions • though this may be a by-product • Achieve test cases that force the desired “coverage” of different structures • Criteria are formal and precise CS 406 Testing

Control-flow based criteria • Statement coverage • Run a series of test cases and ensure that every statement is executed at least once • Simplest form of glass box testing • What is the weakness? Consider the example: int abs (int x) { if (x >= 0) x = 0 - x; return x; } Test inputs: x = 5: What is coverage? x = 0: What is coverage? Error CS 406 Testing

Statement coverage • Not very strong, may leave errors undetected. • Examples: • if statement without else part • conjunctions of predicates • In all these cases, all branches were probably not exercised. • Can you think of a better criterion based on the above observation? CS 406 Testing

Branch coverage • Require that every decision is evaluated to true and false values at least once during testing • Branch coverage implies statement coverage • Each statement is part of some branch CS 406 Testing

Control flow graph of a program • Let G be the graph of a program P. • Node: • Represents a block of statements that are always executed together • Edge (i, j) from node i to node j: • Represents a possible transfer of control after executing the last statement of the block represented by node i to the first statement in the block represented by node j. CS 406 Testing

Control flow graph of a program • Start node: • Node corresponding to a block whose first statement is the start statement of P. • Exit node: • Node corresponding to a block whose last statement is an exit statement of P. • Path: • Finite sequence of nodes (n1, n2, …, nk), k>1 such that there is an edge (ni, ni+1) for all nodes ni in the sequence, except the last node nk. CS 406 Testing

Control flow graph of a program • Complete Path: • Path whose first node is a start node and the last node is an exit node. • All-nodes criterion (statement coverage) • All-edges criterion (branch coverage) CS 406 Testing

An example (xy) • scanf(x, y); if(y < 0) • pow = 0 – y; • else pow = y; • z = 1.0; • while(pow != 0) • { z = z * x; pow = pow – 1; } • if ( y < 0 ) • z = 1.0/z; • printf(z); CS 406 Testing

Control Flow Graph of example 1 3 2 4 5 6 7 9 8 CS 406 Testing

Problems with branch coverage • What if a decision has many conditions (using and, or) • Decision may evaluate to true or false without actually exercising all the conditions int check (int x) { if ((x >= 5) && (x <= 200)) return TRUE; return FALSE; } Test inputs: x = 5: x = -5: Error (should be 100) CS 406 Testing

Solution? • Require all individual conditions to evaluate to true and false • Problem: • Even if individual conditions evaluate to true and false, the decision may not get both true and false values • Solution: • Require both decision / condition coverage!! • Still there is a problem. CS 406 Testing

Path testing • Some errors are related to some combinations of branches. • Presence revealed by an execution of a path that includes those branches. • Solution: • Require all possible paths in the CFG to be executed during testing. • Path-coverage criterion, or all-paths criterion • Path coverage implies branch coverage CS 406 Testing

Software Testing