Software Testing

Software Testing (Adapted from Chuck Cusack’s Notes) Based on material from • Dick Hamlet and Joe Maybee, The Engineering of Software, Addison Wesley Longman, inc., 2001 • Ian Sommerville, Software Engineering, 6th Edition, Pearson Education Limited, 2001 • Sebastian Elbaum, Software engineering lecture notes

The Software Life Cycle • Before we can discuss testing, we need to present the context of testing—the software life cycle. • There are various forms of the software life cycle.

The Life Cycle and Testing • Although testing is not explicitly mentioned at each stage of the lifecycle, every stage has some relationship to testing.

Software That Works • When developing software, it is important that • it work according to the specification, and • it not contain any errors. • Even perfect design and implementation rarely guarantees the first, and never the second. • The solution is verification and validation. • Validation: Are we building the right product? • Verification: Are we building the product right? • There are two techniques. • Software inspection(static) • Software testing(dynamic)

Software Inspection • Software inspection involves analyzing • Requirements • Design diagrams • Source code • As the name indicates, this is performed by looking at the various components (called a walkthrough), not by running any of the code. • Because final code is not required, inspection can be performed at all stages of the life cycle. • There is much more to inspection than this, but that is not the subject of these notes.

Software Testing • Software testing involves running the software with test data to determine whether or not the software performs as required. • There are a several important questions that need to be answered. • When do we develop test data? • What are good test data? • When should testing be done? • How should testing be done? • When should you stop? • These notes attempt to answer these questions.

Why Test: The Optimist • Optimist: The purpose of software testing is to verify that software meets the requirements. • The optimistic view has several problems: • It is virtually impossible to prove that software performs correctly for all possible inputs/situations. • It is in human nature to find what one is looking for. Thus, we can too easily be convinced that the software is correct, when there are (sometimes very subtle) errors.

Why Test: The Pessimist • Pessimist: The purpose of software testing is to find errors. • Although bleak, this is a much better definition. • There is still a slight problem. • What is an error? Errors: • Failure: When the software does something wrong (does not do as specified). • Fault: The error (bug, defect) in the software (the code) which causes a failure.

Why Test: The Realist • Realist:The purpose of software testing is to find failures, so the faults causing the failures can be found and fixed. • This view agrees with the pessimist, but is more precise. • Put another way: “Try to break the system.” • This is a better view than the optimist, since: • If we look for errors, we will probably find some. • When errors are found and fixed, we know that the software is one step closer to being correct.

The “Not”s of Testing • Debugging is not the same as testing. • Debug: Fix faults/remove failures. • Test: Find faults/discover failures. • Testing cannot show the absence of errors, only show the presence of them. • A system cannot (usually) be completely tested. • It cannot be assumed users will not make errors. In other words, users will make errors.

Test Cases and Test Suite • A test case is a set of input data and the expected output. • A test suite is a set of test cases. • The important questions are: • What makes a good test case? • How do you determine the expected output? • When does a test suite contain enough test cases? • We will attempt to answer these questions with a few suggestions, and several examples.

Choosing Test Cases • When choosing test cases, it is important to follow these principles: • Enough test cases should be chosen so that it is fairly certain that the software will work in all situations. • Both valid and invalid inputs should be tested. • Test cases on and near “boundaries” should be chosen. • The possible inputs should be divided into equivalence partitions, with “just enough” test cases from each equivalence partition. • If the system has “states,” sequences of test cases may be required.

Important Test Cases • Strings/arrays of various sizes, including 0 (empty), 1, 2, and larger. • Strings/arrays that are too long. • Strings containing spaces. • Empty input files. • The first, last, and middle element of an array/string. • Index into array/string that is too small/large. • Loops that execute 0 times. • For ADTs (linked-lists, sets, stacks, and queues), empty, partially full, and completely full (if applicable) data structures.

Equivalence Partitions • If a program breaks an array into parts, each part of the array should be treated as an equivalence partition, and the test cases for each partition should include boundaries. • If an input has restrictions, the partitions should correspond to the restrictions. • For instance, if the input should be between 0 and 10, the partitions include numbers less than 0, between 0 and 10, and greater than 10. • Test cases should include –1, 0, 1, 5, 9, 10, and 11, at a minimum.

Example: Binary Search • Recall the binary search algorithm: int BinarySearch(int []A,int val, int left,int right) { if(left<right) { int middle=(left+right)/2; if(A[middle] == val ) return middle else if (A[middle] > val) return BinarySearch(A,val,left,middle-1); else return BinarySearch(A,val,middle+1,right); } else return -1; }

Partitions for Binary Search • The algorithm partitions the array into 3 parts: • This suggests we break the test cases into 4 parts—One for each of the above parts, and one for a failed search. • For each partition, include the boundaries and some case in the middle of each partition (x and y above). • For a failed search, include values smaller and larger than all values, and one missing in the middle. • We should also include an array of size 1.

A Test Suite for Binary Search • One test suite for binary search might be:

Example: Loan Calculator • Specification: The system should allow the user to specify a principal amount, interest rate, and term, and compute the total amount of the loan. • The first questions that should come to mind are: • Can interest rate, principal amount, and/or term be zero? negative? • Should the interest rate be given as a percentage (5 for 5%) or decimal (.05 for 5%). • We will assume no, no, and decimal. • Given this, what test cases are appropriate?

Loan Calculator Test Cases • Test cases should include negative, zero, and positive values for all of the inputs. • Does this require 27 test cases? • If we do not know how the system is to be implemented, this may be the case, since it is possible that it handles one bad input, but not two. • We will assume it is clear from the design and/or code that if any one or more of the inputs is not positive, the same code is executed.

Loan Calculator Test Suite • The following set of test cases seems reasonable. • I have not included the amount (the output), but it should certainly be included in a real test suite. • Also, where amount says “Invalid”, the real result would depend on what the specification states.

Example: Stack ADT • The stack ADT (LIFO) is a data type that supports the following operations • bool Push(item X) Place X on the top of the list. • item Pop( ) Remove and return the top element. • item Peek( ) Return the top element. • When thinking of test cases, we should consider • an empty stack • a full stack, and • a partially full stack.

Testing A Stack • Since we cannot look inside the ADT, how do we test the operations? • To test Push, we need to use either Pop or Peek. • If there is a problem, what caused it? • To further complicate things, is a newly created empty stack the same as a stack that has had 3 elements pushed, and then popped?

A Stack Test Plan • We present a test suite, assuming a stack that holds a maximum of 3 elements. • The test cases in the following test plan should be tested in order on a single instance of a stack object.

Test Case Timeline • As the specification, design, and coding phases are in progress, test cases may come to mind. • Things are easily forgotten, so it is important to write these cases down, even though (in fact, especially since) testing may not occur for some time. • When the time for testing arrives, insights have been gathered from a variety of people during the entire life cycle, making it likely that many of the problems have been anticipated.

Testing: The Boxes • Black-box testing (functional, specification-based) • Based only on the specification and design process • E.g. stack ADT example • White-box testing (structural, program-based, code-based, systematic, clear-box, glass-box, broken-box) • Based on the actual code • E.g. binary search example

Black-Box Testing • As noted before, black-box testing is based on the specification, not the code. • If the specification is written properly, some test cases should be fairly easy to generate. • It can be difficult to determine test cases that are likely to fail. • It can also be difficult to determine equivalence partitions.

White-Box Testing • When implementing “tricky” code, you can develop test cases based on what might go wrong. • You can create test cases to be sure that every line of code is tested. • Finding “boundaries” can be much easier when you can look at the code. • Determining equivalence partitions can be easier. • If there are restrictions on the inputs, the code might give some indication of which invalid inputs might be most problematic.

Which Box? • There is no clear answer to this question. • White-box testing is not always possible, since the source code may not be available. • Code coverage (white-box) can be misleading, since it does not guarantee that separate program units operate correctly together. • White-box and black-box testing discover different types of faults. That is, they are complementary. • Thus, picking test cases using a combination of white- and black-box testing will often result in the best test suite.

Testing: Stages Testing can be separated into 5 stages: • Unit (function, component) • Module (ADT, class, group of functions) • Sub-system • System (The final product) • Acceptance (alpha) In addition, beta testing may be performed.

Unit Testing • Units (functions, components, etc.) are tested in isolation of any other code. • Unit testing can be helpful, since it gives some confidence to the correctness of the “building blocks.” • It is important to note that if all units are correct, it does not guarantee that the entire system will be correct, since the interactions between units might be incorrect.

Module Testing • As noted earlier, a module can be an abstract data type (ADT), an object class, or a collection of related functions. • As with unit testing, module testing can be performed independently of the other parts of the system. • For objects classes and ADTs, testing involves ensuring the object/ADT performs all operations correctly, in all possible states of the object/ADT. • Information-hiding can make testing ADTs and object classes difficult. • For instance to test push, one must use peek or pop. If there is an error, which function caused it?

Sub-system Testing • Tests collections of modules that are integrated into sub-systems. • The most common problems arise from interface mismatch, including: • Interface misuse (syntactic): e.g. wrong parameter list (wrong types, wrong order, wrong number). • Interface misunderstanding (semantic): e.g. passing an unsorted array when a sorted array is expected.

System Testing • Similar to sub-system testing. • System (and sub-system) integration can be performed using • Big-bang integration: Put it all together, and test it (waiting for the “big bang”). • Incremental integration: Put it together in steps, testing as each new sub-system or module is integrated. • These have obvious implications to system testing. • For incremental integration, there are two choices: bottom-up and top-down.

Top-down Integration • With top-down integration, the higher-level components are integrated and tested before the low-level components are integrated. • This requires stub code to be written that either does nothing, or simulates the low-level modules. • Examples: int square(int x) { // Just return a default value return 1; } // Use a pre-defined sorting algorithm for now void quickSort(int []array,int size) { SomeLibrary.Sort(array,size); }

Bottom-up Integration • With bottom-up integration, low-level components are tested before the high-level components are designed. • This requires a test driver (or test harness) to be written that runs the low-level components. • Example: int main() { for(int i=0;i<8;i++) { println(“The square of ” + i + “ is “ + square(i)); } }

Acceptance Testing • When the system has been integrated, it should be tested with real data supplied by the actual user(s) of the system. • This will increase the chance that the system will be error-free, since the actual data may include cases not considered by the software team. • It will also test whether or not the requirements were correctly understood, and if the system will meet the needs it was designed to meet.

Testing Summary • Testing good. • Not testing bad. • Planning for testing good. • Haphazard testing bad.

Software Testing