Software Testing

1. Software Testing Chapter 19

2. Get Out the Test Plan • At the time testing is ready to begin, almost all of our testing decisions and information should be located in our test plan, which has been growing throughout the project. • The test plan shows our decisions and approaches for the project based on the requirements, specification and design.

3. What Should Testing Accomplish? • Testing does not and never will prove that the system is defect free! • Instead, testing increases our confidence that we built software that behaves as specified. • Many testing techniques are closely associated with statistics that indicate a level of confidence.

4. Unit Testing vs. System Testing • Unit testing focuses on individual “units” of the system. • So…what is a unit? A unit can be essentially what ever we want: a subsystem, a group of subsystems, a “module” (compilation unit, or file), or even a specific segment of code. • System testing looks at the behavior of the entire system as if it were “monolithic” .

5. Unit testing • Unit testing focuses on the inputs/outputs of a unit. • Specifically, do we get the outputs we expect from a unit for a given set of inputs. • Testers will also look at trying to “fool” the unit by trying obtuse combinations of inputs. • Also, the tester will make sure that the unit handles errors (erroneous inputs) the way it is supposed to!

6. Test harness and stubs • Test harnesses are systems built to test systems. That is, harnesses are computer programs that supply inputs to another program, then read the outputs and evaluate the response. • Sometimes, testing requires us to replace existing routines with ‘stubs’ to capture the output of a particular unit (or system) before it goes to an output device.

7. Subsystem testing • Subsystem testing almost always requires “instrumentation”, such as a harness, to supply inputs and ‘intercept’ outputs. • Subsystem testing on process control systems can be somewhat difficult: calls to mechanical control subsystems must be intercepted to prevent damage (or catastrophe). How do we test reactor control software without activating a reactor?

8. Testing Abstract Data Types • ADTs have the potential for us to define environments in which the ADT can be tested in isolation. • This proves useful for unit testing, but does not provide for system or subsystem testing. In those cases, testers usually resort to the usual approaches, such as harnesses.

9. Testing object-oriented code • Likewise, object-oriented code has the ability to have objects tested in isolation. • In many instances, objects can (and should) have test methods (or “member functions”) built into the object. • Subsystem and system testing, again, usually is performed with the assistance of a harness.

10. Sequences of inputs to subsystems • Often, inputs must be “sequenced” to achieve a desired state. When testing an “array” object that can dynamically expand it’s capacity, testers must force the capacity expansion to test it. This requires a series of inputs to be transacted before the actual test ‘begins’. • It is not unusual for defects (bugs) to manifest themselves only after a particular set of inputs has been supplied to the system/subsystem/unit. It is important to document what happened to the system before the failure!

11. System testing • System testing can often be thought of in the same manner as unit testing, with the system as a single unit. • System testing is sensitive to the same issues that unit testing are: input sequences and irreversible actions.

12. Inspection vs. testing • ‘Cleanroom’ engineering suggests that with sufficient inspection and engineering diligence, testing becomes a redundant almost needless activity. • Experiments in very closely controlled environments have indicated that there is very much to be gained from inspection, but it will not, in my opinion, be a satisfactory substitute for testing.