Software Testing

1. Software Testing Reference: Software Engineering, Ian Sommerville, 6th edition, Chapter 20

2. Topics Covered • The testing process • Defect testing • Black box testing • Equivalence partitions • White box testing • Equivalence partitions • Path testing • Integration testing • Top-down, bottom-up, sandwich, thread • Interface testing • Stress testing • Object-oriented testing

3. Testing Goal • The goal of testing is to discover defects in programs. • A successful test is a test that causes a program to behave in an anomalous way. • Tests show the presence, not the absence, of defects. • Test planning should be continuous throughout the software development process. • Note that testing is the only validation technique for non-functional requirements.

4. The V-model of Testing

5. The Testing Process • Component (unit) testing • Testing of individual program components (e.g., functions, methods, or classes) • Usually the responsibility of the component developer (except sometimes for critical systems) • Integration and system testing • Testing of groups of components integrated to create a sub-system or entire system • Usually the responsibility of an independent testing team • Tests are based on system specification • Acceptance testing • Run in presence of customer or by customer • Used to validate all system requirements

6. The Testing Process (con’t) • Test data - Inputs that have been devised to conduct the particular test • Expected output – Recorded before test is conducted • Actual output – The output actually received • Pass/fail criteria – Criteria to determine if test passed or failed when the actual results are compared to the expected results. Determined before test is conducted.

7. Black Box Testing • An approach to testing where the program is considered as a “black box” (i.e., one cannot “see” inside of it) • The program test cases are based on the system specification, not the internal workings (e.g., algorithms), of the program. • Use equivalence partitions when conducting black box testing

8. Equivalence Partitioning • Input data and output results often fall into different classes where all members of a class are related. Examples: • Positive (or negative) numbers • Strings with (or without) blanks • Each of these classes is an equivalence partition where the program behaves in an equivalent way for each class member. • Test cases should be chosen from each partition, especially at the boundaries.

9. Equivalence Partitions Example • System accepts 4 to 10 inputs which are 5-digit integers greater than 10,000 • Partition system inputs into groups (partitions) that should cause equivalent behavior. Include both valid and invalid inputs. • If input is a 5-digit integer between 10,000 and 99,999, equivalence partitions are: < 10,000 10,000 - 99,999 > 99,999 • Choose test cases at the boundaries of these partitions: • 9,999 10,000 99,999 100,000

10. Equivalence Partitions Example

11. Derivation of Test CasesSome Practice procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : in out BOOLEAN; L: in out ELEM_INDEX) ; Pre-condition -- the array has at least one element Post-condition -- the element is found and is referenced by L or -- the element is not in the array

12. What Tests Should We Use? • Equivalence Partitions

13. Search Routine Test Cases

14. White Box Testing • Sometimes called structural testing • Derivation of test cases according to program structure (can “see” inside) • Objective is to exercise all program statements at least once • Usually applied to relatively small program units such as functions or class methods

15. Binary Search Routine • Assume that we now know that the search routine is a binary search. • Any new tests?

16. Binary Search Test Cases

17. Path Testing • The objective of path testing is to ensure that the set of test cases is such that each path through the program is executed at least once. • The starting point for path testing is a program flow graph.

18. Binary search flow graph

19. Independent Paths • 1, 2, 8, 9 • 1, 2, 3, 8, 9 • 1, 2, 3, 4, 6, 7, 2 • 1, 2, 3, 4, 5, 7, 2 • Test cases should be derived so that all of these paths are executed. • A dynamic program analyser may be used to check that paths have been executed (e.g., LINT).

20. Cyclomatic Complexity • The minimum number of tests needed to test all statements equals the cyclomatic complexity. • CC = number_edges – number_nodes + 2 • In the case of no goto’s, CC = number_decisions + 1 • Although all paths are executed, all combinations of paths are not executed. • Some paths may be impossible to test.

21. Integration Testing • Tests the complete system or subsystems composed of integrated components • Integration testing is black box testing with tests derived from the requirements and design specifications. • Main difficulty is localizing errors • Incremental integration testing reduces this problem.

22. Incremental Integration Testing

23. Approaches to Integration Testing • Top-down testing • Start with high-level system and integrate from the top-down, replacing individual components by stubs where appropriate • Bottom-up testing • Integrate individual components in levels until the complete system is created • In practice, most integration testing involves a combination of both of these strategies: • Sandwich testing (outside-in) • Thread testing

24. Top-down Testing

25. Bottom-up Testing

26. Method “Pro’s” • Top-down • Bottom-up • Sandwich • Thread

27. Interface Testing • Interface misuse • A calling component calls another component and makes an error in the use of its interface. • parameters in the wrong order • parameter(s) of the wrong type • incorrect number of parameters • Interface misunderstanding • A calling component embeds assumptions about the behavior of the called component that are incorrect. • binary search function called with an unordered array • wrong flag “number” • other preconditions that are violated

28. Interface Types • Parameter interfaces • Data passed from one procedure to another • functions • Shared memory interfaces • Block of memory is shared between procedures • global data • Message passing interfaces • Sub-systems request services from other sub-systems • OO systems • client-server systems

29. Some Interface Testing Guidelines • Design tests so that parameters to a called procedure are at the extreme ends of their ranges (boundaries). • Always test pointer parameters with null pointers. • Design tests that cause the component to fail. • violate preconditions • In shared memory systems, vary the order in which components are activated.

30. Stress Testing • Exercises the system beyond its maximum design load. • Exceed string lengths • Store/manipulate more data than in specification • Load system with more users than in specification • Stressing the system often causes defects to come to light. • Systems should not fail catastrophically. Stress testing checks for unacceptable loss of service or data.

31. Object-oriented Testing • The components to be tested are classes that are instantiated as objects. • No obvious “top” or “bottom” to the system for top-down or bottom-up integration and testing. • Levels: • Testing class methods • Testing the class as a whole • Testing clusters of cooperating classes • Testing the complete OO system

32. Class Testing • Complete test coverage of a class involves • Testing all operations associated with an object • Setting and interrogating all object attributes • Exercising the object in all possible states; i.e., all events that cause a state (attribute(s)) change in the object should be tested • Inheritance makes it more difficult to design class tests as the information to be tested is not localized.

33. Object Integration • Levels of integration are less distinct in object-oriented systems. • Cluster testing is concerned with integrating and testing clusters of cooperating objects. • Identify clusters using knowledge of the operation of objects and the system features that are implemented by these clusters

34. Approaches to Cluster Testing • Use case or scenario testing • Testing is based on user (actor) interactions with the system • Has the advantage that it tests system features as experienced by users • Thread testing • Tests the system’s response to events as processing threads through the system • a button click

35. Scenario-based Testing • Identify scenarios from use cases and supplement these with sequence diagrams that show the objects involved in the scenario

36. Sample Sequence Diagram