Outline

1. Types of errors Component Testing Unit testing Integration testing Testing Strategy Design Patterns & Testing System testing Function testing Structure Testing Performance testing Outline

2. How do we deal with Errors & Faults?

3. Verification?

4. Modular Redundancy?

5. Declaring the Bug a Feature?

6. Patching?

7. Testing?

8. Examples of Faults • Interface specification Faults • Expectation mismatch between the client & the server Cannot happen with Jini • Mismatch between requirements & implementation • Algorithmic Faults • Mechanical Faults • Documentation does not match actual conditions • Timing errors • Throughput / performance errors

9. Some Observations • It is impossible to completely test any nontrivial module • Theoretical limitations: Halting problem • Practical limitations: Prohibitive in time & cost • Testing only shows the presence of bugs, not their absence

10. Testing Requires Creativity • Is testing dirty work? • An effective test requires: • Detailed understanding of the system • Knowledge of test techniques • Skill to apply these techniques in an effective & efficient manner • Book says: Testing is done best by independent testers • XP says: developers devise/do unit tests

11. Testing Activities Requirements Analysis Document Code Requirements Analysis Document Unit System Design Document T est Tested Subsystem User Manual Code Unit T est Integration Tested Subsystem Functional Test Test Functioning System Integrated Subsystems Tested Subsystem Code Unit T est All tests by developer

12. Testing Activities … Client’s Understanding of Requirements User Environment Global Requirements Accepted System Validated System Functioning System Performance Acceptance Installation Test Test Test Tests by developer Tests by client

13. Fault Handling Techniques Fault Handling Fault Avoidance Fault Detection Fault Tolerance Design Methodology Reviews Atomic Transactions Modular Redundancy Verification Configuration Management Testing Integration Testing System Testing Component Testing Correctness Debugging Performance Debugging

14. Component Testing • Unit Testing • Developers confirm that class functions correctly. • Integration Testing • Developers confirm that subsystem interface functions correctly

15. System Testing • System Testing • Developers confirm that the system meets its requirements • Acceptance Testing • Clients confirm that the system meets its requirements

16. Unit Testing • Informal • Incremental coding • Static Analysis • Hand execution: Read the source code • Walk-Through: Informal presentation to others • Code Inspection: Formal presentation to others • Testing • Black-box testing: Test the input/output behavior • White-box testing: Test the internal logic of the class

17. Black-box Testing • I/O behavior • If for any input, we can predict the output, the module passes the test. • Generally intractable to test all possible inputs • Reduce number of tests by equivalence partitioning • Partition the inputs into equivalence classes • Devise a test case for each equivalence class. • Example: If an object is supposed to accept a negative number, include a test of 1 negative number.

18. Black-box Testing (Continued) • Selection of equivalence classes • No rules, only guidelines • Input is valid across range of values. Select 4 test cases from 3 equivalence classes: • Just below the lower boundary • Just above the lower boundary • Just below the upper boundary • Just above the upper boundary • Input is valid if it is from a discrete set. Select test cases from 2 equivalence classes: • Valid discrete value • Invalid discrete value

19. White-box Testing • Thoroughness (Coverage) • Every statement in the component is executed at least once. • 4 types of white-box testing • Statement Testing • Loop Testing • Path Testing • Branch Testing

20. White-box Testing … • Statement Testing • Test single statements • Loop Testing • Execution of the loop is skipped. • Exception: Do-While loops • Loop is executed exactly once • Loop is executed more than once • Path testing (implies conditional testing) • Make sure all paths in the program are executed

21. public int sumPositives( int[] a ) { int total = 0; for ( int i = 0; i < a.length; i++ ) if ( a[i] > 0 ) total += a[i]; return total; } t1 = sumPositives( null ); t2 = sumPositives( { 0 } ); t3 = sumPositives( { 1 } ); t5 = sumPositives( { 1, 2, 3 } ); t6 = sumPositives( { 2, 0, 0 } ); // ? White-box Testing …

22. Comparison of White & Black-box Testing • Black-box tests are devised when: • Functional requirement is defined • Class is defined • White-box tests are devised when class is implemented. • Create a test harness: A program that executes the tests. • First examination of test results is by eye. • Test Oracle: Save output of tests in a file. • Automate regression testing.

23. Bottom-up Testing Strategy • Subsystem classes in the lowest layer of the call hierarchy are tested individually • The next subsystems are tested that call the previously tested subsystems • Do until all subsystems are included in the testing • Test driver A routine that passes a test case to a subsystem.

24. A Layer I D C B Layer II G F E Layer III Bottom-up Integration Test E Test B, E, F Test F Test A, B, C, D, E, F, G Test C Test D,G Test G

25. Bottom-Up Integration Testing • Useful for integrating: • Object-oriented systems • Systems with strict performance requirements • Real-time systems

26. Top-down Testing Strategy • Test the top layer subsystem first • Then test that layer + all subsystems called by that layer. • Do until all subsystems are incorporated into the test • A test stubis needed A program or a method that simulates a missing method or subsystem by returning fake data.

27. A Layer I D C B Layer II G F E Layer III Top-down Integration Testing Test A, B, C, D, E, F, G Test A, B, C, D Test A Layer I Layer I + II All Layers

28. Sandwich Testing Strategy • Combines top-down strategy with bottom-up strategy • The system is viewed as having 3 layers • A target layer in the middle • A layer above the target • A layer below the target • The target layer is tested.

29. 1. Unit test all the classes in the selected component. 2. Dofunctional testing: Define test cases that exercise all uses cases with the selected component 3. Execute performance tests Repeat steps 1 to 3 until the full system is tested. Keep recordsof the test cases.

30. System Testing • Functional Testing • Structure Testing • Performance Testing • Acceptance Testing • Installation Testing Impact of requirements on system testing: • The more explicit the requirements, the easier they are to test. • Quality of use cases determines the ease of functional testing • Quality of subsystem decomposition determines the ease of structure testing • Quality of nonfunctional requirements & constraints determines the ease of performance tests

31. Structure Testing • Essentially the same as white box testing. • Cover all paths in the system design • Exercise all input / output parameters of each component. • Exercise all components & methods • Use conditional & iteration testing as in unit testing.

32. Functional Testing . Essentially the same as black box testing • Test functionality of system • Test cases are designed from the RAD, center around requirements & use cases • The system is treated as black box. .

33. Stress Testing Stress limits of system (maximum # of users, data, peak demands, extended operation) Configuration testing Test the various software and hardware configurations Security testing Try to violate security requirements Timing testing Evaluate response times and time to perform a function Environmental test Test tolerances for heat, humidity, motion, portability Recovery testing Tests system’s response to presence of errors or loss of data. Performance Testing

34. Testing has its own Life Cycle (?) Establish the test objectives Design the test cases Write the test cases Test the test cases Execute the tests Evaluate the test results Change the system Do regression testing

35. Test Team Professional Tester too familiar Programmer with code Analyst System Designer Test User Team Configuration Management Specialist

36. Summary • Testing is still a black art • Testing consists of: • component-testing (unit testing, integration testing) • system testing • Testing has its own lifecycle (?)