A beginners guide to testing

1. A beginners guide to testing Philip Johnson Collaborative Software Development Laboratory Information and Computer Sciences University of Hawaii Honolulu HI 96822

2. Objectives • Understand the basic concepts, terminology, and approaches to software testing. • Be able to apply this information to improve the quality of your own testing efforts during software development.

3. Why is testing important?

4. First National Bank (1996) • Problem: • An error at First National Bank of Chicago resulted in the balance of 800 customers being inflated by a total of $763 billion. • Reason: • Inadequate testing. Bank updated ATM transaction software with new message codes. Message codes were not tested on all ATM protocols, resulting in some ATMs interpreting them as huge increases to customer balances.

5. Therac-25 (1985-1987) • Problem: • Six people were overexposed during radiation treatments for cancer by the Therac-25 radiation therapy machine. Three people were believed to have died from the overdoses. • Reason: • Inadequate testing. Hardware safety locks removed and replaced by software safety locks, which could be overcome by technician “type ahead”.

6. Ariane 5 (1996) • Problem: • Ariane 5 rocket exploded on its maiden flight. • Reason: • Inadequate testing. Navigation package inherited from Ariane-4 without proper testing. New rocket flew faster, resulting in larger values of some variables, resulting in an attempt to convert a 64-bit floating number into a 16 bit integer. Code was caught and action taken was to shut down navigation system.

7. ICS 613, Fall 2011 • Problem: • A student got a bad grade for ICS 613. • Reason: • Inadequate testing. The student did not learn how to write good test cases for their code.

8. Testing fits intoValidation and Verification • Validation: • Establishing the fitness of a software product for its use. • “Are we building the right product?” • Requires interaction with customers. • Verification: • Establishing the correspondence between the software and its specification. • “Are we building the product right?” • Requires interaction with software.

9. Static vs. Dynamic V&V • Static V&V: • Software inspections • Static analysis of source code • Control/data flow analysis • Dynamic V&V: • Defect testing • Looks for errors in functionality • Performance analysis • Looks for errors in scalability, performance, reliability

10. Why is testing hard? • Exhaustive testing: • Execute program on all possible inputs and compare actual to expected behavior. • Could “prove” program correctness. • Not practical for any non-trivial program. • Practical testing: • Select a tiny % of all possible tests. • Goal: executing the tiny % of tests will uncover a large % of defects present! • A “testing method” is essentially a way to decide which tiny % to pick.

11. How do we determine which tests to write? Selected examples: • Functional (black box) testing • Use specification to determine the tests • Structural (white box) testing • Use coverage to assess the quality of tests • Test-driven development • Write tests as specification • Use tests to determine the implementation! • Combinations of methods are acceptable!

12. Black box testing

13. Specification based testing • Specification: • A mapping between the possible “inputs” to the program and the corresponding expected “outputs” • Specification-based testing: • Design a set of test cases to see if inputs actually map to outputs. • Does not require access to source code • Differences with White Box (coverage) testing: • Can catch errors of omission. • Effectiveness depends upon the quality of specification

14. Equivalence classes Goal: Divide the possible inputs into categories such that testing one point in each category is equivalent to testing all points in the category. Provide one test case for each point. • Equivalence class definition is usually an iterative process and goes on throughout development. Use heuristics to get you started designing your test cases.

15. Unit test design heuristics • If input is a value: • maximum value • minimum value • empty value • typical value • illegal value • If input is a sequence: • Single element • Empty sequence • Max and min element values • Sequences of different sizes • Illegal elements • If I/O specification contains conditions: • true • false • If I/O specification contains iterations: • zero times • 1 time • > 1 time

16. Web app design heuristics • Every page is retrieved at least once • Prevent 404 errors. • Every link is followed at least once. • Prevent 404 errors. • All form input fields are tested with: • Normal values • Erroneous values • Maximum/minimum values • Always check response for appropriateness.

17. Tool support: JUnit

18. Tool Support: JUnitReport

19. White box testing

20. Statement coverage • For a test case to uncover a defect, the statement containing the defect must be executed. • Therefore, a set of test cases which guarantees all statements are executed might uncover a large number of the defects present. • Whether or not the defects are actually uncovered depends upon the program state at the time each statement is executed.

21. Control flow coverages • Control flow coverage adds conditions to statement coverage to raise the odds of discovering defects. • Branch coverage: • Every conditional is evaluated as both true and false during testing. • Loop coverage: • Every loop must be executed both 0 times and more than 1 time. • Path coverage: • All permutations of paths through the program are taken

22. JaCoCo: A Java Coverage Tool

23. JaCoCo: Drilldown to Package

24. JaCoCo: Drilldown to Class

25. JaCoCo: Red regions not tested

26. “Coverage-driven test case design” • 1. Write some test cases to exercise your code in new ways. • 2. Run your coverage tool. • 3. If coverage is not 100%, go to 1.

27. “Coverage-driven test case design” is bad! • Using coverage data to drive the design of your tests often results in a poorly designed test case suite. (See readings.) • Instead, use coverage to assess the quality of your test method. • In a nutshell: • Good test design method -> Good coverage • But not the other way around.

28. Limitations of white-box testing • Can catch bugs in code that has been written. • Cannot catch bugs in code that has not been written! • Errors of omission: code that ought to have been written but wasn’t • Missing boolean conditions in IF statements • Missing exception handlers • To catch bugs in code that has not been written, you must compare behavior of program to its specification.

29. Quality assurance in this class • You will use a combination of manual and automated quality assurance. • Automated QA: • Use Checkstyle, PMD, FindBugs, and eliminate all warnings. • Manual QA: • Use JUnit to develop tests, use Jacoco to check coverage. • Use Code Review to look for remaining errors.