Software Testing

1. Software Testing COM 3220 Testing/Spring 98

2. Three meanings of bug • error: mistake made by a developer. Mostly located in people’s head. • fault: an error may lead to one or more faults. Faults are located in text of program. • failure: execution of faulty code may lead to one or more failures. A failure occurs when there is a difference between the results of the correct and incorrect programs. Testing/Spring 98

3. Failure detection • Compare actual output to expected output. • Expected output is from specification. • Specification: any external, independent description of the program, including user documentation. • Are often incomplete, incorrect, ambiguous or contradictory. Specification may be wrong, not the program! Testing/Spring 98

4. Motivation • Derive tests from both the specification and the program. • Derivation is done by ”predicting” likely programmer errors or likely program faults. • Use general rules, e.g., always test boundary conditions. Testing/Spring 98

5. Motivation • Check for faults of omission: missed special cases. • Most common type of fault according to a study by Glass. • Experienced testers have a catalog of programming cliches and associated errors available. See Test Requirement Catalog (low-level omissions). Testing/Spring 98

6. Motivation • First requirement of test design: Be methodical. Three stages: • Finding clues • sources for test requirements • Expanding them into test requirements • useful sets of inputs that should be considered • Writing test specifications • exact inputs and expected outputs Testing/Spring 98

7. Clues • What needs testing? Collect from specification, program, bug reports, etc. • Create a checklist. Testing/Spring 98

8. Test requirements • Create a test requirement catalog Testing/Spring 98

9. Test specifications • Describes input and exact expected output. Testing/Spring 98

10. Supplementary code inspections • Some faults that testing is poor at detecting. Testing/Spring 98

11. Test implementation • Avoid having to write a lot of support code. • It is better to test larger subsystems because less support code needs to be written. • Individual routines are exercised more. • Testing the tests: test coverage as a crude measure. • During test design do not pay attention to coverage criteria. Testing/Spring 98

12. Test implementation • During test design do not pay attention to coverage criteria. Test requirements from other sources should do that anyway. • Complete subsystem testing will usually result in high coverage. • Treat missed branches as clues about weaknesses in the test design. Testing/Spring 98

13. Subsystem Specification Subsystem Code Catalogued Past Experience Clues and Test Requirements Program and Specification Changes Coverage Test Specifications Bug Reports Implemented Tests Testing/Spring 98

14. Application • Graph algorithms: • Depth-first traversal • Finding all paths satisfying some restrictions. • Happens to be be a subsystem of Demeter/Java. • You don’t have to know anything about Demeter. You will learn the minimal things you need. Testing/Spring 98

15. Use Java to write testing code • You will need to write some Java code for testing. Testing/Spring 98

16. Part of Demeter/Java Graph traversal Subsystem Specification Subsystem Code Catalogued Past Experience Clues and Test Requirements Program and Specification Changes Coverage Test Specifications Use Java/Scope Bug Reports Implemented Tests Testing/Spring 98

17. What we want to test • Given a directed acyclic graph G (no multi-edges), traverse all paths from A via B to C. • Given a directed acyclic graph G (no multiedges), traverse all paths from A bypassing B to C. Testing/Spring 98

18. Notation for describing graphs • A = B C D. // node A has three successors • B = E. // node B has only one successor • E = . // E has no successor • This information is put into a file program.cd. • Two files program.beh are given. Contains the traversal specification. Counts visits of C. Testing/Spring 98

19. How to call the program • demjava test • The program will print the paths it traversed and print how often it visits C. Testing/Spring 98

20. Clue list: from A via B to C • What does program do if there is no path from A via B to C? • What if A or B or C do not appear in the graph. • Check that paths from A to C not going through B are excluded: paths of length 1, 2 or 3. Testing/Spring 98

21. Clue list: From A bypassing B to C • What does program do if there is no path from A bypassing B to C? • What if A or B or C do not appear in the graph. Is it ok if B does not appear? • Check that paths from A to C going through B are excluded: paths of length 1, 2 or 3. Testing/Spring 98

22. Test specifications: From A via B to C A=C B X. B=C X. C=. X=C. A=C B. B=C. C=. A A A=B B=C. C=. A B B B C C X C 2 visits 1 visit 1 visit Testing/Spring 98

23. Test specifications: From A via B to C A=C B X Y. Y=B. B=C X. C=. X=C. A Y B C X 4 visits Testing/Spring 98

24. Test specifications: From A bypassing B to C A=C B X Y. Y=B. B=C X. C=. X=C. A Y B C X 2 visits Testing/Spring 98

25. Fundamental Assumptions of Subsystem Testing • Most errors are not very creative. Methodological checklist-based approaches will have a high payoff. • Faults of omission, those caused by a failure to anticipate special cases, are the most important and most difficult type. • Specification faults, especially omissions, are more dangerous than code faults. Testing/Spring 98

26. Fundamental Assumptions of Subsystem Testing • At every stage of testing, mistakes are inevitable. Later stages should compensate for them. • Code coverage is a good approximate measure of test quality. Must be used with extreme care. Testing/Spring 98

27. A summary of subsystem testing • Build the test requirement checklist • Find clues • Expand clues into test requirements • Design the tests • Combine requirements into tests • Check tests for common testing mistakes • Supplement testing with code inspections Testing/Spring 98

28. A summary of subsystem testing • Implement test support code • Implement tests • Evaluate and improve tests • use code coverage tool • find undertested or missing clues • find more test requirements • write more test requirements Testing/Spring 98

29. Testing/Spring 98

30. Test coverage tool • For example: For each traversal, which fraction of traversal methods are used? • How often is each adaptive method called? • Define global counters in Main class. • Use aspect language to instrument code. Generate code. • Testing tool development. Testing/Spring 98

31. Course ideas Advanced OO systems develops testing tools for testing class? Test UML graphical editor. Testing/Spring 98

32. Test strategies • a systematic method used to select and/or generate tests to be included in a test suite. • effective: likely to reveal bugs • Kinds • behavioral = black-box = functional • structural = white-box = glass-box testing • hybrid Testing/Spring 98

33. Testing strategies • behavioral = black-box = functional • based on requirements • structural = white-box = glass-box testing • based on program (coverages) • hybrid • use combination Testing/Spring 98

34. Classification of bugs • unit/component bugs • integration bugs • system bugs Testing/Spring 98

35. Generic Testing Principles • Define the graph • Design node-cover tests (tests that confirm that the nodes are there) • Design edge-cover tests (that confirm all required links and no more) • Design loop tests Testing/Spring 98

36. Generic Testing Principles: Example • Define the graph • UML class diagram • Design node-cover tests (tests that confirm that the nodes are there) • Build at least one object of each class • Design edge-cover tests (that confirm all required links) • use each inheritance edge and association Testing/Spring 98

37. Generic Testing Principles: Example • Define the graph • Finite state machine • Design node-cover tests (tests that confirm that the nodes are there) • Use each state at least once • Design edge-cover tests (that confirm all required links) • use each state transition at least once Testing/Spring 98

38. Testing/Spring 98

39. Quality factors • Correctness • conform to specification • Maintainability • ease with which software can be changed • corrective: error fixing • adaptive: requirement changes MAJORITY • perfective: improve system • Portability Testing/Spring 98

40. Quality factors • Testability • how easy to test? Are requirements clear? • Usability • effort required to learn and operate system • Reliability: mean-time between failures • Efficiency: use of resources • Integrity, Security Testing/Spring 98

41. Quality factors • Reusability • Interoperability • Write Quality Manual to address those issues Testing/Spring 98

42. ISO 9000 Series of Standards(5 years old) • How can customers judge the competence of a software developer? • Adopted by 130 countries. • ISO 9001: Quality Systems - Model for Quality Assurance in Design, Development, Production, Installation and Servicing. (general design) Testing/Spring 98

43. ISO 9000 Series of Standards(5 years old) • ISO 9000-3 Guidelines for the Application of ISO 9001 to the Development, Supply and Maintenance of Software. • ISO 9004-2 Quality Management and Quality System Elements Testing/Spring 98

44. Automatic Verification of Industrial Designs • Based on two papers in: Workshop on Industrial-Strength Formal Specification Techniques, 1995, Boca Raton, Florida, IEEE Computer Society • Automatic Verification of Industrial Designs, pages 88-96 • Timing Analysis of Industrial Real-Time Systems, pages 97-107 Testing/Spring 98

45. Successful formal methodsin industry • Formal methods are mathematical techniques that have been used in the specification and verification of computer systems. • Want to know: Are we building the product correctly? (Different from: are we building the right product). Testing/Spring 98

46. Formal methods • Many different specification languages and proof techniques. • Some are difficult to apply since computers are not good at proving theorems (they need a lot of human help) • Exception: Symbolic Model Checking: Fast, based on OBDD techniques (Ordered Binary Decision Diagrams). Testing/Spring 98

47. Symbolic Model Checking • Determine correctness of finite state systems. • Developed at CMU by Clarke/Emerson • Specifications are written as formulas in a propositional temporal logic. • Temporal logic: expressing ordering of events without introducing time explicitly Testing/Spring 98

48. Temporal Logic • A kind of modal logic. Origins in Aristotle and medieval logicians. Studied many modes of truth. • Modal logic includes propositional logic. Embellished with operators to achieve greater expressiveness. • A particular temporal logic: CTL (Computation Tree Logic) Testing/Spring 98

49. Computation Tree Logic • Used to express properties that will be verified • Computation trees are derived from the state transition graphs • State transition graphs unwound into an infinite tree rooted at initial state Testing/Spring 98

50. Computation Tree Logic • CTL formulas built from • atomic propositions, where each proposition corresponds to a variable in the model • Boolean connectives • temporal operators. Two parts • path quantifier (A, E) • temporal operator (F,G,X,U) Testing/Spring 98