Adequacy of a Class Test Suite

1. Adequacy of a Class Test Suite • Ideally – exhaustively test each class • Practically – impossible or too hard • Worth it to exhaustively test some classes with high risk • Measures of adequacy to increase confidence that we have tested enough • state-based coverage • constraint-based coverage • Constraints are pre and post conditions and the class invariant • code-based coverage ECEN5033 University of Colorado -- Class Testing

2. State-based coverage • How many of the transitions in a state transition diagram are covered by the test suite? • “Covered” = touched at least once • May reveal each transition was covered but test values do not adequately cover value ranges • If test cases were generated correctly from a state transition diagram with typical values and good boundary values, the test cases will achieve adequate state-based coverage • If test cases were generated from pre- and post conditions, then it is useful to check them against the state transition diagram to ensure each transition is covered. ECEN5033 University of Colorado -- Class Testing

3. State-based coverage: object interaction • Note how operations interact w.r.t. transitions Test cases for the transition from State C to State D may work if State C was reached from State A but not if State C was reached from State B. “State” is a function of history. ECEN5033 University of Colorado -- Class Testing

4. State-based coverage: transition pairs • Concerning problem on previous page: • Check that the test cases cover all pairs of transitions in the state transition diagram. • In previous table, create test cases to test: • Transition from State A to State C and transition from State C to State D • Transition from State B to State C and transition from State C to State D • In other words, the event sequences: • Event 1 followed by Event 3 • Event 2 followed by Event 3 ECEN5033 University of Colorado -- Class Testing

5. Statechart for Elevator Control ECEN5033 University of Colorado -- Class Testing

6. Hierarchical statechart for Elevator Control ECEN5033 University of Colorado -- Class Testing

7. Portion enlarged ECEN5033 University of Colorado -- Class Testing

8. Constraint-based coverage • How many pairs of pre- and post conditions have been covered? • Using the technique described earlier for generating test cases from pre and post conditions, if one test case is generated to satisfy each requirement, then the test suite meets this measure ECEN5033 University of Colorado -- Class Testing

9. Constraint-Based coverage: object interaction • For each operation opn that is not an accessor operation, • identify the operations op1, op2, etc. for which their preconditions are met when the post conditions for opn hold. • That is, post condition(opn) satisfies (>=) precondition(op1), precondition(op2), etc. • Then execute test cases for operation sequences opn-op1, opn-op2, etc. ECEN5033 University of Colorado -- Class Testing

10. Elevator Controller Requests-elevator-of Controls 1 Indicates-arrival-to 0…n Elevator Opens-Closes Elevator Doors Timer Controls time: seconds velocity: ±m/s position: meters 1 1 1 1 open: Boolean closed: Boolean jammed: Boolean 1 0…n Travels-to 0…n Travels-from Controls m 1 1 Elevator Button Up Floor Button Floor Contains floor: integer position: meters 1 1 on: Boolean floor: integer on: Boolean 1 1 Turns-on Turns-on 1 1 1 Light Down Floor Button Turns-on Contains 1 1 on: Boolean 1 on: Boolean Class Diagram ECEN5033 University of Colorado -- Class Testing

11. How do we complete the class test? • States do not correspond 1-1 to methods or to classes so the state transition diagram gives a different perspective • We want to adequately test the class • Look at the class diagram and see which classes send messages to it (“it” is the class under test) • It could be difficult to create a driver to simulate each class sending msgs to the class-under-test • If you only test each association in this application, it may be a weak level of testing for that class • Test each state transition, better yet, the pairs ECEN5033 University of Colorado -- Class Testing

12. Constraint coverage completes testing it • If you test class A calling every msg it can in class B, that’s more coverage but … • If you use existing classes to do that, it’s hard to get the right combination set up to make it happen • Alternative: • Write a driver with sequences of messages to access sequences of methods in the class you are testing. • Which sequences? ECEN5033 University of Colorado -- Class Testing

13. Which sequences?? • Look at the post conditions of each method in the class you want to test • Look at preconditions of each • Find sequences where • the Post(m1) >= pre(m2) • the Post(m2) >= pre(m3) • Send msgs to m1; m2; m3 • Do this for all possible combinations ECEN5033 University of Colorado -- Class Testing

14. Completion check • See which associations were tested by comparing to the class diagram • See which state transitions were covered this way by comparing to the statechart • If some state transitions were missed or if some post/pre condition pairs or associations were missed, add a few tests ECEN5033 University of Colorado -- Class Testing

15. Where do the pre and post conditions come from? • Use cases?? • The methods themselves? • The class invariant? ECEN5033 University of Colorado -- Class Testing

16. Class TestingTesting Interactions, OATS(Lecture 3 on Class Testing) Software Engineering of Standalone Programs University of Colorado, Boulder

17. Overview • Class Testing • Testing Interactions between objects • Testing Class Hierarchies ECEN5033 University of Colorado -- Class Testing

18. Testing Interactions • Class Testing • Testing Interactions between objects • Identifying & specifying object interactions • Testing object interactions • Collection classes • Collaborator classes • Testing and Design Approach • Sampling • COTS testing • Patterns • Testing exceptions • Testing Class Hierarchies ECEN5033 University of Colorado -- Class Testing

19. Testing Interactions with other classes • An OO program is made of a collection of objects that collaborate to solve a problem. • Correct collaboration is necessary for program correctness. • In this section, we assume interactions are sequential, not concurrent (that’s a subject for next semester) • Focus of interaction testing: Ensure that messaging occurs correctly between objects whose classes have already been tested separately. • Can happen embedded in application or in special environment using a test harness ECEN5033 University of Colorado -- Class Testing

20. What is an object interaction? • Object interaction is: • request by a sender object to a receiver object • to perform one of the receiver’s operations • all of the processing performed by the receiver to complete the request including a possible reply • Includes messages between • an object and its components • an object and other objects with which it is associated ECEN5033 University of Colorado -- Class Testing

21. Object interaction impact • During the processing of any single method invocation on a receiving object • Multiple object interactions can occur • Want to consider impact of these on • the internal state of the receiving object • those objects with which it has an association • Impact may be • “no change” • changes in attribute values in 1 or more objects involved • state changes in 1 or more objects • including state changes of creation or deletion of objects ECEN5033 University of Colorado -- Class Testing

22. Primitive and Non-Primitive Classes • Interactions implied in a class spec where there are references to other objects • Primitive class can be instantiated and used without needing to create any other instance • Relatively few primitive classes in a program • Since an OO program should model the objects in a problem and all the relationships between them, non-primitive classes are common & essential • Non-primitive classes require the use of other objects in some or all of their operations • Identify the classes of those objects ECEN5033 University of Colorado -- Class Testing

23. Identifying Interactions • Identify interactions by association relationships in the class diagram • regular associations • aggregation • composition • “Includes messages between • an object and its components • an object and other objects with which it is associated “ ECEN5033 University of Colorado -- Class Testing

24. Recognizing collaborators • Collaborators may be addressed • directly, e.g. using a variable name • by a pointer or a reference • dynamic type of the object may be different from the static type associated with the pointer • pointers and references are polymorphic, bound to an instance of any number of classes • Pre- and post conditions for operations in the public interface typically refer to states and/or specific attribute values of any collaborating objects ECEN5033 University of Colorado -- Class Testing

25. Collection and collaboration • Collection class • maintains associations with instances of other classes but never actually interacts with those instances • that is, it never requests a service • Collaborating class • Class with more extensive interactions • Collection classes are less common ECEN5033 University of Colorado -- Class Testing

26. Testing Collection Classes • What collection classes do • store references to these objects, representing one-to-many relationships between objects in a program • create instances of these objects • delete instances of these objects • Specification • refers to other objects • does not refer to values computed based on those objects ECEN5033 University of Colorado -- Class Testing

27. Identifying Collaborating Classes • Non-primitive classes that are not collection classes • Use other objects in 1 or more operations as part of their implementation • When a post condition of an operation in a class’ interface • refers to the state of an instance of an object • and/or specifies that some attribute of that object is used or modified, that class is a collaborating class. ECEN5033 University of Colorado -- Class Testing

28. Choosing testing “chunk” size • Number of potential collaborations can grow too large very quickly • During class test: test composing object’s interactions with its composed attributes • As successive layers of aggregation are integrated, • test the interaction between an object and its associated objects • Defect visibility issue – If too large a “chunk” is chosen, intermediate results may be incorrect butnot seen at the level of test-result verification • The more complex the objects, the fewer one should integrate prior to a round of testing ECEN5033 University of Colorado -- Class Testing

29. What factors increase object complexity? • Number of parameters for each method • Number of methods • Number of state attributes in each object Trying to test a chunk that is too complex can result in defects that are hidden from the tests – for example, leaving an object in an inappropriate state that is not expected and therefore not looked at in the test. ECEN5033 University of Colorado -- Class Testing

30. Location of testing emphasis • What are the implications of defensive and contract design approaches when testing interactions? ECEN5033 University of Colorado -- Class Testing

31. Specifying interactions • Remainder of slides assume • Operations defined by a class are specified by preconditions, post conditions, and class invariants • contract design approach ECEN5033 University of Colorado -- Class Testing

32. Contract More responsibility on human designer Less class-level testing due to fewer paths due to less error-checking code MORE interaction testing required to ensure human designer complied with client side of contract using precondition constraints – (are preconditions in receiver met by sender?) “illegal” for test cases to violate preconditions Defensive More responsibility on error-checking code More class-level testing due to increased paths due to more code Once the class-level error-checking code is tested, the interaction testing is simplified since there is no need to test if preconditions are met Appropriate to violate preconditions to see if receiver catches that Contract vs. defensive approach & testing ECEN5033 University of Colorado -- Class Testing

33. Sampling • Exhaustive testing: every possible test case covering every combination of values • Not reasonable much of the time • Want to select the ones that will find the faults in which we are the most interested • Without prior information, random selection is as good as we can do • A sample is a subset of a population (e.g. all possible test cases) that has been selected based on some probability distribution ECEN5033 University of Colorado -- Class Testing

34. Use profile (operational profile) • Gave us a way to associate each use of the system with a frequency • These can be ranked by frequency • Ranks can be turned into probabilities • The higher the frequency of use, the larger the probability of selection ECEN5033 University of Colorado -- Class Testing

35. Operational profile – stratified sample • Stratified sample is a set of samples in which each sample represents a specific subpopulation • Example • Select test cases that exercise each component of the architecture • Divide a population of tests into subsets so that a subset contains all of the tests that exercise a specific component. • Sample on each subset independent of the others • Need a basis for stratifying ECEN5033 University of Colorado -- Class Testing

36. A basis for stratification of test cases • Use the actors from the use cases • Select a sample of test cases from the uses of each actor in the use cases • Each actor uses some subset of the possible uses with some frequency; rank by frequency • Stratifying the test case samples by each actor provides a way to increase the reliability of system • Running selected tests • uses the system the way it will be used in typical situations; • finds defects likely to be found in typical use • gives largest possible increase in reliability with least effort ECEN5033 University of Colorado -- Class Testing

37. Test data generation for a range of values • Generate test data first using the specification as basis • Select values based on boundary values • just within and just on the boundaries if contract • just outside the boundaries also if defensive • Select values within intervals by sampling ECEN5033 University of Colorado -- Class Testing

38. Test data generation for a range of values, cont. • Consider a random function over range 0.. N such as int (random( ) * N) which generates a pseudo random value between 0 and 1 according to a uniform distribution. • +: many different values will be tested over many iterations • Need to log actual values used in case of failure so that the failed test case can be re-created. • A randomly chosen value causing failure is explicitly added to test suite and used to test the repaired software ECEN5033 University of Colorado -- Class Testing

39. Systematic sampling (+ boundary value testing) • Want to be able to increase level of coverage systematically. • We use boundary values coupled with sampling over ranges • We never omit boundary value test data ECEN5033 University of Colorado -- Class Testing

40. Issues about sampling for interaction testing • Sampling issues • Different states can cause two objects from the same class to behave differently • A class family is a subset of a larger family • Possibility of combinatorial explosion in the number of test configurations ECEN5033 University of Colorado -- Class Testing

41. Impact of inheritance hierarchy on s.t.d.’s • Each member of the family may have different states that can cause two objects from the same class to behave differently. • In families of classes, the state transition tables are related along the lines of the inheritance hierarchy. • as we look down the inheritance hierarchy, there will be the same number of states or more states in the derived class as in the base class • should cover the states defined for each class, emphasizing the new states added at that level in inheritance hierarchy. ECEN5033 University of Colorado -- Class Testing

42. Class families as subsets of others • If a class family is a subset of a larger family, after the tests are designed, they can be applied to any of the classes in the family, assuming the substitution principle has been followed during design (see next slide) ECEN5033 University of Colorado -- Class Testing

43. Substitution Principle • Only the following changes are allowed in defining the behavior associated with a new subclass: • Preconditions for each operation must be the same or weaker – less constraining – than those of the superclass • Post conditions for each operation must be the same or stronger – do at least as much as defined by the superclass • Class invariant – must be the same or stronger; add more constraints ECEN5033 University of Colorado -- Class Testing

44. Orthogonal array testing applied to OO • Orthogonal array testing is a sampling technique to limit the explosion of test cases • Define pair-wise combinations of a set of interacting objects because most faults result from interactions due to two-way interactions • An orthogonal array is an array of values in which each column represents a factor. ECEN5033 University of Colorado -- Class Testing

45. Orthogonal array testing applied to OO • An orthogonal array is an array of values in which each column represents a factor. • A factoris • a variable in an experiment; in our case, a specific class family in the software system • Or states of that class family • Each variable takes on a certain set of values called levels (rows); in our case, specific classes or states of those classes • Each cell is an instance of the class, that is, a specific object or a particular state ECEN5033 University of Colorado -- Class Testing

46. OATS – orthogonal array testing system • The factors (columns) are combined pair-wise rather than representing all possible combinations of the levels for the factors. • Suppose we have three factors – A, B, C – and each has three levels • How many possible combinations of these values are there? 3 x 3 x 3 • How many pair-wise combinations? That is, how many combinations where a given level appears exactly twice? factor=class family; level=class; cell=instance ECEN5033 University of Colorado -- Class Testing

47. How many pair-wise combinations? • A B C • 1 1 1 3 • 2 1 2 2 • 3 1 3 1 • 4 2 1 2 • 5 2 2 1 • 6 2 3 3 • 7 3 1 1 • 8 3 2 3 • 9 3 3 2 ECEN5033 University of Colorado -- Class Testing

48. Mapping the terms of OATS to OO sw A state of that nstance A state of that nstance A state of that nstance ECEN5033 University of Colorado -- Class Testing

49. OATS – uses a balanceddesign • Every level of a factor appears exactly the same number of times as every other level of that factor • Think of the rows of a table as test cases • This is a systematic way of reducing the number of test cases. • If we decide to add more later, we know which ones have not been run. • Also logical – most errors are between pairs of objects rather than among several objects ECEN5033 University of Colorado -- Class Testing

50. Look at elevator example (M&S, p. 230) • Want to test interactions between elevator controller, elevator, and wall button-panel • Suppose: • Elevator can be basic, hi-speed, or express • Elevator Controller can control basic, hi-speed, or express • Wall button-panel can be one-button panel or two-button panel • Elevator states: Moving to a floor, stopped, idle • Elevator Controller states: idle, scheduling, dispatching • Wall button-panel states? ECEN5033 University of Colorado -- Class Testing