Specifying interactions

1. Specifying interactions • Remainder of slides assume • Operations defined by a class are specified by preconditions, post conditions, and class invariants • contract design approach ECEN5033 University of Colorado -- Class Testing

2. Contract More responsibility on human designer Less class-level testing due to fewer paths due to less error-checking code MORE interaction testing required to ensure human designer complied with client side of contract using precondition constraints – (are preconditions in receiver met by sender?) “illegal” for test cases to violate preconditions Defensive More responsibility on error-checking code More class-level testing due to increased paths due to more code Once the class-level error-checking code is tested, the interaction testing is simplified since there is no need to test if preconditions are met Appropriate to violate preconditions to see if receiver catches that Contract vs. defensive approach & testing ECEN5033 University of Colorado -- Class Testing

3. Sampling • Exhaustive testing: every possible test case covering every combination of values • Not reasonable much of the time • Want to select the ones that will find the faults in which we are the most interested • Without prior information, random selection is as good as we can do • A sample is a subset of a population (e.g. all possible test cases) that has been selected based on some probability distribution ECEN5033 University of Colorado -- Class Testing

4. Use profile (operational profile) • Gave us a way to associate each use of the system with a frequency • These can be ranked by frequency • Ranks can be turned into probabilities • The higher the frequency of use, the larger the probability of selection ECEN5033 University of Colorado -- Class Testing

5. Operational profile – stratified sample • Stratified sample is a set of samples in which each sample represents a specific subpopulation • Example • Select test cases that exercise each component of the architecture • Divide a population of tests into subsets so that a subset contains all of the tests that exercise a specific component. • Sample on each subset independent of the others • Need a basis for stratifying ECEN5033 University of Colorado -- Class Testing

6. A basis for stratification of test cases • Use the actors from the use cases • Select a sample of test cases from the uses of each actor in the use cases • Each actor uses some subset of the possible uses with some frequency; rank by frequency • Stratifying the test case samples by each actor provides a way to increase the reliability of system • Running selected tests • uses the system the way it will be used in typical situations; • finds defects likely to be found in typical use • gives largest possible increase in reliability with least effort ECEN5033 University of Colorado -- Class Testing

7. Test data generation for a range of values • Generate test data first using the specification as basis • Select values based on boundary values • just within and just on the boundaries if contract • just outside the boundaries also if defensive • Select values within intervals by sampling ECEN5033 University of Colorado -- Class Testing

8. Test data generation for a range of values, cont. • Consider a random function over range 0.. N such as int (random( ) * N) which generates a pseudo random value between 0 and 1 according to a uniform distribution. • +: many different values will be tested over many iterations • Need to log actual values used in case of failure so that the failed test case can be re-created. • A randomly chosen value causing failure is explicitly added to test suite and used to test the repaired software ECEN5033 University of Colorado -- Class Testing

9. Systematic sampling(+ boundary value testing) • Want to be able to increase level of coverage systematically. • We use boundary values coupled with sampling over ranges • We never omit boundary value test data ECEN5033 University of Colorado -- Class Testing

10. Issues about sampling for interaction testing • Sampling issues • Different states can cause two objects from the same class to behave differently • A class family is a subset of a larger family • Possibility of combinatorial explosion in the number of test configurations ECEN5033 University of Colorado -- Class Testing

11. Impact of inheritance hierarchy on s.t.d.’s • Each member of the family may have different states that can cause two objects from the same class to behave differently. • In families of classes, the state transition tables are related along the lines of the inheritance hierarchy. • as we look down the inheritance hierarchy, there will be the same number of states or more states in the derived class as in the base class • should cover the states defined for each class, emphasizing the new states added at that level in inheritance hierarchy. ECEN5033 University of Colorado -- Class Testing

12. Class families as subsets of others • If a class family is a subset of a larger family, after the tests are designed, they can be applied to any of the classes in the family, assuming the substitution principle has been followed during design (see next slide) ECEN5033 University of Colorado -- Class Testing

13. Substitution Principlea reminder • Only the following changes are allowed in defining the behavior associated with a new subclass: • Preconditions for each operation must be the same or weaker – less constraining – than those of the superclass • Post conditions for each operation must be the same or stronger – do at least as much as defined by the superclass • Class invariant – must be the same or stronger; add more constraints ECEN5033 University of Colorado -- Class Testing

14. Orthogonal array testing applied to OO • Orthogonal array testing is a sampling technique to limit the explosion of test cases • Define pair-wise combinations of a set of interacting objects because most faults result from interactions due to two-way interactions • An orthogonal array is an array of values in which each column represents a factor. ECEN5033 University of Colorado -- Class Testing

15. Why PAIR-wise • Goal: choose a fairly small subset of test cases that finds a great many defects. (duh!) • How? Test all pairs of whatever it is that is varying. • Pairs are fewer: Suppose 4 input parameters, each of which could take on 3 values. Number of combinations is 3^4 = 81. Can cover all pairwise input combinations in 9 tests • 13 input params, each of which as 3 values; number of comb.’s is 3^13. Can cover all pairwise input combinations in 15 tests. ECEN5033 University of Colorado -- Class Testing

16. More on pairwise • AT&T wanted to test a LAN-based electronic mail system. Original plan was to develop and execute 1500 test cases. Switched to pairwise testing – 50 percent less effort and found 28% more defects. • NIST reviewed 15 years of defect data on recalled medical devices. They concluded: 98% of the reported software flaws would have been detected by testing all pairs of parameter settings. • Why does it work so well? ECEN5033 University of Colorado -- Class Testing

17. Orthogonal array testing applied to OO • An orthogonal array is an array of values in which each column represents a factor. • A factoris • a variable in an experiment; in our case, a specific class family in the software system • Or states of that class family • Each variable takes on a certain set of values called levels (rows); in our case, specific classes or states of those classes • Each cell is an instance of the class, that is, a specific object or a particular state ECEN5033 University of Colorado -- Class Testing

18. What are orthogonalarrays • Consider numbers 1 and 2 – how many pair combinations are there? • An orthogonal array is a 2-dimensional array of numbers that has this property: Choose any two columns of the array; all the pairwise combinations of its values will occur in every pair of columns. ECEN5033 University of Colorado -- Class Testing

19. Example Subscript is number of rows in the array. Superscript is number of columns. 2 says each value is a 1 or a 2. ECEN5033 University of Colorado -- Class Testing

20. another ECEN5033 University of Colorado -- Class Testing

21. Mixed arrays ECEN5033 University of Colorado -- Class Testing

22. How to use • Identify the variables • Determine the number of choices for each variable • Locate an orthogonal array with a column for each variable and values within the columns corresponding to the number of choices. • Map the test problem onto the array • Construct test cases with these combinations ECEN5033 University of Colorado -- Class Testing

23. Sample – see excerpt ECEN5033 University of Colorado -- Class Testing

24. ECEN5033 University of Colorado -- Class Testing

25. When mapped to all of array • Each row becomes a test case • Decode the level numbers for a row in the array back to the individual lists for each factor • Sometimes, you don’t need all of the columns • We can ignore them because we don’t need them. • Or we may decide we can include one more class family and its states in the future ECEN5033 University of Colorado -- Class Testing

26. What about mismatches? • When there is a difference in the number of levels, we can use a column that matches or exceeds the maximum • What about a class with 2 states but there can be a 3 in the array we picked • Interpret the 3 as if it were either 1 or 2. What do you want to emphasize in the testing? • Arbitrary assignment • High frequency • High risk ECEN5033 University of Colorado -- Class Testing

27. Other what-ifs? • See the Copeland excerpt on Class Lecture materials page of web site ECEN5033 University of Colorado -- Class Testing

28. OATS adequacy criteria • Ability to vary how completely the software under test is covered • Exhaustive – all combinations of all factors • Minimal – only interactions between the base classes from each hierarchy are tested • Random – tester haphazardly selects cases from several of the classes; not statistically random • Representative – uniform sample ensures every class is tested to some level • Weighted representative – adds cases to the representative approach based on relative importance or risk associated with the class ECEN5033 University of Colorado -- Class Testing