Download Presentation
## Software Testing and Reliability

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**Software Testingand Reliability**Southern Methodist University CSE 7314**A few words on combinatorics**• Based on the Cartesian product of sets, we can count the number of possible inputs that a program has, i.e. | I |**Example**• Assume a program has a single input, Customer ID (CID) • May be any value in the domain {00000-99999} • What is | I | ?**Example**• Now assume we add a second input, the Order ID (OID) • This may be any value in the domain {00000-99999} as well • Now what is | I |?**Example**• Finally, add a credit card number to the input • This is a 12 digit number • | I | has now reached 10**22 • If we can execute 1 million tests per second, it will take 1016 seconds, or about 300 million years!!**Example**• Since we cannot know what data may exercise a given statement/path in general, we may attempt to resort to exhaustive testing • This attempt is doomed to fail due to the combinatorial explosion**Engineering the testing process**• Any engineered product (and most other things) can be tested in one of two ways • Knowing the specified function that a product has been designed to perform, tests can be conducted that demonstrate each function is fully operational while at the same time searching for errors in each function**Engineering the testing process**• Knowing the internal workings of a product, tests can be conducted to ensure that “all gears mesh” (internal operations performed according to specifications)**Structural testing**• Uses knowledge of the internal workings • Also known as Clear box/glass box • Code based • Can be useful for finding interesting inputs • Misses an entire class of faults, missing code**Behavioral**• Uses knowledge of the specific function that is to be performed • Based solely on the specification without regard for the internals • Also known as Black box • More user oriented • Misses an entire class of faults, extra code (surprises) except by accident**Passing criteria**• How do we know when • 1. a single test has passed • 2. when we are done testing**Passing criteria**• A single test passes when its output is correct • This requires a specific definition of correct and ties into the automated oracle problem**When are we done?**• Conway Criteria: • No syntactic errors (it compiles) • No compile errors or immediate execution failures • There exists Some set of data for which the program gives the correct output • A typical set of data produces the correct output**When are we done?**• Difficult sets of data produce the correct output. • All possible data sets in the problem specification produce the correct output • All possible data sets and likely erroneous input succeeds. • All inputs produce the correct output**Nature of software defects**• Logic errors and incorrect assumptions are inversely proportional to the probability that a program path will be executed**Nature of software defects**• We often believe that a logical path is not likely to be executed when, in fact, it may be executed on a regular basis • Typographical errors are random More of a case for WHITE box testing……**Logical coverage**• What does statement coverage imply ?**Example**• 1 IF (( X > 1) AND ( Y == 1)) • 2 Z = Z / X • 3 END • 4 IF ((X == 2) OR ( Z >1)) • 5 Z = Z + 1 • 6 END What is required for statement coverage ?**Example**• To achieve statement coverage, one can choose x = __, y = __ • Many possible errors are missed: • 1 IF ((X > 1) OR (Y ==1)) • 1 IF ((X >=1) AND (Y ==1)) • 1 IF ((X > 1) AND (Y >=1))**Blindness**• Statement coverage is blind to several classes of error when choosing values to achieve coverage**Assignment blindness**• Due to assignment of a particular value to a variable, the error does not propagate**Equality blindness**• Due to an Equality check of a particular variable, the error does not propagate**Self blindness**• Conditional itself covers up the error**Assignment Blindness example**• 1 IF (<conditional>) • 2 X = 1 • 3 END • 4 IF (X + Y > 1) //Should have been (Y > 0) • 5 .... • 6 END**Assignment Blindness example**• In this example, values are chosen to make the conditional true, the statement X=1 is executed and the error in line 4 is not seen**Assignment Blindness example**• If every path forces the same assignment, then the 'error' doesn't really matter, (does it exist??) • For instance, if the conditional in statement 1 always evaluated to true**Equality Blindness Example**• 1 IF (A == 2) • 2 ..... • 3 IF (B > 1) // Should have been (A + B > 3)**Equality Blindness Example**• In this example, the value of 2 is chosen for A to force execution of the body • The error in statement 3 is missed**Self Blindness**• 1 IF (X < 1) // Should have been (2X < 1) • In this example, the value of 0 is chosen for X**Observation**• Statement coverage is the weakest form of coverage • Many classes of errors can escape this testing • Typical projects have less than 80% statement coverage!!**Branch coverage**• Each branch of a decision must be executed at least once • This is stronger that statement coverage, but it is still weak • Note: A decision is a logical combination of one or more conditions**Branch coverage example**• 1 IF (( X > 1) AND ( Y == 1)) • 2 Z = Z / X • 3 END • 4 IF ((X == 2) OR ( Z >1)) • 5 Z = Z + 1 • 6 END**Branch coverage example**• To achieve branch coverage, one option is to choose • x = __, y = __ • x = __, y = __**Branch coverage example**• Many possible errors are still missed: • 1 IF ((X > 1) OR ( Y ==1) • 1 IF ((X >=1) AND (Y ==1)) • 1 IF ((X > 1) AND (Y <=1))**Blindness in branch coverage**• Branch coverage is blind to several classes of error when choosing values to achieve coverage • Compound decisions (more than one condition) are weakly tested • Boundaries of conditions (within a decision) are not explicitly tested**Condition coverage**• Each condition within a decision must assume all possible values • This is 'potentially' stronger than branch coverage, but not always • It may, in fact, be weaker**Example of condition coverage**• 1 IF (( X > 1) AND ( Y == 1)) • 2 Z = Z / X • 3 END • 4 IF ((X == 2) OR ( Z >1)) • 5 Z = Z + 1 • 6 END**Example of condition coverage**• To achieve condition coverage for statement 1, one must choose values such that: • X > 1 ~(Y == 1) • ~(X > 1) Y == 1**Example of condition coverage**• Both of these vectors miss the execution of statement 2 • A better choice may be: • ~(X > 1) ~(Y == 1) • X > 1 Y == 1**Multi-condition coverage**• Every combination of possible condition values within a decision must be chosen • This is strictly stronger than branch or condition coverage • still has weaknesses!**Multi-condition coverage example**• 1 IF (( X > 1) AND ( Y == 1)) • 2 Z = Z / X • 3 END • 4 IF ((X == 2) OR ( Z >1)) • 5 Z = Z + 1 • 6 END**Multi-condition coverage example**• To achieve multi-condition coverage for statement 1, one must choose values such that: • 1. X > 1 Y == 1 • 2. X > 1 ~(Y == 1) • 3. ~(X > 1) Y == 1 • 4. ~(X > 1) ~(Y == 1)**Multi-condition coverage example**• To achieve multi-condition coverage for statement 2, one must choose values such that: • 5. X == 2 Z > 1 • 6. X == 2 ~(Z > 1) • 7. ~(X == 2) Z > 1 • 8. ~(X == 2) ~(Z > 1)**Multi-condition coverage example**• There are 9 paths through this code and this test set only covers 4 of those**Multi-condition coverage example**• Consider this error • Line 2 is incorrectly written as Z = Z –X • Only the second vector will execute this line but for the values chosen, Z-X = Z/X**Multi-condition coverage example**• Also consider that no vector will execute both line 2 and line 5 • In a more complex example, it is easy to imagine variables being set in line 2 that are then incorrectly used in line 5**Multi-condition coverage example**• For instance: • 2. Z = Z/X; A = 0 • .... • 5. Z = Z + 1; B = 1/A;