1 / 19

When to Test Less

When to Test Less. Authors: Tim Menzies and Bojan Cukic Presenter: Genet Balaker. Agenda. Introduction The Theory of Testing The Realty of Testing The Average Shape of Software Implementation Conclusion. Introduction.

lynn
Download Presentation

When to Test Less

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. When to Test Less Authors: Tim Menzies and Bojan Cukic Presenter: Genet Balaker

  2. Agenda • Introduction • The Theory of Testing • The Realty of Testing • The Average Shape of Software • Implementation • Conclusion

  3. Introduction • Testing software in the large is different from testing it in the small • Small projects often lack funds to set up software test environments and have to depend on manual and simple automatic testing schemes.

  4. Introduction (cont.) • a program's shape can help determine whether rapid testing with limited resources will be as effective as elaborate and expensive testing regimes. • for many small-scale projects, a small number of randomly selected tests will adequately probe the software

  5. Theory of Testing • Black-box testing to detect faults in a software • How many random black-box tests do we need to find a fault? • A randomly chosen input has odds x that it stumble across some fault, • This input will miss that fault with odds of 1-x • If we conduct N random black-box tests, the odds of a failure not occurring is (1-x)N, so, the probability y of finding a fault in N random tests is y = 1- (1-x) N

  6. Theory of Testing (cont.) • Black-box testing can be expensive. • Formal-methods testing • Benefit of formally checking a system is that such formal proofs can find faults more systematically than standard testing. • A single formal first-order query is equivalent to many black-box test inputs. • The cost of such rigorous formal analysis ca be impractically high

  7. Theory of Testing (cont.) • We can only use formal methods to test small, critical portions of the system. • Even when applying formal verification, its recommended to use black-box testing. • Another approach to testing small-scale software is white-box testing. • Test inputs exercise K different partitions. • Each partition exercises one interesting feature of a program

  8. Theory of Testing (cont.) • The odds of detecting a fault with white-box methods are nearly the same as black-box methods. • White-box method using k partitions is only k times better at finding errors than black-box method testing, but it slightly reduces the number of tests required • Building the partitions is time consuming.

  9. The Realty of Testing • Given the mathematics of black-box testing, software-in-the-small project doesn’t crash more often. • Program shape = the shape of the pathways within the program • Numerous and tangled like spaghetti • Few and not complex • What should the shape of the software be to adequately test it using limited resources?

  10. The realty of Testing (cont.) • Studies suggest that simple shapes are common • Du-pathways • A link from where a variable is defined to where it is used • As the du-pathways shrink, the number of inputs required to reach part of the program also shrinks • Control flow diagrams • They link program statements: while, repeat, and for statements • They grow as the program grow • A worst-case control-flow • Saturation effect • Consistent with programs containing either many portions with simple shapes or many portions that are so twisted in shape

  11. Shape of a software • If it is possible to show a software has the right shape, it can be tested quickly. • What’s the right shape of a software? • What’s an average shape of a software?

  12. Shape of a software (cont.) • Assumptions about the tested program’s structure to find out the shape • Programs are networks connecting system concepts • A fault explanation tree is a tree whose leaves are inputs and whose root is fault • Testing is a process of trying to generate a fault explanation tree from the program network • An explanation tree has edges and nodes • Testing is the process of extracting NAYO trees (no, and, yes and or)

  13. Implementation • Based on the above assumptions, it’s possible to simulate the construction of an explanation tree across a program • It is possible to characterize the tree’s shape by the number of tests N required to reach it’s root to find a fault • Complex shaped trees require large N

  14. Implementation (cont.) simple if 0 < N < 102 Shape = moderate if 102 <= N <104 hard 104 <= N <106 overly complex if N >= 106

  15. Implementation Given in inputs to a NAYO network with V nodes, the odds of hitting a fault straight away from the inputs is Xo = in/v (A) The probability of reaching an and node with andp parents is the probability of reaching all its parents: Xand = xiandp (B) where xi is the probability we computed in the prior step of the simulation (and the base case of x i = 0 is computed from Equation A). The probability of reaching an or node with orp parents is the probability of not missing any of its parents; that is, Xor= 1 − (1 − x)orp (C)

  16. Implementation (cont.) If the ratio of and nodes in a NAYO network is andf, then the ratio of or nodes in the same network is or f = 1 – andf. The odds of reaching some random node xj is the weighted sum of the probabilities of reaching and nodes or or nodes xj = andf * xand * orf * xor (D) The number of tests required to be 99% sure of finding a fault with probability xj: y = 0.99 = 1 − ((1 − xj)N) (E) Therefore: N = log(1- 0.99) / log(1-xj)

  17. Implementation (cont.) Results from the Simulation Runs ____________________________________ Classification Threshold Percent Simple 0 < N < 102 36 Moderate 102 <=N < 104 19 Hard 104 <= N < 106 25 Overly complex N > =106 20

  18. Implementation (cont.) • Over half of the time, 100 random tests will yield as much information as much more prolonged series of tests • 10,000 randomly selected tests will probe the program as much as very prolonged series of tests will.

  19. conclusion • Software-in-the-small projects should routinely execute overnight test runs in which tens of thousands of test cases are generated at random • software-in-the-small projects, a cost-cutting heuristic is to avoid elaborate and expensive testing regimes.

More Related