when to test less l.
Skip this Video
Loading SlideShow in 5 Seconds..
When to Test Less PowerPoint Presentation
Download Presentation
When to Test Less

Loading in 2 Seconds...

play fullscreen
1 / 19

When to Test Less - PowerPoint PPT Presentation

  • Uploaded on

When to Test Less. Authors: Tim Menzies and Bojan Cukic Presenter: Genet Balaker. Agenda. Introduction The Theory of Testing The Realty of Testing The Average Shape of Software Implementation Conclusion. Introduction.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'When to Test Less' - lynn

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
when to test less

When to Test Less

Authors: Tim Menzies and

Bojan Cukic

Presenter: Genet Balaker

  • Introduction
  • The Theory of Testing
  • The Realty of Testing
  • The Average Shape of Software
  • Implementation
  • Conclusion
  • Testing software in the large is different from testing it in the small
  • Small projects often lack funds to set up software test environments and have to depend on manual and simple automatic testing schemes.
introduction cont
Introduction (cont.)
  • a program's shape can help determine whether rapid testing with limited resources will be as effective as elaborate and expensive testing regimes.
  • for many small-scale projects, a small number of randomly selected tests will adequately probe the software
theory of testing
Theory of Testing
  • Black-box testing to detect faults in a software
  • How many random black-box tests do we need to find a fault?
  • A randomly chosen input has odds x that it stumble across some fault,
  • This input will miss that fault with odds of 1-x
  • If we conduct N random black-box tests, the odds of a failure not occurring is (1-x)N,

so, the probability y of finding a fault in N random tests is

y = 1- (1-x) N

theory of testing cont
Theory of Testing (cont.)
  • Black-box testing can be expensive.
  • Formal-methods testing
  • Benefit of formally checking a system is that such formal proofs can find faults more systematically than standard testing.
  • A single formal first-order query is equivalent to many black-box test inputs.
  • The cost of such rigorous formal analysis ca be impractically high
theory of testing cont7
Theory of Testing (cont.)
  • We can only use formal methods to test small, critical portions of the system.
  • Even when applying formal verification, its recommended to use black-box testing.
  • Another approach to testing small-scale software is white-box testing.
    • Test inputs exercise K different partitions.
    • Each partition exercises one interesting feature of a program
theory of testing cont8
Theory of Testing (cont.)
  • The odds of detecting a fault with white-box methods are nearly the same as black-box methods.
  • White-box method using k partitions is only k times better at finding errors than black-box method testing, but it slightly reduces the number of tests required
  • Building the partitions is time consuming.
the realty of testing
The Realty of Testing
  • Given the mathematics of black-box testing, software-in-the-small project doesn’t crash more often.
  • Program shape = the shape of the pathways within the program
    • Numerous and tangled like spaghetti
    • Few and not complex
  • What should the shape of the software be to adequately test it using limited resources?
the realty of testing cont
The realty of Testing (cont.)
  • Studies suggest that simple shapes are common
  • Du-pathways
    • A link from where a variable is defined to where it is used
    • As the du-pathways shrink, the number of inputs required to reach part of the program also shrinks
  • Control flow diagrams
    • They link program statements: while, repeat, and for statements
    • They grow as the program grow
    • A worst-case control-flow
  • Saturation effect
    • Consistent with programs containing either many portions with simple shapes or many portions that are so twisted in shape
shape of a software
Shape of a software
  • If it is possible to show a software has the right shape, it can be tested quickly.
  • What’s the right shape of a software?
  • What’s an average shape of a software?
shape of a software cont
Shape of a software (cont.)
  • Assumptions about the tested program’s structure to find out the shape
    • Programs are networks connecting system concepts
    • A fault explanation tree is a tree whose leaves are inputs and whose root is fault
    • Testing is a process of trying to generate a fault explanation tree from the program network
    • An explanation tree has edges and nodes
    • Testing is the process of extracting NAYO trees (no, and, yes and or)
  • Based on the above assumptions, it’s possible to simulate the construction of an explanation tree across a program
  • It is possible to characterize the tree’s shape by the number of tests N required to reach it’s root to find a fault
  • Complex shaped trees require large N
implementation cont
Implementation (cont.)

simple if 0 < N < 102

Shape = moderate if 102 <= N <104

hard 104 <= N <106

overly complex if N >= 106


Given in inputs to a NAYO network with V nodes, the odds of hitting a

fault straight away from the inputs is

Xo = in/v (A)

The probability of reaching an and node with andp parents is the probability

of reaching all its parents:

Xand = xiandp (B)

where xi is the probability we computed in the prior step of the simulation (and the base case of x i = 0 is computed from Equation A).

The probability of reaching an or node with orp parents is the probability

of not missing any of its parents; that is,

Xor= 1 − (1 − x)orp (C)

implementation cont16
Implementation (cont.)

If the ratio of and nodes in a NAYO network is andf, then the ratio of or

nodes in the same network is or f = 1 – andf. The odds of reaching some random node xj is the weighted sum of the probabilities of reaching and nodes or or nodes

xj = andf * xand * orf * xor (D)

The number of tests required to be 99% sure of finding a fault with probability xj:

y = 0.99 = 1 − ((1 − xj)N) (E)

Therefore: N = log(1- 0.99) / log(1-xj)

implementation cont17
Implementation (cont.)

Results from the Simulation Runs


Classification Threshold Percent

Simple 0 < N < 102 36

Moderate 102 <=N < 104 19

Hard 104 <= N < 106 25

Overly complex N > =106 20

implementation cont18
Implementation (cont.)
  • Over half of the time, 100 random tests will yield as much information as much more prolonged series of tests
  • 10,000 randomly selected tests will probe the program as much as very prolonged series of tests will.
  • Software-in-the-small projects should routinely execute overnight test runs in which tens of thousands of test cases are generated at random
  • software-in-the-small projects, a cost-cutting heuristic is to avoid elaborate and expensive testing regimes.