testing and cost benefit n.
Skip this Video
Loading SlideShow in 5 Seconds..
Testing and Cost / Benefit PowerPoint Presentation
Download Presentation
Testing and Cost / Benefit

Loading in 2 Seconds...

play fullscreen
1 / 25

Testing and Cost / Benefit - PowerPoint PPT Presentation

  • Uploaded on

Testing and Cost / Benefit. Tor Stålhane. Why cost / benefit – 1 . For most “real” software systems, the number of possible inputs is large. Thus, we can use a large amount of resources on testing and still expect to get some extra benefit from the next test.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Testing and Cost / Benefit' - javan

Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
why cost benefit 1
Why cost / benefit – 1

For most “real” software systems, the number of possible inputs is large.

Thus, we can use a large amount of resources on testing and still expect to get some extra benefit from the next test.

At some point, however, we will reach a point where the cost of the next test will be larger – maybe even much lager – that the benefit that we can expect to receive.

why cost benefit 2
Why cost / benefit – 2

The reason for cost / benefit analysis is the need to answer the question:

“When should we stop testing?”

In economic terms, this can be answered by

  • Comparing costs and benefits
  • Stop when the cost is greater than the expected benefits
why cost benefit 3
Why cost / benefit – 3

The result of a cost / benefit analysis will depend strongly on which costs and benefits we choose to include.

Thus, a cost / benefit analysis can never be completely objective.

which costs should be included
Which costs should be included

As a minimum we need to include costs to

  • Develop new tests
  • Run new tests
  • Correct newly discovered defects

In addition we may include costs incurred by

  • Not being able to use the personnel for other, more profitable activities
  • Dissatisfied customers – bad-will, bad PR
which benefits should be included
Which benefits should be included

As a minimum we need to include benefits from

  • Finding the defects before release – lower correction costs

In addition we may include benefits from

  • Better reputation in the marketplace
  • More satisfied customers
  • Alternative use of personnel resources
cost benefit
Cost / benefit

When the costs and benefits are identified, the decision can be made based on the value of the difference:

Benefits – costs

The main challenge is how to identify the important costs and benefits.

several alternatives
Several alternatives

One of the important benefits of stopping testing is alternative uses of freed personnel resources – e.g. develop a new product or improve an existing product.

Thus, there can be several possible costs and benefits. They are best compared using the concept of leverage.

Leverage = (Benefit – Cost) / Cost

hard costs and soft benefits
Hard costs and “soft” benefits

The main problem for a cost / benefits analysis of testing is that

  • Many of the costs are “now” and easy to identify and enumerate
  • Many of the benefits are “later” and difficult to identify and enumerate.
how to assign value to soft benefits
How to assign value to “soft” benefits

Many of the ”hard” benefits are related to saving money. However,

The company’s main goal cannot be to save money.

The main goals are about increasing e.g.

  • Profit or shareholder value
  • Market shares
  • Market reputation
creation of value
Creation of value

Improving market share or reputation, building a strong brand etc. is all about value creation. This is achieved through creativity and for this we need people.

Thus, moving people from testing and error correction is not about saving money but about creating new value.

soft benefits
“Soft” benefits

To assign value to soft benefits we need to:

  • Identify important company goals and factors (means) contributing to these goals
  • Map the factors onto events related to testing
  • Ask the company what they would be willing to pay to
    • Get an increase of a certain factor – e.g. market share
    • Avoid a decrease of a certain factor – e.g. customer complaints
soft benefits example 1
“Soft” benefits – example (1)

Goal and mean identification

  • Important company goal: Increase sales
  • Testing benefit: Better reputation in the market place

The important questions are:

  • How much will a product with few defects contribute to the company’s reputation?
  • How much will this increase in reputation increase sales?
soft benefits example 2
“Soft” benefits – example (2)

The answers to the to latest questions will have to be answered by management.

Usually they will not be able to give you a number but they will, in most cases, be able to give you an interval.

Thus, the answer to the question “How much will this increase in reputation increase sales?” can be something like 10% to 30%, which then can be used for a benefit assessment.

simple cost benefit
Simple cost / benefit


  • Cost = assessed total costs
  • Benefits(low) = hard benefits + minimum soft benefits

This is a good idea if Cost < Benefits(low)

testing and information 1
Testing and information – 1

As said earlier – cost / benefits are used to decide when to stop testing.

For this decision, we need to consider to parameters:

  • P(wrong) – the probability of making the wrong decisions.
  • Cost(wrong) – the cost of making the wrong decision.
testing and information 2
Testing and information – 2

Before running a test we need to consider what more information we will have if the test case (1) fails or (2) runs OK.

Wrong approach: Make a test, run it and see what we can learn.

Right approach: What info do we need? Design and run test to get the needed info.

testing and information 3
Testing and information – 3

Based on cost and probability, we can compute the risk of the decision:

Risk = P(wrong) * Cost(wrong)

The risk should be added to the costs in the cost / benefit analysis, e.g. in the leverage expression.

the value of information
The value of information

Without any information, the probability of making the wrong decision will be 0.5.

We can decrease this probability by collecting more information. In our case this means running more tests.

It is, however, important to remember that running more tests also will increase our costs. Thus, the two factors risk and cost need to be considered together.


As the name implies, regret is the assessed value of something we regret we did not do. In cost / benefit analysis, it is an opportunity that we do not grab.

The question used to assess the value of the regret is:

“If you do not grab this opportunity, how much would you be willing to pay to have it another time?”

leverage risk and regret
Leverage, risk and regret

We can easily include the assessed risk and regret in the leverage computation:

Total benefit = Regret + Benefit

Total cost = Risk + Cost

L = (Total benefit – Total cost) / Total cost

advanced cost benefit
Advanced cost / benefit

To reduce the complexity of the problem, we will assume that

  • The cost of a wrong decision is constant
  • The benefits are constant until a time T. After T, the benefits will drop to 0, e.g. because a “window of opportunity” has been closed.
  • P(wrong) decreases exponentially with the money invested in information collection.
example 2
Example – 2

We see that everything

done after day 7 is a

waste of time and


After this, we spend money getting rid of a steadily smallerrisk.

  • Understand the company’s goals. Saving money is probably not the most important one.
  • Include all benefits. Use available company knowledge and experience.
  • More knowledge has both a benefit and a cost.