Testing and Cost / Benefit. Tor Stålhane. Why cost / benefit – 1 . For most “real” software systems, the number of possible inputs is large. Thus, we can use a large amount of resources on testing and still expect to get some extra benefit from the next test.
For most “real” software systems, the number of possible inputs is large.
Thus, we can use a large amount of resources on testing and still expect to get some extra benefit from the next test.
At some point, however, we will reach a point where the cost of the next test will be larger – maybe even much lager – that the benefit that we can expect to receive.
The reason for cost / benefit analysis is the need to answer the question:
“When should we stop testing?”
In economic terms, this can be answered by
The result of a cost / benefit analysis will depend strongly on which costs and benefits we choose to include.
Thus, a cost / benefit analysis can never be completely objective.
As a minimum we need to include costs to
In addition we may include costs incurred by
As a minimum we need to include benefits from
In addition we may include benefits from
When the costs and benefits are identified, the decision can be made based on the value of the difference:
Benefits – costs
The main challenge is how to identify the important costs and benefits.
One of the important benefits of stopping testing is alternative uses of freed personnel resources – e.g. develop a new product or improve an existing product.
Thus, there can be several possible costs and benefits. They are best compared using the concept of leverage.
Leverage = (Benefit – Cost) / Cost
The main problem for a cost / benefits analysis of testing is that
Many of the ”hard” benefits are related to saving money. However,
The company’s main goal cannot be to save money.
The main goals are about increasing e.g.
Improving market share or reputation, building a strong brand etc. is all about value creation. This is achieved through creativity and for this we need people.
Thus, moving people from testing and error correction is not about saving money but about creating new value.
To assign value to soft benefits we need to:
Goal and mean identification
The important questions are:
The answers to the to latest questions will have to be answered by management.
Usually they will not be able to give you a number but they will, in most cases, be able to give you an interval.
Thus, the answer to the question “How much will this increase in reputation increase sales?” can be something like 10% to 30%, which then can be used for a benefit assessment.
This is a good idea if Cost < Benefits(low)
As said earlier – cost / benefits are used to decide when to stop testing.
For this decision, we need to consider to parameters:
Before running a test we need to consider what more information we will have if the test case (1) fails or (2) runs OK.
Wrong approach: Make a test, run it and see what we can learn.
Right approach: What info do we need? Design and run test to get the needed info.
Based on cost and probability, we can compute the risk of the decision:
Risk = P(wrong) * Cost(wrong)
The risk should be added to the costs in the cost / benefit analysis, e.g. in the leverage expression.
Without any information, the probability of making the wrong decision will be 0.5.
We can decrease this probability by collecting more information. In our case this means running more tests.
It is, however, important to remember that running more tests also will increase our costs. Thus, the two factors risk and cost need to be considered together.
As the name implies, regret is the assessed value of something we regret we did not do. In cost / benefit analysis, it is an opportunity that we do not grab.
The question used to assess the value of the regret is:
“If you do not grab this opportunity, how much would you be willing to pay to have it another time?”
We can easily include the assessed risk and regret in the leverage computation:
Total benefit = Regret + Benefit
Total cost = Risk + Cost
L = (Total benefit – Total cost) / Total cost
To reduce the complexity of the problem, we will assume that
We see that everything
done after day 7 is a
waste of time and
After this, we spend money getting rid of a steadily smallerrisk.