Methods for Estimating Defects

1. Methods for Estimating Defects • Catherine V. Stringfellow • Mathematics and Computer Science Department • New Mexico Highlands University • October 20, 2000

2. Problem and Questions To improve the efficiency of system testing by helping testers determine when to stop testing and release software. When should testers stop testing without sacrificing effectiveness? Can defect estimations be used to make release decisions?

3. Data available in defect database • release name • phase defect reported • component name • date defect reported • and much more

4. Approach • Estimate number of remaining components with defects using: • Capture-Recapture Methods • Curve-Fitting Methods • Experience-Based Methods • Estimate number of remaining failures using: • SRGM Selection Method • Compare estimates to acceptability thresholds to make release decisions

5. Estimating Components with Defects in Post-Release, but not in Test • Capture/recapture Methods • Curve Fitting Models • Experience-based Estimation Method (Uses historical data in the estimation)

6. Capture/Recapture Methods • Derived from Wildlife Biological Models • Software Defect Estimation Models • Use multiple independent reviewers to count faults (instead of animals) • A reviewer captures a certain fault and other reviewers who identify same fault are said to “recapture it”

7. x x x x x x x x x x x x x x x x x x x x Capture/Recapture • Model requires an overlap • if most overlap, few remaining (unless MANY found) • if few overlap, many remaining

8. Five CRC Estimation Methods • M0ML • MTML • MTChpm (for two reviewers) • MHJK • MthChao

9. Detection probabilities Defect number Detection probabilities Detection probabilities Defect number Defect number Models’ Assumptions

10. CRC with MtML Example Estimated Remaining is 17-12 = 5 Correct remaining = 8 MLE method tends to underestimate

11. Curve Fitting Models • Detection Profile Method(fit with decreasing exp curve) • Cumulative Method(fit with increasing exponential curve) Estimated # total defects is smallest x-value with y-value <= 0.5 Estimate is x-value at which exp function is asymptotic

12. Sample Data

13. DPM Curve Fitting Example DPM Estimate: approximately 8 remaining Correct Answer: 8

14. Results (3 sites) Ranks: 4 2 5 8 7 6 3 1

15. SRGMs • Some Models’ Assumptions • Test according to operational profile. • Failure rate proportional to number of failures remaining. • Defect repair is immediate. • Defect repair is perfect. • No new code is introduced during the test period.

16. SRGMs in practice • Assumptions violated. • Previous studies show SRGMs are robust: SRGMs perform well in predicting failure rates.

17. SRGMs • Exponential Model (Musa’s) • Delayed S-shaped • Gompertz curve • Yamada Exponential • Yamada Raleigh

18. Release 1

19. Release 2

20. Release 3

21. SRGM Selection Method

22. Integrating Estimation Methods • Logical AND: all methods must say stop • Sequential • If mhjk says stop, stop. • Else if SRGM selection method and one other method says stop, stop. • Else if at least one of the following methods, m0ml, mtml, dpm(linear), AND the experienced-based method say stop, stop. • Else continue.

23. Integrating Estimation Methods • Majority • Group m0ml, mtml, dpm(linear) together. If one says stop, stop. • In case of tie: • a) continue test for another week; or • b) compare number of defects in last week to • acceptability threshold (5)