Estimating the number of components with defects post-release that showed no defects in testing C. Stringfellow A. Andre

1. Estimating the number of components with defects post-release that showed no defects in testing C. Stringfellow A. Andrews C. Wohlin H. Peterson Jeremy Mange

2. For nearly any product, defects will appear after release The defects that were not detected in testing point to problems in the test process We would like to know how many of these post-release defects to expect Motivation

3. Using these estimates, release / re-test decisions can be made The authors wish to compare methods of post-release defect estimation Motivation

4. The paper compares three methods of this type of estimation Experience-based Capture-recapture Curve-fitting Experience-based requires historical data,Capture-Recapture and Curve-fitting do not Defect Estimation

5. Requires some sort of historical data: Defect data Change data Historical data can be from previous products or prior releases of the same product If from previous products, they must be similar for these estimations to work Experience-based Estimation

6. Defect data Number of faults in a module can be estimated using the defect history of that module Change data Number of faults in a module can be estimated using the number of file changes Recent changes can be weighted more heavily Experience-based Estimation

7. A type of model originally designed for reviews and inspections Compares the number and types of defects found from multiple test sites Applies statistical estimation to the data based on assumptions about test sites Capture-recapture Estimation

8. Differences between test sites are accounted for in two areas: Ability to find defects Probabilities of finding specific defects This yields four models: Same ability, same probabilities Same ability, different probabilities Different ability, same probabilities Different ability, different probabilities Capture-recapture Estimation

9. Capture-recapture Estimation

10. Fits a mathematical curve to the data points to estimated remaining defects Two basic types: Decreasing – plot number of test sites that found each defect, sorted in decreasing order Increasing – plot cumulative number of defects found by each testing event Curve-fitting Estimation

11. With both increasing and decreasing models, the fitted curve is used to predict the number of remaining defects Both exponential and linear prediction models exist for each type Curve-fitting Estimation

12. 1. Collect the data For each discovered defect, count the number of test sites that discovered it Approach

13. 2. Apply non-historical methods Both capture-recapture and curve-fitting models are applied These provide estimates of the number of total defects Approach

14. 3. Apply experience-based method Use historical data to estimate the number of post-release defects For each past project (or release), calculate: Approach

15. 4. Using the calculations, estimate post- release defects For non-historical models, subtract the number of defects found in testing from the calculated total defects estimate Approach

16. 5. Compare this estimate to a decision threshold value If the number of expected post-release defects is acceptable, the product can be released If not, further testing must be performed Of course, the actual release decision will be based on many criteria, this is just one of those Approach

17. This approach was carried out in a case study on a medical record system Results: Capture-recapture and curve-fitting: 5%-20% error Experience-based:3%-5% error Results

18. If data for similar projects is available, experience-based models provide the best estimations for post-release defects However, non-historical (capture-recapture and curve-fitting) models also provide fairly accurate estimates independent of past data Post-release defect estimates should be used in release decisions Conclusion

19. Questions?