Verification and Validation

1. Software Inspections Verification and Validation

2. Today • Placing of inspection in V&V practices • Overview of software inspections • Different reading techniques • Defect estimation • Inspections vs. Testing

3. Historical background • Inspection technique more than 30 years old • Was developed by Michael Fagan in 1970’s • Is one of the software quality assurance techniques

4. Static V&V Reliability Dynamic V&V Process Validation Inspection’s place in VV

5. Software Defects On 4 June 1996, the maiden flight of the Ariane 5 launcher ended in a failure. The failure of the Ariane 501 was caused by the complete loss of guidance and attitude information 37 seconds after start of the main engine ignition sequence (30 seconds after lift- off). This loss of information was due to specificationand design errors in the software of the inertial reference system. The exception was due to a floating-point error: a conversion from a 64-bit integer to a 16-bit signed integer… There was no explicit exception handler to catch the exception, so it followed the usual fate of uncaught exceptions and crashed the entire software, hence the on-board computers, hence the mission.

6. Inspection Objectives • To find problems at the earliest possible point in the software development process • Ensure that necessary rework will be done • Verify that any rework done meets the predefined criteria

7. Inspection Process Fagan • System overview presented to inspection team • Documents are distributed to inspection team in advance • Inspection takes place and discovered faultsare noted • Modifications are made to repair discovered faults • Re-inspection may or may not be required

8. Software Inspection Characteristics • Can be characterized by terms: • Objective • # of participants (subject of discussion) • Reading techniques • Participant’s roles • Inspection meeting • Other

9. Modification to Fagan’s process • Active Design (1985) • Two Person Inspection (1989) • N-fold Inspection (1990) • Phased Inspections (1993) • Inspection without meeting (1993) • Gilb Inspection (1993)

10. Different reading techniques • Ad hoc reading • Checklist • Stepwise abstraction • Scenario based reading • Perspective based reading And • TCD inspections

11. Checklist for Requirements Inspection

12. Checklist for Code Inspections

13. What do inspections cost? About 10-15% of the development budget (Not included the start-up costs) - Tom Gilb Example of costs: Planning; Training; Inspecting; Analysis Other? What are the benefits? 42% of all defects result from lack of traceability from code to design… 50-90% of the defects are caught by the inspections. Cost-benefit analysis

14. Inspections Vs. Testing • To think about: • Which of the V&V technique should be applied and when? • The trade off between inspection and testing? • How to combine the inspections and program testing to reach the best results?

15. V&V of Inspections • How we can improve? • How do we know that inspections work for us?

16. Inspection Metrics • # of errors found • Inspection pace • # of documents inspected • Document size • Other?

17. Defect Content Estimations Objective: • Improve estimation of remaining fault content after review. Motivation: • Process control is essential. • Reviews are an important means for control. • Fault content estimations from reviews give regular feedback on the quality status.

18. Reviews and estimation Normally, the number of found defects are documented in a protocol from the review, but no estimation of remaining fault content is performed. Capture-recapture has been proposed as a way of estimating remaining number of faults. Different statistical methods can be applied in the estimation process.

19. Basic idea of CRC Case 1 Case 2 Reviewer1 Reviewer1 Reviewer3 Reviewer2 Reviewer3 Reviewer2 Isthere more defects remaining in case 1 or case 2?

20. Types of estimation methods

21. Method : Jack-knife method Assumption: Detection ability the same for all reviewers.

22. Method : Maximum-Likelihood Assumption: Detection ability the same for all defects.

23. Graphic estimation methods Previous methods are statistical. Proposal: sort data and fit function Two approaches: • Detection profile method, • exponential decreasing function • Cumulative method, • exponential increasing function

24. Detection profile method Main idea: • sort data: the defect found by most reviewers is defect number 1 and so forth, • fit an exponential function, • estimate (last time the function is above 0.5)

25. Cumulative method Main idea: • sort data: same numbering, plot cumulative, • fit an exponential function (C maximum) • estimate (C - number of defects found) => likely to overestimate C

26. Some general conclusions • Software Inspections are important QA technique • Software companies can benefit from calculating the trade-off between inspections and testing • There exist methods to estimate the defect content in the software artifact, though further research is required

27. Reading Reference • Books: • Steven. R. Rakitin, “Software Verification and Validation For Practitioners and Managers”, Chapters 5,6. • I. Sommerville, “Software Engineering”, Excerpt: Verification & Validation, Addison-Wesley, 7-th addition, pp. 521-527. • Articles: • A. Aurum, H. Petersson and C. Wohlin, “State-of-the-Art: Software Inspections after 25 Years”, Software Testing Verification and Reliability, 2001. • H. Petersson and T. Thelin, ”A Survey of Capture-Recapture in Software Inspections”, Proc. First Swedish Conf. on Software Engineering Research and Practice, October 2001. • S. Biffl, “Using Inspection Data for Defect Estimation”, IEEE Software, November/ December 2000, pp. 36-43.