Cleanroom Software Engineering

1. Cleanroom Software Engineering Lecture 23&24

2. Objectives • What is Clean room SE? • How it works? • Strategies • Process • Box structure • Statistical usage testing • Conclusion

3. Cleanroom SE Philosophy • Mistakes create rework. • Rework takes time and increases cost. • Do it right first time!

4. Where did it come from? • No chance of error • The philosophy focuses on defect avoidance rather than defect removal • It combines many of the formal methods and software quality methods • We will cover in coming weeks.

5. Cleanroom is a Shift in Practice; • From • Individual craftsmanship • Sequential development • Individual unit testing • Informal coverage testing • Unknown reliability • Informal design • To • Peer reviewed engineering • Incremental development • Team correctness verification • Statistical usage testing • Measured reliability • Disciplined engineering specification and design

6. What is it? • Cleanroom software engineering involves the integrated use of • software engineering modeling • program verification • statistical software quality assurance. • Verifies design specification using mathematically-based proof of correctness • Relies heavily on statistical use testing to uncover high impact errors • Generally follows an incremental development process

7. Where is Cleanroom? • Manufacturing areas in microelectronics • Pharmaceutical and biotechnology allow only zero-tolerant contamination in the room. • Everything is suspected even your own body, air-conditioners, fans, your shoes, even your formal clothes. • Everyone has to pass through strictly controlled chambers to make sure that everything is de-contaminated.

8. Cleanroom

9. Objective Cleanroom software engineering yields software that is correct by mathematically sound design, and software that is certified by statistically valid testing.

10. Benefits • Zero failures in the field • that’s the goal any way • a realistic expectation is < 5 failures per KLOC on first program execution in the first team project • Short development cycles • results from use incremental strategy and avoidance of rework • new teams should experience a two-fold productivity increase on the first project and continue the increase • Longer product life • investments detailed specifications and usage models help keep a product viable longer

11. Cleanroom Experience Error Rate: From first execution through completion of certification testing

12. Cleanroom Experience

13. What is it? Formal design and requirements methods + Statistical Usage Testing = Little or No Defects

14. Why are Cleanroom Techniques Not Widely Used • Some people believe cleanroom techniques are too theoretical, too mathematical, and too radical for use in real software development • Relies on correctness verification and statistical quality control rather than unit testing (a major departure from traditional software development) • Organizations operating at the ad hoc level of the Capability Maturity Model, do not make rigorous use of the defined processes needed in all phases of the software life cycle

15. The Cleanroom Strategy • A pipeline of software increments • Independent SE teams • Integrated after certification

16. Cleanroom Principles - part 1 • Small teams • independent specification, development, and certification sub-teams • Incremental development under statistical quality control • performance assessed during each increment using measure like errors per KLOC • feedback is used for process improvement and the development plan is adjusted as needed

17. Cleanroom Process Teams • Specification team • develops and maintains the system specification • Development team • develops and verifies software • the software is not compiled or executes during verification • Certification team • develops set of statistical test to exercise software after development • reliability growth models used to assess reliability

18. Cleanroom Principles - part 2 • Software development based on mathematical principles • the box principle is used for specification and design • formal verification is used to confirm correctness of implementation of specification • program correctness is verified by team reviews using questionnaires • Testing based on statistical principles • operational usage profiles needed • test cases are randomly generated from the usage model • failure data is interpreted using statistical models

19. Cleanroom Process Overview

20. Cleanroom Strategy - part 1 • Increment planning. • The project plan is built around the incremental strategy. • Requirements gathering. • Customer requirements are produced and refined for each increment using traditional methods. • Box structure specification. • Box structures isolate and separate the definition of behavior, data, and procedures at each level of refinement.

21. Cleanroom Strategy - part 2 • Formal design. • Specifications (black-boxes) are iteratively refined to become architectural designs (state-boxes) and component-level designs (clear boxes). • Correctness verification. • Correctness questions are asked and answered, formal mathematical verification is used as required.

22. Cleanroom Strategy - part 3 • Code generation, inspection, verification. • Box structures are translated into program language; inspections are used to ensure conformance of code and boxes, as well as syntactic correctness of code; followed by correctness verification of the code. • Statistical test planning. • A suite of test cases is created to match the probability distribution of the projected product usage pattern.

23. Cleanroom Strategy - part 4 • Statistical use testing. • A statistical sample of all possible test cases is used rather than exhaustive testing. • Certification. • Once verification, inspection, and usage testing are complete and all defects removed, the increment is certified as ready for integration.

24. Increment Planning - Purpose • Developing the right systems the first time, requires customer involvement and feedback throughout the development process • Facilitates the customer’s clarification of system requirements • Requires management control of resources and technical control of complexity • Product quality requires process measurement and control throughout the SW development cycle

25. Increment Planning - Benefits • Concurrent engineering by scheduling parallel development and certification • Stepwise integration through testing cumulative increments • Continuous quality feedback from statistical process control • Continuous customer feedback from actual use • Risk management by treating high-risk elements in early increments • Change management by systematic accommodation of changes

26. The Cleanroom Process Model

27. Cleanroom Method Steps • Increment Planning and Requirements Analysis • High-level Design • Detailed Design • Coding by increment • Pretest by increment • Statistical Testing by increment

28. Increment Planning • Project plan that adopts incremental strategy. • For each increment determine: • Functionality • Projected size • Schedule • Synchronization and integration with other increments

29. Small Teams • Project team is a small team with independent specification, development, and certification sub-teams. • Typically six to eight persons in size • In a large project, small teams for each subsystem, enabling concurrent engineering after the top-level architecture has been established. • Work in a disciplined fashion to ensure the intellectual control of work in progress. • Peer review of all work products results in identification of defects as early as possible in the development cycle.

30. INCREMENTAL DEVELOPMENT • Typical system < 100KLOC • Increment: 2 - 15KLOC • Team size: 6 - 8 • Overlapped development of increments • 12 - 18 weeks from beginning of specification to end of test • (Partitioning is difficult and critical)

31. Box Structured Specifications • Represents system behavior in a hierarchy of black box, state box, and “clear box” forms. • Isolates the creative definition of behavior, data, and procedures at each level of refinement. • Benefits: • demonstrable completeness and consistency in specification • verifiable correctness in implementation. • The development of a system from the highest level of abstraction to the lowest level of procedure is a stepwise process of box structure decomposition. • Specification and design are developed in parallel, resulting in a box structure hierarchy affording complete traceability.

32. Box Structure Specification f: S*  R R S

33. S R Function Black Box State S R State Box State + Algorithm Program S R Clear Box Box Structured Specifications and Design stimuli response current stimulus, old state response, new state

34. Design Refinement & Verification If a function f is expanded into a sequence g and h, the correctness condition for all input to f is: • Does g followed by h do f? When a function f is refined into a conditional statement (if-then-else), the correctness condition for all input to f is: • Whenever condition <c> is true does g do f and whenever <c> is false, does h do f?

35. Design Refinement & Verification • When function f is refined as a loop, the correctness conditions for all input to f is: • Is termination guaranteed? • Whenever <c> is true does g followed by f do f, and whenever <c> is false, does skipping the loop still do f?

36. Advantages of Design Verification • It reduces verification to a finite process. • It lets cleanroom teams verify every line of design and code. • It results in a near zero defect level. • It scales up. • It produces better code than unit testing.

37. Software Testing Based On Statistical Principles • In statistical testing, software testing is viewed as a statistical experiment. • A subset of all possible uses (use cases) of the software is generated, and performance on the subset is used as a basis for conclusions about general operational performance. • In standard experimental parlance, a "sample" is used to draw conclusions about a "population."

38. Statistical Usage Testing • Description of how system will be used • Defined for all possible code scenarios with probability of occurrence • Hierarchical usage breakdown and probability distribution • Concentrates on finding defects that are statistically most significant

39. STATISTICAL USAGE TESTING • Certification of reliability • Process control • Cost effective orientation • Guidelines for test completion (desired reliability reached) or redesign (too many failures found)

40. Certification • Certification implies that reliability can be specified for each component or increment • Reusable components can be stored along with their usage scenarios, program stimuli, and probability distribution. • Each component would have a certified reliability under the usage scenario.

41. Certification Steps 1. Usage scenarios must be created. 2. A usage profile is specified. 3. Test cases are generated from the profile. 4. Tests are executed and failure data are recorded and analyzed. 5. Reliability is computed and certified.

42. Certification Models Sampling model. Software testing executes m random test cases and is certified if no failures or a specified numbers of failures occur. The value of m is derived mathematically to ensure that required reliability is achieved. Component model. A system composed of n components is to be certified. The component model enables the analyst to determine the probability that component i will fail prior to completion. Certification model. The overall reliability of the system is projected and certified.

43. Comparison

44. Summary • Approach that can lead to high reliability. • Uses box structured specification for analysis and design. • Emphasizes correctness verification, rather than testing, for finding and removing errors. • Statistical control testing is used to certify the component • Does not emphasize on unit or integration testing.