Errorless Software Development for High Quality: The Cleanroom Approach

1. Cleanroom Software Engineering Kris ReadPaul Werbicki

2. Introduction • Name borrowed from hardware cleanrooms • Software that exhibits zero defect rate • Uses mathematical and statistical models to prevent and detect defects K. Read & P. Werbicki

3. What is Cleanroom? • Incremental development cycle • Mathematically based design specifications • Statistical verification and certification • Software re-engineering K. Read & P. Werbicki

4. Why Cleanroom? • Very high quality • Higher productivity • Reduced life cycle costs • Higher return on investment K. Read & P. Werbicki

5. When is Cleanroom used? • Suitable for critical applications • Used when • defects can cause loss of life • defects can cause critical financial loss • software cannot be updated at a later time K. Read & P. Werbicki

6. It’s all about no errors • “Errorless Software Development” • If no errors enter development, no errors need to be tested or debugged • Elimination of testing and debugging means faster product development • Introduce no errors in the first place K. Read & P. Werbicki

7. Figure. Cleanroom Process Flow (Linger & Trammell, 1996)

8. CUSTOMERINTERACTION CUSTOMERINTERACTION CUSTOMERINTERACTION CUSTOMERINTERACTION CUSTOMERINTERACTION QUALITYEVALUATION ANALYSIS & SPECIFICATION QUALITYEVALUATION QUALITYEVALUATION QUALITYEVALUATION QUALITYEVALUATION QUALITYEVALUATION ANALYSIS & SPECIFICATION ANALYSIS & SPECIFICATION ANALYSIS & SPECIFICATION ANALYSIS & SPECIFICATION ANALYSIS & SPECIFICATION REENGINEERINGDESIGN & VERIFICATION USAGE MODELING & TEST PLANNING REQUIREMENTSSPECIFICATION FUNCTIONSPECIFICATION USAGESPECIFICATION STATISTICAL CERTIFICATIONTESTING DEVELOPMENT DEVELOPMENT DEVELOPMENT DEVELOPMENT DEVELOPMENT INCREMENTPLANNING INCREMENTPLANNING INCREMENTPLANNING INCREMENTPLANNING INCREMENTPLANNING

9. Incremental Development Cycle • Early and continual quality assessment • Increased user feedback • Facilitates process improvements • Permits requirements changes • Avoids integration risk K. Read & P. Werbicki

10. Mathematically Based Design Specifications • All design and code are verified as mathematically correct • Cleanroom achieves this through • Increment Planning • Box Structure Specification and Design K. Read & P. Werbicki

11. Running Example • Phone Number entry component • GUI based • Allows users to enter phone numbers in various common formats K. Read & P. Werbicki

12. Increment Planning • Based on Referential Transparency • Pipeline of user-function software increments • Early and continual quality assessment and user feedback • Avoids risks inherent in late component integration • Permits systematic management of requirement changes over the development cycle K. Read & P. Werbicki

13. Box Structure Specification and Design • A way of specifying software designs • Semi-formal program/component verification • Three box structure stages • Black box • State box • Clear box K. Read & P. Werbicki

14. Black Box Structure • Program or component is modeled as a mathematical function • Defines the system boundaries • Defines all stimuli and responses K. Read & P. Werbicki

15. Figure. Example Black Box Diagram

16. State Box Structure • Maps stimulus and response to old and new state transitions • Defines state data and initial functions • Defines transition functions • Used to verify Black Box Structure K. Read & P. Werbicki

17. Figure. Example State Box Diagram

18. Clear Box Structure • Used to design controls and operations • Shows use of new and reused Black Boxes • Used to verify State Box Structure K. Read & P. Werbicki

19. Figure. Example Clear Box Diagram

20. Figure. Refinement and Verification (Bohner, 2001)

21. Repeated Verification • After each box stage the design is refined • There is a review by all team members • Refinement continues until there is code K. Read & P. Werbicki

22. Integrating Box Structure with Z • Design and code are verified as mathematically correct • Cleanroom uses a semi formal design specification already • Focus is on correct design • Code is not written, compiled or executed until final stage of testing K. Read & P. Werbicki

23. Mathematically Based Design Specifications • Catch errors before they enter the code • No unit testing • Statistical quality testing forms the only real test of the software • Software results can be certified K. Read & P. Werbicki

24. Statistical Quality • Cleanroom uses statistical quality control • Encompasses development and testing • Focuses on detecting the statistically significant defects K. Read & P. Werbicki

25. Statistical Usage Testing • Software has an infinite number of paths • It is impossible to test all paths • A large enough random sample can represent the entire system K. Read & P. Werbicki

26. Advantages of this Approach • Mathematically sound measurement • Conducted as scientific experiments • Proven in hardware engineering • Tests for fitness of use • Highly efficient K. Read & P. Werbicki

27. The Testing Procedure • Pre-test Correctness Verification • Create a Usage Model • Generate random test cases • Execute statistical tests • Execute additional tests if needed K. Read & P. Werbicki

28. Correctness Verification • Function-theoretic static code analysis • Verifies correctness of code compared to specifications • Defined conditions code must meet • Mental/verbal proofs performed during reviews • Based on control structures K. Read & P. Werbicki

29. Correctness Example • User Requirement: All characters shall be numeric digits for the phone number to be considered valid. • Functional Specification: A method called checkNumeric will return true if the character is a digit, false otherwise. K. Read & P. Werbicki

30. Correctness Example function boolean checkNumeric (Char C) { boolean x; if ( ! C.isNumber() ) x = true; else x = false; return x; } K. Read & P. Werbicki

31. Correctness Example • When C.isNumber() is true… • Does x = true make the function true? • When C.isNumber() is false… • Does x = false make the function false? • YES + YES = Passed Correctness K. Read & P. Werbicki

32. Correctness Pros and Cons • Highly effective in practice • Requires good specifications • Can be done incrementally • Code re-use may require re-engineering K. Read & P. Werbicki

33. Usage Models • Define all possible scenarios of use • Includes probabilities for each scenario • Multiple models can be built • Generate test cases automatically K. Read & P. Werbicki

34. Usage Specification • Identify and classify users, scenarios, and environments • Establish hierarchical usage breakdown and probability distribution for software • Obtain agreement with the customer on the basis for software certification. K. Read & P. Werbicki

35. Usage Specification Format • Not defined explicitly by Cleanroom • Often represented as a Markov chain • A directed graph • Nodes are states of use • Arcs are stimuli that cause transitions K. Read & P. Werbicki

36. Usage Model Example • Assume only two user types: USA and Japan • Assume everyone is running the program in an identical environment K. Read & P. Werbicki

37. 30% EMPTY PART DONE 10% EMPTY PART 15% EMPTY PART DONE 9% EMPTY PART DONE USA STARTINGSTATE FIRSTINPUT SECONDINPUT . . . LASTINPUT

38. Test Execution • Tests are randomly chosen • Tests are generated from the models • Controlled tests are executed • Result validated against project goals K. Read & P. Werbicki

39. Conclusions • Survey says: • A wonderful advance in software development • Downright weird approach • Maybe a little of both • Theorists love Cleanroom • Some parts are reasonable, some are not • Similar resistance to formal languages K. Read & P. Werbicki

40. Conclusions • Mathematically based software development is difficult to do • Main-stream software isn’t written this way • Specific to certain types of software • Still fairly unproven as a software development process K. Read & P. Werbicki

41. References Bohner, A. (2001). Software Engineering CS5704. http://www.nvc.cs.vt.edu/~bohner/cs5704/CS5704-Class14.pdf Carnige Mellon Software Engineering Institute (2000). Cleanroom Software Engineering – Suftware Technology Review. http://www.sei.cmu.edu/activities/str/descriptions/cleanroom_body.html Cleansoft – Cleanroom Software Engineering Inc.http://www.cleansoft.com/ Hendrix, S. (2000). CSCI 3308 – Spring 2001. http://www.cs.colorado.edu/~hendrixs/classes/lectures/lecture_18.pdf Linger, R. and Trammel, C. (1996). Cleanroom Software Engineering Reference Model Version 1.0. http://www.sei.cmu.edu/pub/documents/96.reports/pdf/tr022.96.pdf Linger, R. (1993). Cleanroom Software Engineering for Zero Defect Software. http://www.bsn.usf.edu/departments/isds/faculty/hevner/ism6124/Linger1.pdf Wolack, C. (2001). Taking The Art Out Of Software Development – An In-Depth Review of Cleanroom Software Engineering. http://www.scisstudyguides.addr.com/papers/cwdiss725paper1.htm Oshana, R, and Linger, R. (1999). Capability Maturity Model Software Development Using Cleanroom Software Engineering Principles - Results of an Industry Projecthttp://www.computer.org/proceedings/hicss/0001/00017/00017042.PDF Pressmen and Associates (2000). Cleanroom Engineering Resources. http://www.rspa.com/spi/cleanroom.html Stavely, A. (2000). Integrating Z and Cleanroom. http://archive.larc.nasa.gov/shemesh/Lfm2000/Proc/cpp13.pdf THANK YOU! K. Read & P. Werbicki