Trustworthy Systems Through Quantitative Software Engineering

1. Trustworthy Systems Through Quantitative Software Engineering Larry Bernstein Stevens Institute of Technology Castle Point, Hoboken, NJ 07030 USA

2. Trustworthy Software is: • Safe: Does no harm • Reliable: No crash or hang. • Secure: No Hacking Possible

3. What is a Requirement? • A property that must be exhibited by a system to solve some problem. • Requirements may be • Functional providing product capabilities • Non-Functional constraining the implementation

4. System Performance Resulting from Robust Requirements vs. Discrete Specifications Discrete Specifications Ideal Volume Robust Requirements Dynamic Range

5. Relative Project Costs 4x 2x 1.5x 1.25x Relative Cost Range x 0.8x 0.67x 0.5x Concept of Operation Requirements Specifications Product Design Specifications Detailed Design Specifications Accepted Software 0.8x Feasibility Plans and Requirements Product Design Detailed Design Development and Test Phases and Milestones

6. Top Ten Software Risk Items

7. Highlights of Quantitative Approach • Lambda Protocol • Overlaps with Systems Engineering • Industrial Strength Requirements for Software Intensive Systems-of-Systems

8. QSE Lambda Protocol • Prospectus • Measurable Operational Value • Prototyping or Modeling • sQFD • Schedule, Staffing, Quality Estimates • ICED-T • Trade-off Analysis

9. Universal Software Engineering Equation Reliability (t) = ℮-k t when the error rate is constant and where k is a normalizing constant for a software shop and • = Complexity/ effectiveness x staffing

10. Conditions That Cause Unreliability • Poor Algorithms • Missing Deadlines • Roundoff Error Build Up • Memory Leaks • Broken Pointers • Register Misuse

11. QSE Characteristics • Solving the right problem the right way • Certified against requirements. • Certified against problem • Bounded execution domain

12. Case Study: SchedulerPro Prospectus User friendly, efficient interface for students to create and modify class schedules. Features: • Visual schedule creation and editing • Schedule suggestion • Schedule comparison view • Monitor closed-out sections

13. SchedulerPro Prototype Screen

14. SchedulerPro Prototype Screen

15. SchedulerPro Notification Emails

16. Measurable Operational Value SchedulerPro MOV Course scheduling and registration time reduced by an average of 20 minutes per student per semester.

17. SchedulerPro Functional Goals Schedule Classes and Personal Time  Searching  Course Placement  Course Detail Viewing  Course Removal  Scheduling Personal Blocks  Notification (optional)  Course Suggestions (optional)

18. Functional Requirements • Search available classes by: • Same professor • Similar time • Same or equivalent class but different sections • Register and track registrations • Color classes and arbitrary time-blocks by user choice

19. SchedulerPro Nonfunctional Requirements • Integrate with “Web for Students’ and existing authentication systems and avoid incompatibilities • Allow schedules to be saved/accessed from a server or local file • Provide a scaled time-accurate visual representation of the schedule

20. More Non-functional requirements • Make schedules available even if the application is down, provided an internet connection is available • Perform some functions without a live connection to the ‘Web for Students’ registrar web site • Make compatible with all popular browsers • Display section states and print schedules without loss of detail

21. sQFD

22. SchedulerPro Product Reliability • Two hours of unavailability allows for daily backups, service, and reboots of the system • Connections to server are minimized, reducing overall activity on the server

23. Parnas reliability checklist Failures in communication, secondary storage, memory, or any hardware that may interrupt a transaction: • The SQL Server DBMS will not commit incomplete transactions. User will be notified of the error, and will have to redo the transaction. Operator error: • Important operations are confirmed before they are completed to avoid large accidental errors.

24. SchedulerPro Estimate of Reliability R(t) = 1 - F(t) F(t) = P(T ≤ t) • During load testing, we discovered the test server can support 1500 user queries a minute. • P(failures/query) = 55/1500 = 0.036 • Thus, F(t) = 3.6%, which means the software is 96.4% reliable

25. SchedulerPro Reliability Estimate 1/ λ = MTTF = εE/kC k = scaling constant = 1 C is complexity = 2.78 E is the development effort = 36.4 ε is the expansion factor = 1.5 λ = 0.05 t is the continuous execution time for the software R(t) = 95.12%

26. Complexity Chart - Client • Project Type: online transaction • Problem Domain: 2 • Architecture Complexity: 3 • Logic Design – Data: 2 • Logic Design – Code: 3 • Total Score: 10 • Complexity = (10/18) * 5 = 2.78

27. Complexity Chart - Server • Project Type: online transaction • Problem Domain: 1 • Architecture Complexity: 2 • Logic Design – Data: 2 • Logic Design – Code: 2 • Total Score: 7 • Complexity = (7/18) * 5 = 1.94

28. Complexity Chart - Overall • Project Type: client/server • Problem Domain: 2 • Architecture Complexity: 3 • Logic Design – Data: 2 • Logic Design – Code: 3 • Total Score: 10 • Complexity = (10/18) * 5 = 2.78

29. Jan. Function Point Est.

30. April Function Points Est.

31. History of Function Points *Using COCOMO Model

32. Web Points Effort = A * P cdi (Size)P1 Because it is a small web project, assume: A=2.1, B=2.0, P1=1.00, and P2=.5 Language Expansion Factor = 25 From this, we estimate 2.43 KLOC Effort = A* KLOC = 2.1 * 2.43 = 5.103 staff-months Duration =B * EffortP2 = 2.0 * 5.103.5 = 4.518 calendar months

33. ICED-T

34. Creeping Featurism • Endemic to the Software Industry • Occurs on more than 70% of all applications of over 1000 function points • From a 60 project sample • Average creep was 35% • Maximum observed was 200% • Creeping requirements change about 1% per month • For a 3 year project, 1/3 of the delivered requirements would have been added after requirements were initially defined • Rate of Requirements change is higher than for other forms of engineering (electrical, mechanical, civil)

35. Root Causes of Creeping Requirements • Uncertainty in resolving true user needs • For multi-year projects, changes in normal business environment • Failure to adopt methodologies that limit the risk associated with creeping requirements • Primitive fundamental technologies for exploring and modeling requirements • Failure to use technology to measure the impact of creeping requirements • Engineering trade-off analysis is impossible

36. Requirements Management • Establish and maintain a business case to support funding • Strategic linkages to business and technology organizations –AVOID SHELFWARE • Continuous customer agreement on requirements • Requirements agreement used as a basis for estimating, planning, implementing and tracking • FORMAL COMMITMENT PROCESS

37. SchedulerPro Installation Plan Installation 1.Third Party Software Required Scheduler Pro requires the following products to be already installed on the target machine. Please consult the documentation of each product for installation instructions specific to each. - Windows 2000, XP, or 2003 Server - Microsoft IIS, version 5.0 or higher - Microsoft .NET, version 1.1 - Microsoft SQL Server 2000 - Message Queuing Service (Windows component) - ASP.NET State Service

38. SchedulerPro Installation Plan 2. Installing the Scheduler Pro application To install Scheduler Pro, please follow these steps: I. Setting up the web site 1. Create a virtual directory for the site in IIS 2. Copy the contents of the “site” folder to the directory on your hard drive represented by the IIS virtual directory II. Setting up the database 1. Using the SQL Server Enterprise Manager tool, attach the database located under the “sqldb” directory. 2. Create a new user account for accessing this database, and give it read/write access for the database. III. Installing the Suggest, Notified, and Data Loader Windows Services Open the directory titled “services”. Run each of these files: - InstallSuggestService.exe - InstallNotifierService.exe - InstallDataLoaderService.exe Each installer will properly install the service. The Data Loader installer will also ask you for the location and name of the course data file it will load.

39. SchedulerPro Installation Plan 3. Configuring Your Installation Open the directory where you installed the site files (step 2A above). Open the file “web.config”, and find the properties labeled “DBLocation”, “DBUser”, “DBPassword”, and “SecretAuthenticateKey”. These are set to default values that you should modify. Make sure that the location, username, and password match those you set up in SQL Server in step 2B above. Set SecretAuthenticateKey to the random string you are using in your authentication web page. Technical Information Scheduler Pro uses HTTP and XMLTHTTP protocols: - HTTP: As a typical client/server application, an HTTP request is sent to server. Server processes the request and returns HTTP response. - XMLHTTP: client opens XMLHTTP connection to the server. Server executes query on the database and returns needed data. Refer to Development view and process view from the “4+1” architecture views for further technical details.

40. People Software Trustworthiness depends on people: I propose that customers insist that software products identify a Software Architect and Software Project Manager in their contracts

41. Software Architect: • Affirms that the software product solves the customer’s problem • Affirms that the software product is suitably reliable, easy-to-use, extendible, not harmful and robust. That it is trustworthy. • Affirms that the requirements are valid.

42. Software Project Manager: • Affirms that the software was successfully tested against the requirements. • Affirms and identifies the good software engineering processes were used in the software development and integration. • Affirms that the project is within budget, on-time and performs satisfactorily.