SENG 521 Software Reliability & Testing

1. SENG 521Software Reliability & Testing Part 1: Overview of Software Reliability Engineering Department of Electrical & Computer Engineering, University of Calgary B.H. Far （far@ucalgary.ca） http://www.enel.ucalgary.ca/People/far/Lectures/SENG521/ far@ucalgary.ca

2. Contents • About this course. • What is software reliability? • What factors affect software quality? • What is software reliability engineering? • What is software testing? • Software reliability engineering process. far@ucalgary.ca

3. Part 1 Section 1 Introduction to Software Reliability Engineering far@ucalgary.ca

4. Realities … • Software development is a very high risk task. • About 20% of the software projects are canceled. (missed schedules, etc.) • About 84% of software projects are incomplete when released (need patch, etc). • Almost all of the software projects costs exceed initial estimations. (cost overrun) far@ucalgary.ca

5. Software Engineering /1 • Business software has a large number of parts that have many interactions (i.e., complexity). • Software engineering paradigms provide models and techniques that make it easier to handle complexity. • A number of contemporary software engineering. paradigms have been proposed: • Object-orientation • Component-ware • Design patterns • Software architectures • etc. far@ucalgary.ca

6. Languages that have their conceptual basis determined by machine architecture Increase of Complexity Languages that have their key abstractions rooted in the problem domain Software Engineering /2 • Evolution of software engineering paradigms: • Assembly languages • Procedural and structured programming • Object Oriented programming • Component-ware • Design patterns • Software architectures … • Software Agents … time far@ucalgary.ca

7. A Central Question • In spite of having many development methodologies, a central question remained unanswered is: How often will the software fail? • This is exactly what we are going to learn about! far@ucalgary.ca

8. A Typical Problem • Our company provides microcontrollers for oil and gas industry. One of our major customers has said that they are considering moving to a competitor’s product that is more reliable. Our job is to put together a plan to determine and improve the reliability of our product. This includes both hardware and software reliability. For hardware reliability, in our company we have a long tradition of quality control. But what can we do for the software reliability? far@ucalgary.ca

9. Another Typical Problem • Unit manufacturing cost of a software product is 50$. The company decides to offer one year free update to its customers. Suppose that failure intensity of the product at the release time is  = 0.01 failures/month. What should be the unit cost of the product including warranty services? far@ucalgary.ca

10. About This Course … • The topics discussed include: • Concepts and relationships • Analytical models and supporting tools • Techniques for software reliability improvement, including: • Fault avoidance, fault elimination, fault tolerance • Error detection and repair • Failure detection and retraction • Risk management far@ucalgary.ca

11. At The End … • What is software reliability engineering (SRE)? • Why SRE is important? How does it affect software quality? • What are the main factors that affect the reliability of software? • Is SRE equivalent to software testing? What makes SRE different from software testing? • How can one determine how often will the software fail? • How can one determine the current reliability of the software under development? • How can one determine whether the product is reliable enough to be released? • Can SRE methodology be applied to the current ways of software development: like object-oriented, component-based and agile development? • What are challenges and difficulties of applying SRE? • What are current research topics of SRE? far@ucalgary.ca

12. Moral • Industrial oriented course. • Attend lectures: there is more to lectures than to the notes. • Sit close-up: some of the material is hard to see from the back. • Feel free to ask questions. far@ucalgary.ca

13. A Few Words at the Beginning … • From last year’s students comments for Prof. X who received the “Best Instructor Award” This prof can teach circuits even to a monkey! • Sorry, here is not a circus … and • I’m not an animal trainer … • You must work hard to grasp the concepts and I’ll be with you and help you all the way to the end far@ucalgary.ca

14. Quality Cost Time People Technology What Affects Software? • Time: • Meeting the project deadline. • Reaching the market at the right time. • Cost: • Meeting the anticipated project costs. • Quality (reliability): • Working fine for the designated period on the designated system. far@ucalgary.ca

15. Terminology Failures Faults Errors Treats Attributes Means Availability Reliability Safety Confidentiality Integrity Maintainability Dependability Fault prevention Fault tolerance Fault removal Fault forecasting far@ucalgary.ca

16. Definition: Service • The service delivered by a system is its behaviour as it is perceived by its users; a user is another system (physical, human) that interacts with the former at the service interface. • The function of a system is what the system is intended to do, and is described by the functional specification. • Correct service is delivered when the service implements the system function. • The delivery of incorrect service is a system outage. • A transition from incorrect service to correct service is service restoration. far@ucalgary.ca

17. Definition: Dependability • Dependability of a computing system is • The ability of a system to deliver service that can justifiably be trusted. • The ability of a system to avoid failures that are more frequent or more severe, and outage durations that are longer, than is acceptable to the users. far@ucalgary.ca

18. Error cause Fault cause Failure Dependability: Treats • An error is a human action that results in software containing a fault. • A fault (bug) is a cause for either a failure of the program or an internal error (e.g., an incorrect state, incorrect timing). • Among the 3 factors only failure is observable. far@ucalgary.ca

19. Definition: Failure • Failure: • A system failure is an event that occurs when the delivered service deviates from correct service. A failure is thus a transition from correct service to incorrect service, i.e., to not implementing the system function. • Any departure of system behavior in execution from user needs. A failure is caused by a fault and the cause of a fault is usually a human error. • Failure Intensity: the number of failures per natural or time unit. Failure intensity is way of expressing reliability, e.g., 5 failures per hour; 2 failures per 1000 transactions. far@ucalgary.ca

20. Definition: Fault • Fault: A fault is a cause for either a failure of the program or an internal error (e.g., an incorrect state, incorrect timing) • A fault must be detected and then removed • Fault can be removed without execution (e.g., code inspection, design review) • Fault removal due to execution depends on the occurrence of associated “failure”. • Occurrence depends on length of execution time and operational profile. • Defect: refers to either fault (cause) or failure (effect) far@ucalgary.ca

21. Require- ments Functional Test System Test Field Use Coding Design Fault Origin 50 % 40 % 10 % 50 % 7 % 25 % 5 % 3 % 10 % 20 KDM Cost per Fault 12 KDM 1 KDM 1 KDM 1 KDM 6 KDM 1 KDM = 1,000 Deutsch Marks Cost of a Defect Fault Detection CMU. Software Engineering Institute far@ucalgary.ca

22. Require- ments Functional Test System Test Field Use Coding Design Phase Fault Origin 10% 50% 40% 5 40 % <5 % 5 % 20 % 20 % 10 % 4 30 % 30 % 5 % 3 % 20 % 12 % 3 38 % 32 % 2 % 20 % 0 % 8 % 2 50 % 30 % 3 % 0 % 0 % 17 % 50 % 33 % 2 % 0 % 0 % 1 15 % Cost per Fault 20 KDM 12 KDM 1 KDM 1 KDM 1 KDM 6 KDM 1 KDM = 1,000 Deutsch Marks Testing and Process Maturity Fault Detection CMU. Software Engineering Institute far@ucalgary.ca

23. Definition: Error • Error has two meanings: • A discrepancy between a computed, observed or measured value or condition and the true, specified or theoretically correct value or condition. • A human action that results in software containing a fault. • Human errors are the hardest to detect. far@ucalgary.ca

24. Dependability: Attributes /1 • Availability: readiness for correct service • Reliability: continuity of correct service • Safety: absence of catastrophic consequences on the users and the environment • Confidentiality: absence of unauthorized disclosure of information • Integrity: absence of improper system state alterations • Maintainability: ability to undergo repairs and modifications far@ucalgary.ca

25. Dependability: Attributes /2 • Dependability attributes may be emphasized to a greater or lesser extent depending on the application: availability is always required, whereas reliability, confidentiality, safety may or may not be required. • Other dependability attributes can defined as combinations or specializations of the six basic attributes. • Example: Security is the concurrent existence of • Availability for authorized users only; • Confidentiality; and • Integrity with improper taken as meaning unauthorized. far@ucalgary.ca

26. Definition: Availability • Availability: a measure of the delivery of correct service with respect to the alternation of correct and incorrect service far@ucalgary.ca

27. Definition: Reliability /1 • Reliability is a measure of the continuous delivery of correct service • Reliability is the probability that a system or a capability of a system functions without failure for a “specified time” or “number of natural units” in a specified environment. (Musa, et al.) Given that the system was functioning properly at the beginning of the time period • Probability of failure-free operation for a specified time in a specified environment for a given purpose (Sommerville) • A recent survey of software consumers revealed that reliability was the most important quality attribute of the application software far@ucalgary.ca

28. Definition: Reliability /2 Three key points: • Reliability depends on how the software is used Therefore a model of usage is required • Reliability can be improved over time if certain bugs are fixed (reliability growth) Therefore a trend model (aggregation or regression) is needed • Failures may happen at random time Therefore a probabilistic model of failure is needed far@ucalgary.ca

29. Definition: Safety • Safety: absence of catastrophic consequences on the users and the environment • Safety is an extension of reliability: safety is reliability with respect to catastrophic failures. • When the state of correct service and the states of incorrect service due to non-catastrophic failure are grouped into a safe state (in the sense of being free from catastrophic damage, not from danger), safety is a measure of continuous safeness, or equivalently, of the time to catastrophic failure. far@ucalgary.ca

30. Dependability: Means • Fault prevention: how to prevent the occurrence or introduction of faults • Fault tolerance: how to deliver correct service in the presence of faults • Fault removal: how to reduce the number or severity of faults • Fault forecasting: how to estimate the present number, the future incidence, and the likely consequences of faults far@ucalgary.ca

31. Definition: Fault Prevention • To avoid fault occurrences by construction. • Fault prevention is attained by quality control techniques employed during the design and manufacturing of software. • Fault prevention intends to prevent operational physical faults. • Example techniques: design review, modularization, consistency checking, structured programming, etc. far@ucalgary.ca

32. Fault Prevention • Activities: • Requirement review • Design review • Clear code • Establishing standards (ISO 9000-3, etc.) • Using CASE tools with built-in check mechanisms All these activities are included in ISO 9000-3: Guidelines for application of ISO 9001 to the development, supply and maintenance of software (1991) far@ucalgary.ca

33. Definition: Fault Tolerance • A fault-tolerant computing system is capable of providing specified services in the presence of a bounded number of failures • Use of techniques to enable continued delivery of service during system operation • It is generally implemented by error detection and subsequent system recovery • Based on the principle of: • Act during operation while • Defined during specification and design far@ucalgary.ca

34. Fault Tolerance Process 1. Detection • Identify faults and their causes (errors) 2. Assessment • Assess the extent to which the system state has been damaged or corrupted. 3. Recovery • Remain operational or regain operational status 4. Fault treatment and continued service • Locate and repair the fault to prevent another occurence far@ucalgary.ca

35. Definition: Fault Removal /1 • Fault removal is performed both during the development phase, and during the operational life of a system. • Fault removal during the development phase of a system life-cycle consists of three steps: verification, diagnosis, correction. • Verification is the process of checking whether the system adheres to given properties, termed the verification conditions. If it does not, the other two steps follow: diagnosing the faults that prevented the verification conditions from being fulfilled, and then performing the necessary corrections. far@ucalgary.ca

36. Definition: Fault Removal /2 • After correction, the verification process should be repeated in order to check that fault removal had no undesired consequences; the verification performed at this stage is usually termed non-regression verification. • Checking the specification is usually referred to as validation. Uncovering specification faults can happen at any stage of the development, either during the specification phase itself, or during subsequent phases when evidence is found that the system will not implement its function, or that the implementation cannot be achieved in a cost effective way. far@ucalgary.ca

37. Definition: Fault Forecasting • Fault forecasting is conducted by performing an evaluation of the system behaviour with respect to fault occurrence or activation • Evaluation has two aspects: • qualitative, or ordinal, evaluation, which aims to identify, classify, rank the failure modes, or the event combinations (component failures or environmental conditions) that would lead to system failures • quantitative, or probabilistic, evaluation, which aims to evaluate in terms of probabilities the extent to which some of the attributes of dependability are satisfied; those attributes are then viewed as measures of dependability far@ucalgary.ca

38. Quality Models: CUPRIMDA • Quality parameters (parameters for fitness): • Capability • Usability • Performance • Reliability • Installability • Maintainability • Documentation • Availability Reference: S.H. Kan (1995) far@ucalgary.ca

39. Quality Models: Boehm’s far@ucalgary.ca

40. Quality Models: McCall’s far@ucalgary.ca

41. far@ucalgary.ca

42. Part 1 Section 2 Software and Hardware Reliability far@ucalgary.ca

43. Reliability Theory • Reliability theory developed apart from the mainstream of probability and statistics, and was used primarily as a tool to help nineteenth century maritime and life insurance companies compute profitable rates to charge their customers. Even today, the terms “failure rate” and “hazard rate” are often used interchangeably. From Engineering Statistics Handbook far@ucalgary.ca

44. Reliability: Natural System • Natural system life cycle. • Aging effect: Life span of a natural system is limited by the maximum reproduction rate of the cells. Figure from Pressman’s book far@ucalgary.ca

45. Reliability: Hardware • Hardware life cycle. • Useful life span of a hardware system is limited by the age (wear out) of the system. Figure from Pressman’s book far@ucalgary.ca

46. Reliability: Software • Software life cycle. • Software systems are changed (updated) many times during their life cycle. • Each update adds to the structural deterioration of the software system. Figure from Pressman’s book far@ucalgary.ca

47. Software vs. Hardware • Software reliability doesn’t decrease with time, i.e., software doesn’t wear out. • Hardware faults are mostly physical faults, e.g., fatigue. • Software faults are mostly design faults which are harder to measure, model, detect and correct. far@ucalgary.ca

48. Software vs. Hardware • Hardware failure can be “fixed” by replacing a faulty component with an identical one, therefore no reliability growth. • Software problems can be “fixed” by changing the code in order to have the failure not happen again, therefore reliability growth is present. • Software does not go through production phase the same way as hardware does. • Conclusion:hardware reliability models may not be used identically for software. far@ucalgary.ca

49. Reliability: Science • Exploring ways of implementing “reliability” in software products. • Reliability Science’s goals: • Developing “models” (regression and aggregation models) and “techniques” to build reliable software. • Testing such models and techniques for adequacy, soundness and completeness. far@ucalgary.ca

50. What is Engineering? • Engineering = • Analysis + • Design + • Construction + • Verification + • Management • What is the problem to be solved? • What characters of the entity are used to solve the problem? • How will the entity be realized? • How is it constructed? • What approach is used to uncover errors in design and construction? • How will the entity be supported in the long term? far@ucalgary.ca