1 / 24

Chapter 3 Quality Assurance

Chapter 3 Quality Assurance. SE 471 - Software Testing and Quality Assurance. 1. Overview. Classification: QA as Dealing with Defects Defect Prevention Education and training Formal method Other defect prevention techniques Defect Reduction

ocean
Download Presentation

Chapter 3 Quality Assurance

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Chapter 3Quality Assurance SE 471 - Software Testing and Quality Assurance 1

  2. Overview • Classification: QA as Dealing with Defects • Defect Prevention • Education and training • Formal method • Other defect prevention techniques • Defect Reduction • Inspection: Direct fault detection and removal • Testing: Failure observation and fault removal • Other techniques and risk identification • Defect Containment • Software fault tolerance • Safety assurance and failure containment 2

  3. Quality Assurance from the Correctness Centered Viewpoint (CCV) • According to CCV, Quality means: • few, if any, defects remain in the software system when delivered • the remaining defects cause minimal damage • With the correctness centered quality definition given above, we can view different QA activities as attempting to • prevent, • reduce, or • contain the damage • Briefly discussed here - detailed descriptions are presented in later chapters 3

  4. Dealing with Pre-/Post-release Defects • Focus is to remove as many defects as possible before product release - why? • Cost • Reputation • Pre-release QA activities • Defect prevention, detection, removal • Several approaches including development methodologies, programming techniques, defect removing tools, testing, beta tests, etc 4

  5. Dealing with Pre-/Post-release Defects • Dormant defects: Faults left in the finished software • May stay dormant for sometime • Potential to cause problem • After release QA activities • Aim at minimizing the negative impact of the remaining faults during operational use after product release • Fixing defects reported by customers • Failure containment: • Most defect containment techniques involve many redundancies • Require significantly more development effort to design/implement • typically limited to the situations where failures are associated with substantial damage 5

  6. Graphical Depiction of the Classification Scheme • QA activities act as Barriers - dotted lines • Each barrier • removes or blocks defect sources • prevents undesirable consequences Defect Prevention Activities Defect Removal Activities Fault Tolerance Activities 6

  7. Classification of QA Approaches • We can classify these QA alternatives into the following three generic categories: • Defect prevention • Defect reduction • Defect containment 7

  8. Defect Prevention • Defect prevention: can be done in two ways: • Eliminating certain error sources. Examples: • Eliminating ambiguities - SRS • Correcting human misconception – lack of knowledge • Fault prevention or blocking missing human actions through the use of certain tools or technologies. Examples: • Missing to put ‘{‘ or ‘(‘ can be avoided by using “Syntax-Directed Editors”. Automatically balances out parenthesis 8

  9. Defect Prevention • There are known error sources or missing/ incorrect actions that result in certain fault injections • So? What is needed? • Root cause analysis • Find root causes for injected or potential faults • Error source & Remedy • If human misconceptions are the error sources → education and training • If imprecise designs and implementations that deviate from product specifications → formal methods • If non-conformance to selected processes or standards is the problem → process conformance/standard enforcement • Certain tools or technologies may reduce fault injections 9

  10. Defect Prevention • Education & Training • Product and domain specific knowledge • developers unfamiliar with embedded software may design software without considering its environmental constraints - interface problems • General software development knowledge and expertise • lack of expertise with requirement analysis - rework in subsequent design, coding, and testing activities • Knowledge about the specific development methodologies and tools used by the organization • In Cleanroom technology if the developers are not familiar with the key components of formal verification or statistical testing - little chance for producing high-quality products • Knowledge about the software process used by the organization • incremental software development - uncoordinated development may lead to many interface or interaction problems 10

  11. Defect Prevention • Formal Method • Formal specification and formal verification • Formal specification is concerned with producing an unambiguous set of product specification • Formal verification checks the conformance of software design or code against these formal specifications • Examples • Axiomatic approach • Planguage, Z notation, VDM • Most influential formal method is the axiomatic approach • Meaning of program statements is abstracted as a set of axioms • Rules are used to compose axioms to build up proofs of entire program correctness • Program behavior is explained as a set of pre-conditions and post-conditions • Obstacle to formal methods: high cost (automated support needed), difficult task (human intensive activities) 11

  12. Defect Prevention • Process conformance/standard enforcement may reduce fault injections. Examples: • A better managed process can also eliminate many systematic problems e.g. no process for configuration management may lead to inconsistencies • Lean Software – avoid low quality ‘fat’ software • Certain Tools & Technologies may reduce fault injections. Examples: • Tools, e.g. Syntax directed editor automatically balances out each open parenthesis 12

  13. Defect Reduction • Defect prevention activities to be 100% effective – an unrealistic expectation • Need effective techniques to remove as many of the injected faults as possible • Defect reduction: detect and remove faults once they have been injected. • Done in two ways: • Static Analysis: e.g. Inspection of software code, design etc • Dynamic Analysis: e.g. Testing of programs by executing 13

  14. Defect Reduction • Inspection • Critical reading and analysis of software code or other software artifacts, such as designs, product specifications, test plans, user manuals etc • Typically conducted by multiple human inspectors • Identified faults need to be removed & removal verified • Used throughout the development process • An economical QA alternative • Two ways to do it: Informal & Formal • Informal reviews: • self conducted reviews/inspections • Formal inspections: • Fagan inspection - most influential work in inspection • Gilb inspection • Inspection processes vary, typically include some planning & follow up activities 14

  15. Defect Reduction • Testing • most important parts of QA • the most commonly performed QA activity • Involves execution & observation • When to start • Logically testing can only take place after implementation • However, preparation can start earlier • In the incremental or iterative process, testing can usually get started much earlier • can be divided into various sub-phases starting from the coding phase up to post-release product support, including: unit testing, component testing, integration testing, system testing, acceptance testing, beta testing, etc 15

  16. Defect Reduction • Testing • What to test • Black-box (or functional) testing verifies the correct handling of the external functions • White-box (or structural) testing verifies the correct implementation of internal units, structures, and relations among them • White-box test examines source code with focus on: • Control flow • Data flow 16

  17. Defect Reduction • Other static & dynamic techniques for defect reduction • Static: formal model based analysis techniques • Algorithm analysis, boundary value analysis, finite state machine, control and data flow analysis, software fault trees etc. • Dynamic: • Simulation and prototyping • Timing and performance analysis for real-time systems • Accident analysis and reconstruction using software fault trees and event trees for safety critical systems • Some Risk identification techniques Will be covered later in detail 17

  18. Defect Containment • Defect reduction activities can only reduce the number of faults to a fairly low level, but not completely eliminate them • Still a problem for software systems where failure impact is substantial such as many real-time control software sub-systems used in medical, nuclear, transportation • Few remaining (dormant) faults may be triggered under rare conditions or unusual dynamic scenarios 18

  19. Defect Containment • Unrealistic to expect planning of huge number of test cases • Some other means are needed to: • Either contain the damage to local areas so that there is no global failures observable to users • Or limit the damage caused by the software system failure 19

  20. Defect Containment • Done in two ways: • Using Fault Tolerance Techniques: • Motivation • Traditional hardware: spare parts and backup units • Fault Tolerance Techniques - Examples • Recovery blocks: rollback and redo • NVP: N-version programming (inspired by hw FT) • Containment Measures: • Avoid catastrophic consequences e.g. death etc • Examples: • Failure containment for real time control software used in nuclear reactors may include concrete walls to encircle and contain radioactive materials in case of reactor melt-down due to software failure 20

  21. Defect Containment • Fault Tolerance (FT) VS Exception Handling (EH): • EH deviates from the specification • FT attempts to provide services compliant with the specification after detecting a fault • Important to realize the difference between trying to construct robust software versus trying to construct reliable software • Reliable software by FT or EH? • High profile disaster due to uncaught exception: • On June 4, 1996, the maiden flight of the European Ariane 5 launcher crashed about 40 seconds after takeoff. • The loss was $500,000,000 uninsured • An uncaught exception caused entire software to crash 21

  22. Defect Containment • Safety Assurance and Fault Containment • Prevent accidents from happening • accident: failure with severe consequences • In addition to the QA techniques discussed before, various specific techniques are also used for safety critical systems based on analysis of hazards • Hazard elimination: • eliminate hazard sources • Hazard reduction: • reduce the probability of hazard • Hazard control: • reduce the severity of failures • Damage control: • reduce the severity of damage 22

  23. Self-Study Assignment – Included in Exam • Read article “Quantifying Quality Requirements Using Planguage”, understand it and practice it for simple examples. Its easy & interesting! • Planguage is derived from Planning & Language. • Designed to quantify qualitative statements in plans, specifications, and designs, Planguage is a keyword-driven language that allows measurable, testable quality requirements to be written. • Planguage has many benefits; it is easy to learn, flexible, compact, extensible, and prevents omissions by providing a consistent set of parameters for quality requirements. • In this article, Planguage keywords and syntax are introduced. Examples of quality requirements before and after using Planguage are given, and the experiences of introducing Planguage are discussed. • You may be given a question in exam/quiz! 23

  24. Homework No 2 • Read article “Design by Contract: The Lessons of Ariane” from: http://archive.eiffel.com/doc/manuals/technology/contract/ariane/ • Then read another interesting article which is in fact the critique of the above one titled “Critique of ‘Put it in the contract: The lessons of Ariane’” from: http://home.flash.net/~kennieg/ariane.html 24

More Related