1 / 40

INFO 631 Prof . Glenn Booker

INFO 631 Prof . Glenn Booker. Week 1 – Defect Analysis and Removal. Motivation. Much of software measurement is devoted to improving the quality of the product To do so, it helps to understand how, when, and why mistakes are made during the software life cycle, resulting in defects

rhett
Download Presentation

INFO 631 Prof . Glenn Booker

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. INFO 631Prof. Glenn Booker Week 1 – Defect Analysis and Removal INFO631 Week 1

  2. Motivation • Much of software measurement is devoted to improving the quality of the product • To do so, it helps to understand how, when, and why mistakes are made during the software life cycle, resulting in defects • It is assumed that defects are queued for being fixed (removed) after they are detected INFO631 Week 1

  3. Defect Analysis Overview • Defect analysis looks at: • When a defect was created (“injected”) • When a defect was found (“detection”) • What caused a defect (type of defect, and/or orthogonal defect classification) • How a defect was found (“triggers”) • Yes, there are often many terms for the same activity! INFO631 Week 1

  4. Defect Creation (Injection) • Defects may be created, detected, and removed during every phase of the software life cycle, including during maintenance • Most are created during requirements analysis, design, and coding • Testing and maintenance create relatively few defects INFO631 Week 1

  5. Requirements Analysis High Level Design Low Level (Detailed) Design Coding Component (Integration) Testing System Testing Unit Testing Maintenance Waterfall Life Cycle INFO631 Week 1

  6. Defect Detection • Defects may be found: • During quality assurance activities between phases of the life cycle (such as during major reviews and inspections); and often • Within each phase (during preliminary reviews, peer reviews and testing activities) • Here we mostly consider the former (between-phase activities) INFO631 Week 1

  7. Defect Detection • Defects are found (discovered) by • Reviews and formal inspections • For requirements analysis, high level design, low level design, and coding • Testing • Unit testing • Integration testing • System testing • Acceptance testing INFO631 Week 1

  8. Origin of Defects & Type • What kind of defects are made in each life cycle phase? • Requirements - incorrect specification; missing requirements • High level design - design does not cover all requirements, or is inflexible • Low level design - mismatch between HLD and LLD; design does not cover all requirements INFO631 Week 1

  9. Origin of Defects & Type • Coding - code errors • Integration – interface, compatibility problems • Unit testing - bad fixes • Component testing - bad fixes • System testing - bad fixes • Acceptance testing - bad fixes • Maintenance (enhancement) - bad requirements, design, coding, testing INFO631 Week 1

  10. Defect Causal Analysis • Regardless of when defects are created or discovered, they may be analyzed to determine their cause • Causes may range from the mundane to the esoteric • Once identified, causes should be recorded to support defect prevention activities INFO631 Week 1

  11. Types of Defect Causes • Some causes may include: • Unclear requirements • Incorrect architecture • Incomplete design • Unfamiliarity with programming language • Typos • Inadequate understanding of interfaces • Inadequate understanding of standards (e.g. TCP/IP, ODBC, SQL, etc.) INFO631 Week 1

  12. Types of Defect Causes • and: • Poor documentation for legacy system • Design based on outdated requirements • Code written to outdated design • Poor design for future expansion (e.g. hardwired constants, buried assumptions, etc.) • Conflicting requirements • Bad fix of a previous problem • …and many more INFO631 Week 1

  13. Orthogonal Defect Classification • ODC is a formal type of defect causal analysis • Classify defects by the type of defect, and the life cycle phase when it was created • This particular scheme is still somewhat experimental, but the basic concept is widely accepted INFO631 Week 1

  14. ODC - Defect Type • A possible set of definitions: • Function error–affects capability, user, product or hardware interfaces, global data structures • Assignment error – errors with initialization of control blocks or data structure • Interface error – errors in interacting with other components, modules, or or device drivers • Checking – errors in validating data and values INFO631 Week 1

  15. ODC - Defect Type • Timing/serialization – errors in management of shared and real-time resources • Build/package/merge – errors due to mistakes in library systems, change management, or version control • Documentation – errors in publications and maintenance notes • Algorithm – errors regarding efficiency or correctness that affect the task INFO631 Week 1

  16. Orthogonal Defect Classification TypePhase Caused Functional Design Interface Low Level Design (LLD) Checking LLD or Coding Assignment Coding Timing LLD Build/package Library tools Documentation Publications Algorithm LLD INFO631 Week 1

  17. Defect Trigger • Is a condition that allows a defect to surface (become visible); examples include: • Design conformance • Logic or data flow incomplete • Workload or stress (performance) • Boundary conditions (extremes) • Bug fix • Recovery timing • User code INFO631 Week 1

  18. Defect Removal Modeling • Defect removal is critical to reducing (development or maintenance) cycle time and cost, and improving quality • Michael Fagan and Capers Jones are noteworthy authors • See, for example, “Fagan Inspections” INFO631 Week 1

  19. Defect Removal Model (DRM) • Software development phase-based DRM covers • Defect injection into a phase • Defect removal during a phase • Defect carryover across phases • Effectiveness of defect removal within a phase INFO631 Week 1

  20. Defect Removal Model • DRM is a quality management tool • Provides insights into the defect removal process and where this process might be improved • Used after a project is completed; provides post mortem information about that project • If defect removal process is similar for a new project, then the DRM can be used to improve the defect removal process of that new project INFO631 Week 1

  21. Defect Removal Model • Defects are created (injected) by incorrect requirements gathering, analysis, design, coding, or by bad fixes • Any given defect can be removed in the phase in which it was injected or in a later phase • Remove defect means fix bug, redo specs and/or design as needed INFO631 Week 1

  22. Defect Removal Effectiveness • Phase based defect detection activities • Requirements analysis & inspection • Design inspection • Code inspection, Build verification testing • Unit, string, integration, regression, system testing • Fix verification inspection & testing INFO631 Week 1

  23. Defects Injected (new mistakes) Undetected Defects Net Defects Defect Detection (inspection) Life Cycle Phase Unfixed Defects + Bad Fix Defects NetDefectsfrom previous Phase Known Defects Bad Fix Defects Fixed Defects Defect Removal Model Applies to each life cycle phase INFO631 Week 1

  24. Defects found by removal operation ________________________________ X 100 Removal Effectiveness = Defects present at removal operation Defect Removal Effectiveness • For any given phase of defect removal • Defects present at removal operation • Defects found during removal operation + defects found later • May be determined ex post facto (after the fact), or using a statistical prediction model INFO631 Week 1

  25. Defect Removal Effectiveness • Defect removal effectiveness = (# of defects found by inspection) /(# of defects originally present) *100 • Early detection percentage = (# of major inspection errors) /(# of major and minor errors) * 100 INFO631 Week 1

  26. Defect Matrix Assumptions • Defects are removed in the same life cycle phase when they are found • No defects are knowingly left unfixed • No bad fixes • Or at least they are blended into the number of defects created in that life cycle phase INFO631 Week 1

  27. When Originated (injected, or created) When Found/Fixed 0 0 1 Sample Defect Matrix—When Originated vs. When Found INFO631 Week 1

  28. Defects Created this Phase Defects passed from Previous life cycle phases Defects passed to Next life cycle phase Life cycle Phase(s) Defects Found & removed this Phase Defect Removal Effectiveness Next = (Previous + Created) – FoundDRE = Found / (Previous + Created) * 100 INFO631 Week 1

  29. High Level Design Effectiveness • There are no requirements defects removed; 122 defects are passed to HLD • High (Top) Level Design (I0) Inspection Effectiveness • Defects found and removed at I0: 730 • Defects existing on step entry (escapes from requirements phase: 122 • Defects injected in current phase: 859 • E(I0) = 730/(122+859) x 100 = 74% INFO631 Week 1

  30. Low Level Design Effectiveness • Low Level Design (I1) Inspection Effectiveness • Defects found and removed at I1: 729 • Defects existing on step entry (escapes from requirements phase and I0): 122+859-730 = 251 • Defects injected in current phase: 939 • E(I1) = 729/(251+939) x 100 = 61% INFO631 Week 1

  31. Code Inspection Effectiveness • Code Inspection (I2) Effectiveness • Defects found and removed at I2: 1095 • Defects present on step entry (escapes from requirements phase, I0, and I1): 251+939-729 = 461 • Defects injected in current phase: 1537 • E(I2)= 1095/(461+1537) x 100 = 55% INFO631 Week 1

  32. Unit Testing Effectiveness • Unit Testing (UT) Effectiveness • Defects found and removed at UT: 332 • Defects existing on step entry (escapes from previous phases): 461+1537-1095 = 903 • Defects injected in current phase (bad fixes): 2 • E(UT) = 332/(903+2) x 100 = 37% • Can follow the same pattern for the other testing phases and post-release INFO631 Week 1

  33. Summary Effectiveness Measures • Overall Design & Coding Inspection Effectiveness • IE = (730+729+1095)/(122+859+939+1537) x 100 = 74% • Overall Effectiveness of all Testing activities • TE = (332+387+111)/(903+2+4+1)x100 = 91% • Overall Defect Removal Effectiveness of the development process (not including release) • DRE = (0+730+729+1095+332+387+111)/(122+859+ 939+1537+2+4+1) x 100 = 3384/3464*100 = 97.7% INFO631 Week 1

  34. Rayleigh Model • The Rayleigh Model describes the number of defects which will be discovered, by development phase • It’s a special case of the Weibull family of distributions, which we’ll cover later INFO631 Week 1

  35. Rayleigh Model Assumptions • Defect rate observed during development process is positively correlated with defect rate in field • Given the same defect injection rate, if more defects are discovered and removed earlier, fewer will remain in later stages, leading to fewer defects in the field INFO631 Week 1

  36. Rayleigh Model Inspections • I0 = High Level Design Inspection • I1 = Low Level Design Inspection • I2 = Code Inspection • UT = Unit Testing • CT = Component (Integration) Testing • ST = System Testing • GA = after General Availability (release or Fielding of the system) INFO631 Week 1

  37. Number of Defects Development Phase I0 I1 I2 UT CT ST GA Rayleigh Model Each bar represents the number of defects found during that life cycle phase’s inspection activity INFO631 Week 1

  38. Desired Rayleigh Curve Trends Effect of Early Defect Removal and Reducing Error Injection INFO631 Week 1

  39. Cost of Defect Removal by Phase • Defect removal and rework is less costly the closer that the defects are found relative to the phase in which they are injected • Rework in the I0, I1, and I2 inspection levels can be 10 to 100 times less expensive than if it is done during formal testing • Reviews can reduce number of defects reaching testing phases by factor of 10 • These reductions cut testing costs by 50-80%, even including the review costs INFO631 Week 1

  40. Cost Effectiveness of DRM • Relative cost of fixing a problem found in design/coding, testing, or after release are: 1:20:82 (Remus, 1983) 1:13:92 (Kan, 1989) • Cost of defect removal can be analyzed by inspection type, testing phase, defect severity, defect origin, etc. INFO631 Week 1

More Related