Presentation Transcript

1. SE 501 Software Development Processes

   Dr. Basit Qureshi College of Computer Science and Information Systems Prince Sultan University Lecture for Week 12
2. Contents Software Quality Assurance? Reviews, Inspection… The cost of quality? Defect Tracking Systems Software Reliability Engineering Configuration Management SCM Repository SCM Process Quality SCM SE 501 Dr. Basit Qureshi: Lecture for Week 12
3. Bibliography Roger Pressman, Software Engineering: A practitioners approach, MC GrawHill, 2009. Dave Penny, Software Engineering Course at utoronto.ca Ian Sommerville, Software Engineering, 9th edition, Addison Wesley, 2010. J. Li, T. Stålhane, R. Conradi, Jan M.W. Kristiansen, "Enhancing Defect Tracking Systems to Facilitate Software Quality Improvement", IEEE Software, Vol. 29, No.2, pp. 59-66, 2012. SE 501 Dr. Basit Qureshi: Lecture for Week 12
4. Software quality assurance SE 501 Dr. Basit Qureshi: Lecture for Week 12
5. Software Quality Assurance Quality “A characteristic or attribute of something” Measurablecharacteristics that we can compare to known standards In software it involves such measures as cyclo-maticcomplexity, cohesion, coupling, function points, and source lines of code Includes variation control A software development organization should strive to minimize the variation between the predicted and the actual values for cost, schedule, and resources They should make sure their testing program covers a known percentage of the software from one release to another One goal is to ensure that the variance in the number of bugs is also minimized from one release to another SE 501 Dr. Basit Qureshi: Lecture for Week 12
6. Software Quality Assurance Two kinds of quality are sought out Quality of design The characteristic that designers specify for an item This encompasses requirements, specifications, and the design of the system Quality of conformance (i.e., implementation) The degree to which the design specifications are followed during manufacturing This focuses on how well the implementation follows the design and how well the resulting system meets its requirements Quality also can be looked at in terms of user satisfactionUser satisfaction = compliant product + good quality + delivery within budget and schedule SE 501 Dr. Basit Qureshi: Lecture for Week 12
7. Software Quality Assurance Quality Control: Involves a series of inspections, reviews, and tests used throughout the software process Ensures that each work product meets the requirements placed on it Includes a feedback loop to the process that created the work product This is essential in minimizing the errors produced Combines measurement and feedback in order to adjust the process when product specifications are not met Requires all work products to have defined, measurable specifications to which practitioners may compare to the output of each process SE 501 Dr. Basit Qureshi: Lecture for Week 12
8. Software Quality Assurance Software Quality Assurance: Serves as an umbrella activity that is applied throughout the software process Involves doing the software development correctly versus doing it over again Reduces the amount of re-work, which results in lower costs and improved time to market SE 501 Dr. Basit Qureshi: Lecture for Week 12
9. Software Quality Assurance Software Quality Assurance Process: Specific quality assurance and quality control tasks (including formal technical reviews and a multi-tiered testing strategy) Effective software engineering practices (methods and tools) Control of all software work products and the changes made to them A procedure to ensure compliance with software development standards “Measurement and reporting mechanisms” SE 501 Dr. Basit Qureshi: Lecture for Week 12
10. Software Quality Assurance Software Quality Assurance Functions: Consists of a set of auditing and reporting functions that assess the effectiveness and completeness of quality control activities Provides management personnel with data that provides insight into the quality of the products Alerts management personnel to quality problems so that they can apply the necessary resources to resolve quality issues SE 501 Dr. Basit Qureshi: Lecture for Week 12

11. Software Reviews

12. Purpose of Reviews Serve as a filter for the software process Are applied at various points during the software process Uncover errors that can then be removed Purify the software analysis, design, coding, and testing activities Catch large classes of errors that escape the originator more than other practitioners Include the formal technical review (also called a walkthrough or inspection) Acts as the most effective SQA filter Conducted by software engineers for software engineers Effectively uncovers errors and improves software quality Has been shown to be up to 75% effective in uncovering design flaws (which constitute 50-65% of all errors in software) Require the software engineers to expend time and effort, and the organization to cover the costs
13. Formal Technical Review (FTR) Objectives To uncover errors in function, logic, or implementation for any representation of the software To verify that the software under review meets its requirements To ensure that the software has been represented according to predefined standards To achieve software that is developed in a uniform manner To make projects more manageable Serves as a training ground for junior software engineers to observe different approaches to software analysis, design, and construction Promotes backup and continuity because a number of people become familiar with other parts of the software May sometimes be a sample-driven review Project managers must quantify those work products that are the primary targets for formal technical reviews The sample of products that are reviewed must be representative of the products as a whole
14. The FTR Meeting Has the following constraints From 3-5 people should be involved Advance preparation (i.e., reading) should occur for each participant but should require no more than two hours a piece and involve only a small subset of components The duration of the meeting should be less than two hours Focuses on a specific work product (a software requirements specification, a detailed design, a source code listing) Activities before the meeting The producer informs the project manager that a work product is complete and ready for review The project manager contacts a review leader, who evaluates the product for readiness, generates copies of product materials, and distributes them to the reviewers for advance preparation Each reviewer spends one to two hours reviewing the product and making notes before the actual review meeting The review leader establishes an agenda for the review meeting and schedules the time and location (More on next slide)
15. The FTR Meeting (continued) Activities during the meeting The meeting is attended by the review leader, all reviewers, and the producer One of the reviewers also serves as the recorder for all issues and decisions concerning the product After a brief introduction by the review leader, the producer proceeds to "walk through" the work product while reviewers ask questions and raise issues The recorder notes any valid problems or errors that are discovered; no time or effort is spent in this meeting to solve any of these problems or errors Activities at the conclusion of the meeting All attendees must decide whether to Accept the product without further modification Reject the product due to severe errors (After these errors are corrected, another review will then occur) Accept the product provisionally (Minor errors need to be corrected but no additional review is required) All attendees then complete a sign-off in which they indicate that they took part in the review and that they concur with the findings (More on next slide)
16. The FTR Meeting (continued) Activities following the meeting The recorder produces a list of review issues that Identifies problem areas within the product Serves as an action item checklist to guide the producer in making corrections The recorder includes the list in an FTR summary report This one to two-page report describes what was reviewed, who reviewed it, and what were the findings and conclusions The review leader follows up on the findings to ensure that the producer makes the requested corrections
17. FTR Guidelines Review the product, not the producer Set an agenda and maintain it Limit debate and rebuttal; conduct in-depth discussions off-line Enunciate problem areas, but don't attempt to solve the problem noted Take written notes; utilize a wall board to capture comments Limit the number of participants and insist upon advance preparation Develop a checklist for each product in order to structure and focus the review Allocate resources and schedule time for FTRs Conduct meaningful training for all reviewers Review your earlier reviews to improve the overall review process
18. The Cost of Quality Includes all costs incurred in the pursuit of quality or in performing quality-related activities Is studied to Provide a baseline for the current cost of quality Identify opportunities for reducing the cost of quality Provide a normalized basis of comparison (which is usually dollars) Involves various kinds of quality costs (See next slide) Increases dramatically as the activities progress from Prevention  Detection  Internal failure  External failure "It takes less time to do a thing right than to explain why you did it wrong." Longfellow
19. Kinds of Quality Costs Prevention costs Quality planning, formal technical reviews, test equipment, training Appraisal costs Inspections, equipment calibration and maintenance, testing Failure costs – subdivided into internal failure costs and external failure costs Internal failure costs Incurred when an error is detected in a product prior to shipment Include rework, repair, and failure mode analysis External failure costs Involves defects found after the product has been shipped Include complaint resolution, product return and replacement, help line support, and warranty work
20. Six Sigma Popularized by Motorola in the 1980s Is the most widely used strategy for statistical quality assurance Uses data and statistical analysis to measure and improve a company's operational performance Identifies and eliminates defects in manufacturing and service-related processes The "Six Sigma" refers to six standard deviations (3.4 defects per a million occurrences) (More on next slide)
21. Six Sigma (continued) Three core steps Define customer requirements, deliverables, and project goals via well-defined methods of customer communication Measure the existing process and its output to determine current quality performance (collect defect metrics) Analyze defect metrics and determine the vital few causes (the 20%) Two additional steps are added for existing processes (and can be done in parallel) Improve the process by eliminating the root causes of defects Control the process to ensure that future work does not reintroduce the causes of defects
22. Six Sigma (continued) All of these steps need to be performed so that you can manage the process to accomplish something You cannot effectively manage and improve a process until you first do these steps (in this order): Manage and improve the work process Control the work process Analyze the work process Measure the work process Define the work process The work to be done

23. Defect Tracking

    Lecture 9
24. Defect Tracking Keeping track of all the defects that have been discovered Keeping track of all the steps required to validate, correct, and take preventative action for a defect Necessary because to not lose any reported defects to co-ordinate defect resolution to ensure coders don’t work on non-defects Features masquerading as defects Wasting time fixing something that isn’t broken Wasting time chasing down a badly reported defect to control defect correction activity ensure the right defects are being worked on In practice: A database of defect records A workflow driven by the state and owner fields. Lecture 9
25. Aside on “Bugs” Why not call defects “bugs”? First used because a moth flew into a vacuum-tube computer and ate away at the wires. “bugs” are things that happen to you, outside of your control A “defect” is something that is caused by a coder making a correctable mistake. Gets you into the right mindset Lecture 9
26. Defect Information Where Found product, release, version, hardware, os, drivers, general area Who Found It customer, internal, when Description of the Defect summary, description, how to reproduce, associated data links to related defects or features Triage severity, likelihood → priority Audit Trail all changes to the defect data, by whom, when State state, owner Lecture 9
27. Priority Matrix Submitter of defect chooses severity and likelihood May later correct if determined to be an exaggeration or in error System assigns a priority according to the priority matrix Humans may change the priority using their judgment No need to stick to “the matrix”, which is after all too simple to account for every contingency Lecture 9
28. customer QA Defect Workflow WIP Valid Fixed Closed New Disputed issue Lecture 9
29. Coder Assignment Matrix Defect is auto-assigned to a coder based upon The product in which it was found The functional area of the defect Catch-all category (misc.) goes to a team lead charged with defect assignment and overview for assignment elsewhere. Keeps track of the defect load by priority on all coders Balanced the load Chips in where needed Developers may move the defect to the appropriate coder without management permission. May also move to team lead for re-assignment Natural corollary to auto-assignment. Lecture 9
30. Management Controls One of main purposes is to provide defect visibility to enable management to ensure defects are appropriately prioritized. Management must: overview all active defect records ensure priorities are good if languishing too long in a given state, act ensure coders are working on defects of appropriate priority at any given time System Support Most systems can be configured to send e-mail and/or re-assign to manager when certain conditional action thresholds are reached E.g., prio 1 defect with state unchanged for 24 hrs. Post daily reports of overdue defects Lecture 9
31. Controls on the System Most defect tracking systems allow permissions to be set up. each user is given various group memberships: coders, testers, managers, builders, … Permissions can then be set up by group, state, field Don’t do it! Q. what are you trying to control? A. source code Putting restrictions on defect control system will not help you to gain control of the source it will hurt coders will work around silly security restrictions defect system will not accurately reflect what is being worked on Dirty data will go uncorrected Lecture 9
32. Metrics Another purpose for defect tracking is to enable gathering of good, clean defect arrival/departure data. Gives insight into productivity of coders fixing defects testers finding defects Clean data is essential e.g., if no way to validate defects lots of arrivals may be due to bad code or to bad defect triage may expend a lot of effort on coding initiatives and numbers will go the wrong way! Must quickly get defects out of New and Fixed. Arrivals: defects per day entering into Valid Departures: defects per day going from Fixed to Closed Total: sum of defects in states Valid, WIP, and Fixed. Lecture 9
33. Metrics WIP arrivals Valid departures Fixed HURRY! HURRY! HURRY! Closed New Disputed Lecture 9
34. 100 80 60 total arrivals 40 defects departures net 20 0 1 3 5 7 9 11 13 15 17 19 21 23 25 -20 days Metrics Example Lecture 9
35. Towards GA These metrics should be tracked: by product by priority Company should establish shipping thresholds e.g., no known priority 1 or 2 defects arrival rate for priority 1-3 < 1 defect per day Watch trends, compare to last release. If bad: “bug Olympics” “bug blitz weekends” slip date clean up the architecture Lecture 9
36. Depot Change Report Lecture 9
37. Defect Attribution Beginning to understand what are the systemic root causes of defects. Include as data in the defect tracking system that must be there before defect is closed Should record time taken to deal with it, or at least a “difficulty” field (high, medium, low) Attribute to: where in the source code can identify modules whose re-design will add most bang-for-the-buck which developer introduced it organizationally tricky but very useful during what phase spec, design, code Lecture 9
38. Customer Issue Tracking Distinct from defect tracking Customers have many issues: how to use software installation issues perceived problems problems that have already been resolved in a previous patch known issues ship me a manual, please … Some of these issues will result in new defects Requirements of issue tracking systems will include: customer relationship management tie-in searchable knowledge bases customer tracking of issue progress … Lecture 9
39. Shipping Known Defects 0-defects is not a sustainable software business how many defects are acceptable? how many are you shipping? Defect seeding inject defects, see how many are found, use the ratio hard to work this in practice Must measure customer satisfaction with perceived level of defects and correlate to known defects at ship. e.g., If we ship with 350 known defects and customers are down on the release then it’s too high If we ship with 50 and customers say “best release ever” super stable, then it’s good. Might want to use 50 as the shipping threshold, and then gradually lower that over time Lecture 9
40. Testing/Coding Effort Changes Can only compare across releases if have a consistent testing effort same number of testers, same productivity, same time, same general size of the release If increase size of testing team relative to coding team, ratio of known to unknown defects decreases Assume ratio is 50% ship with 50 known, actually shipping 100 defects Add testers, raising ratio to 75% ship with 75 known, actually shipping 100 defects good to know. If increasing testing effort without increasing coding efforts, will be hard-pressed to meet the old thresholds Add coders, lowering ratio to 25% ship with 25 known, actually shipping 100 defects Add coders and testers ratios stay the same but will reach the thresholds faster for the same sized effort Lecture 9
41. Release Notes When shipping point releases, good to say which defects are fixed hard to get this info! Start with sccs and defect tracking to see which defect corrections have been checked in since the last point release Must describe the defect in terms the users will understand e.g., load this data file it crashes good enough to find and fix the defect not good enough for release notes must track down the root cause, and extrapolate into what kind of situations will trigger the defect. If doing this, must make it a part of the defect correction process Lecture 9
42. Automated Patching Facilities Ability for the software to query a server to see if it is up-to-date. If not, then download an appropriate, ideally small, patch and apply it Distinguish “critical” from “optional” Run immediately after install Facility must be able to chain patches Determine smallest download combo to get you from where you are to current version Need excellent build/release disciplines to ensure release numbers completely identify the file set will want to provide binary diff files as patches – need to be sure will dbl-check a checksum on all files before applying anything! Lecture 9
43. Automated Patching Technology A patch always starts with a complete image of the software installed into the file system. A normal, regular release Test it as such Use a patching utility to generate a binary diff patch Point at release A and release Z Will generate a small patch self-installer that will move you from A to Z Point at releases A,B,C and Z Will generate a somewhat larger patch self-installer that is capable of moving the software form any of the releases A,B, or C to Z Larger, but saves due to common files between A,B,C If no common files, is a waste. End user may have to download: Patch 1: from A to W Patch 2: from W to Z Lecture 9
44. Reliability and Availability Software failure Defined: Nonconformance to software requirements Given a set of valid requirements, all software failures can be traced to design or implementation problems (i.e., nothing wears out like it does in hardware) Software reliability Defined: The probability of failure-free operation of a software application in a specified environment for a specified time Estimated using historical and development data A simple measure is MTBF = MTTF + MTTR = Uptime + Downtime Example: MTBF = 68 days + 3 days = 71 days Failures per 100 days = (1/71) * 100 = 1.4 Software availability Defined: The probability that a software application is operating according to requirements at a given point in time Availability = [MTTF/ (MTTF + MTTR)] * 100% Example: Avail. = [68 days / (68 days + 3 days)] * 100 % = 96%
45. Configuration management
46. Configuration management Configuration management (CM) is a process for establishing and maintaining consistency of a product’s performance and functional and physical attributes with its requirements, design and operational information throughout its life. The CM process is widely used by military engineering organizations to manage complex systems, such as weapon systems, vehicles, and information systems SE 501 Dr. Basit Qureshi: Lecture for Week 12
47. Configuration management In software engineering, software configuration management (SCM) is the task of tracking and controlling changes in the software. “Somebody did something, how can one reproduce it?” SE 501 Dr. Basit Qureshi: Lecture for Week 12
48. What is Change Management Also called software configuration management (SCM) It is an umbrella activity that is applied throughout the software process It's goal is to maximize productivity by minimizing mistakes caused by confusion when coordinating software development SCM identifies, organizes, and controls modifications to the software being built by a software development team SCM activities are formulated to identify change, control change, ensure that change is being properly implemented, and report changes to others who may have an interest (More on next slide)
49. What is Change Management (continued) SCM is initiated when the project begins and terminates when the software is taken out of operation View of SCM from various roles Project manager -> an auditing mechanism SCM manager -> a controlling, tracking, and policy making mechanism Software engineer -> a changing, building, and access control mechanism Customer -> a quality assurance and product identification mechanism
50. Software Configuration The Output from the software process makes up the software configuration Computer programs (both source code files and executable files) Work products that describe the computer programs (documents targeted at both technical practitioners and users) Data (contained within the programs themselves or in external files) The major danger to a software configuration is change First Law of System Engineering: "No matter where you are in the system life cycle, the system will change, and the desire to change it will persist throughout the life cycle"
51. Have you established a baseline yet?
52. Baseline An SCM concept that helps practitioners to control change without seriously impeding justifiable change IEEE Definition: A specification or product that has been formally reviewed and agreed upon, and that thereafter serves as the basis for further development, and that can be changed only through formal change control procedures It is a milestone in the development of software and is marked by the delivery of one or more computer software configuration items (CSCIs) that have been approved as a consequence of a formal technical review A CSCI may be such work products as a document (as listed in MIL-STD-498), a test suite, or a software component
53. Baselining Process A series of software engineering tasks produces a CSCI The CSCI is reviewed and possibly approved The approved CSCI is given a new version number and placed in a project database (i.e., software repository) A copy of the CSCI is taken from the project database and examined/modified by a software engineer The baselining of the modified CSCI goes back to Step #2
54. The SCM Repository
55. Paper-based vs. Automated Repositories Problems with paper-based repositories (i.e., file cabinet containing folders) Finding a configuration item when it was needed was often difficult Determining which items were changed, when and by whom was often challenging Constructing a new version of an existing program was time consuming and error prone Describing detailed or complex relationships between configuration items was virtually impossible Today's automated SCM repository It is a set of mechanisms and data structures that allow a software team to manage change in an effective manner It acts as the center for both accumulation and storage of software engineering information Software engineers use tools integrated with the repository to interact with it
56. Automated SCM Repository(Functions and Tools) Versioning Requirements tracing Functions Data integrity Information sharing Tool integration Data integration Methodology enforcement Document standardization SCM Repository Dependency tracking Configuration management Change management Audit trails (Explained on next two slides)
57. Functions of an SCM Repository Data integrity Validates entries, ensures consistency, cascades modifications Information sharing Shares information among developers and tools, manages and controls multi-user access Tool integration Establishes a data model that can be accessed by many software engineering tools, controls access to the data Data integration Allows various SCM tasks to be performed on one or more CSCIs Methodology enforcement Defines an entity-relationship model for the repository that implies a specific process model for software engineering Document standardization Defines objects in the repository to guarantee a standard approach for creation of software engineering documents
58. Toolset Used on a Repository Versioning Save and retrieve all repository objects based on version number Dependency tracking and change management Track and respond to the changes in the state and relationship of all objects in the repository Requirements tracing (Forward tracing) Track the design and construction components and deliverables that result from a specific requirements specification (Backward tracing) Identify which requirement generated any given work product Configuration management Track a series of configurations representing specific project milestones or production releases Audit trails Establish information about when, why, and by whom changes are made in the repository
59. List of Configuration Management Tools http://www.daveeaton.com/scm/CMTools.html
60. The SCM Process
61. Primary Objectives of the SCM Process Identify all items that collectively define the software configuration Manage changes to one or more of these items Facilitate construction of different versions of an application Ensure the software quality is maintained as the configuration evolves over time Provide information on changes that have occurred (Compare this process to the five SCM tasks)
62. SCM Questions How does a software team identify the discrete elements of a software configuration? How does an organization manage the many existing versions of a program (and its documentation) in a manner that will enable change to be accommodated efficiently? How does an organization control changes before and after software is released to a customer? Who has responsibility for approving and ranking changes? How can we ensure that changes have been made properly? What mechanism is used to appraise others of changes that are made?
63. Status reporting Configuration auditing Version control Change control Identification CSCI CSCI CSCI CSCI SCM Tasks (More on next slide)
64. SCM Tasks (continued) Concentric layers (from inner to outer) Identification Change control Version control Configuration auditing Status reporting CSCIs flow outward through these layers during their life cycle CSCIs ultimately become part of the configuration of one or more versions of a software application or system
65. Identification Task Identification separately names each CSCI and then organizes it in the SCM repository using an object-oriented approach Objects start out as basic objects and are then grouped into aggregate objects Each object has a set of distinct features that identify it A name that is unambiguous to all other objects A description that contains the CSCI type, a project identifier, and change and/or version information List of resources needed by the object The object realization (i.e., the document, the file, the model, etc.)
66. Change Control Task Change control is a procedural activity that ensures quality and consistency as changes are made to a configuration object A change request is submitted to a configuration control authority, which is usually a change control board (CCB) The request is evaluated for technical merit, potential side effects, overall impact on other configuration objects and system functions, and projected cost in terms of money, time, and resources An engineering change order (ECO) is issued for each approved change request Describes the change to be made, the constraints to follow, and the criteria for review and audit The baselined CSCI is obtained from the SCM repository Access control governs which software engineers have the authority to access and modify a particular configuration object Synchronization control helps to ensure that parallel changes performed by two different people don't overwrite one another
67. Version Control Task Version control is a set of procedures and tools for managing the creation and use of multiple occurrences of objects in the SCM repository Required version control capabilities An SCM repository that stores all relevant configuration objects A version management capability that stores all versions of a configuration object (or enables any version to be constructed using differences from past versions) A make facility that enables the software engineer to collect all relevant configuration objects and construct a specific version of the software Issues tracking (bug tracking) capability that enables the team to record and track the status of all outstanding issues associated with each configuration object The SCM repository maintains a change set Serves as a collection of all changes made to a baseline configuration Used to create a specific version of the software Captures all changes to all files in the configuration along with the reason for changes and details of who made the changes and when
68. Configuration Auditing Task Configuration auditing is an SQA activity that helps to ensure that quality is maintained as changes are made It complements the formal technical review and is conducted by the SQA group It addresses the following questions Has the change specified in the ECO been made? Have any additional modifications been incorporated? Has a formal technical review been conducted to assess technical correctness? Has the software process been followed, and have software engineering standards been properly applied? Has the change been "highlighted" and "documented" in the CSCI? Have the change data and change author been specified? Do the attributes of the configuration object reflect the change? Have SCM procedures for noting the change, recording it, and reporting it been followed? Have all related CSCIs been properly updated? A configuration audit ensures that The correct CSCIs (by version) have been incorporated into a specific build That all documentation is up-to-date and consistent with the version that has been built
69. Status Reporting Task Configuration status reporting (CSR) is also called status accounting Provides information about each change to those personnel in an organization with a need to know Answers what happened, who did it, when did it happen, and what else will be affected? Sources of entries for configuration status reporting Each time a CSCI is assigned new or updated information Each time a change is approved by the CCB and an ECO is issued Each time a configuration audit is conducted The configuration status report Placed in an on-line database or on a website for software developers and maintainers to read Given to management and practitioners to keep them appraised of important changes to the project CSCIs
70. Summary Software Quality Assurance? Reviews, Inspection… The cost of quality? Defect Tracking Systems Software Reliability Engineering Configuration Management SCM Repository SCM Process SE 501 Dr. Basit Qureshi: Lecture for Week 12