Software Metrics/Quality Metrics

1. Software Metrics/Quality Metrics • Software “Quality” Metrics: • Product • Pre-Release and Post-Release Process • Project • Data Collection Product Characteristics Project Characteristics Process Characteristics

2. Software Metrics • Software Product • All the deliverables • Focus has been on Code, but interested in all artifacts • Product metrics include concerns of complexity, performance,size, “quality,” etc. • Pre-Release & Post Release Processes • Focus has been on Pre-Release processes • Easiest staring point for metrics – Testing and the number of “bug” found • Process metrics used to improve on development and support activities • Process metrics include defect removal effectiveness, problem fix response time, etc. • Project • Cost • Schedule • HR staffing and other Resources • Customer Satisfaction • Project metrics used to improve productivity, cost, etc. • Project metrics include cost such as effort/size, speed such as size/time, etc. • Project and Process metrics are often intertwined. Will talk about this more Function point

3. ProductQualityMetrics • What are all the deliverables ? • Code and Help Text • Documentation (function, install, usage, etc. in requirements & design specifications) • Education (set-up/configure, end-user, etc.) • Test Scenarios and Test Cases • Quality Question : (mostly intrinsic to the product but affects external customer satisfactions ) • When/where does it fail; how often • how many; defect rate

4. GQM (one more time from Basili) • A reminder on generating measurements: • In coming up with metrics think of GQM • What’s the goal • What’s the Question • What’s the metric • Goal: is Improved Quality • Question: What is the Post Release Defect Rate? • Metric: Number of problems found per user months

5. Some Definitions of Error to Failure • Error – human mistake that results in incorrect software (one or more fault or defect) • Defect or Fault – a mistake in the software product that may or may not be encountered • Problem – a non-functioning behavior of the software as a result of a defect/fault in the product. • Note that an error can cause one or more defects, and a defect can cause one or more problems. But a problem may never surface even if there is a defect which was caused by a human error.

6. When/Where Do Product Failures Occur • When/Where are somewhat intertwined • Right away – e.g. happens at install • Sometimes - e.g. happens at initialization-configuration • Sometimes – e.g. happens at certain file access • Generalized Metric • Mean time to failure (MTTF) • Difficult to assess • What should be the goal (8 hours, 30 days, 6 months), or should we just say --- “lessen the failure rate”? • Hard to test for and analyze (especially- prod. education, doc., etc.) • Applies better for simple logic (like stays up for z amount of time) Meantime to failure for install problem should probably be close to 0

7. Product Defects and Defect Rate • Most of the metric has been asked in terms of code but should be more inclusive: • Defect Volume: How many defects (for the complete product - not just for code) • Defect Rate= defects/(opportunity of defect) • Defects of all kind or by type (e.g. code, test cases, design, etc.) • Defects by severity (not quite a rate – more by category) • Opportunity of defect by(* also used to assess volume) : • Code : loc, function point, module • Documentation : pages, diagrams • Education or training: # of power point slides (doc) or amt. of time (delivery)

8. Code Defect Opportunity (LOC) • Using Lines of code (loc) “problems” • Executable, non-executable (comments) • Test cases and scaffolding code • Data and file declaration • Physical line or logical line • Language difference (C, C++, assembler, Visual Basic, etc.)

9. Possible Code Defect Rate Metrics • Often used : • Valid Unique Defect per line of executable and/or data code released(shipped) • IBM’s total valid unique defects / KSSI • Total valid unique defects / KCSI (only changed code) • Valid Unique Defect of “high severity” per line of executable and/or data code released (shipped) • What about all “in-line comments”; should they not count ? These provide opportunity of defects too. (especially for pre and post condition specifications) • What about Help text ?

10. Product Quality Metric (User View) • Defect rate is not as useful from user perspective: • What type of problems do users face?: • screen interface • data reliability/(validity) • functional completeness • end user education • product stability - crashes • error message and recovery • Inconsistencies in the handling of similar fucntionalities • How often are these types of defect encountered? ---- counted with -- MTTF -- means more to users? • Possible metric is : Problems per User Month(PUM) • user month is dependent on length of period and the number of users (this takes some tracking effort) • More Broader Customer Satisfaction issues • CUPRIMDSO – capability, usability, performance, rel. etc. (IBM), • FURPS – functionality, usability, reliability, etc. (HP)

11. Begin Function Point Separate Segment

12. Function Point (product size or complexity) metric • Often used to assess the software complexity and/or size • May be used as the “opportunity for defect” part of defect rate • Started by Albrecht of IBM in late 70’s • Gained momentum in the 90’s with IFPUG as software service industry looked for a metric • Function Point does provide some advantages over loc • language independence • don’t need the actual lines of code to do the counting • takes into account of many aspects of the software product • Some disadvantages include : • a little complex to come up with the final number • consistency (data reliability) sometimes varies by people

13. Function Point Metric via GQM • Goal: Measure the Size(volume) of Software • Question: What is the size of a software in terms of its: • Data files • Transactions • Metrics: • amount/difficulty of “Functionalities” to represent size/volume • consider Function Points ---- (defined in this lecture) What kind of validity problem might you encounter? – “construct”: applicability, “predictive”: relational ; “content” : coverage?

14. FP Utility • Where is FP used? • Comparing software in a “normalized fashion” independent of op. system, languages, etc. • Benchmarking and Prediction: • size .vs. cost • size vs development schedule • size vs defect rate • Outsourcing Negotiation

15. Methodology • Identify and Classifying: • Data (or files/tables) • Transactions • Evaluation of Complexity Level • Compute the Initial Functional Point • Re-assess the range of other factors that may influence the computed Functional Point and Compute the Function Point

16. 1) Identifying & Classifying5 “Basic Entities” • Data/File: • Internally generated and stored (logical files and tables) • Data maintained externally and requires an external interface to access (external interfaces ) • Transactions: • Information or data entry into a system for transaction processing (inputs) • Information or data “leaving” the system such as reports or feeds to another application (outputs) • Information or data displayed on the screen in response to query (query) Note: - What about “tough” algorithms and other function oriented stuff? (We take of that separately in a separate 14 “Degree of Influences”)

17. 2) Evaluating Complexity • Using a complexity table, each of the 5 basic entity is evaluated as : • low • average • high • Complexity table uses 3 attributes for decisions • # ofRecord Element Types (RET): e.g. employee data type, student record type ---- # offile types • # of unique attributes (fields) or Data Element Types (DET) for each record : e.g. name, address, employee number, and hiring date would make 4 DETs for the employee file • # ofFile Type Referenced (FTR): e.g an external payroll record file that needs to be accessed

18. 5 Basic Entity Types uses the RET, DET, and FTRfor Complexity Evaluation For Logical Files and External Interfaces (DATA): # of RET1-19 DET20-50 DET50+ DET 1 Low Low Ave 2 -5 Low Avg High 6+ Avg High High For Input/Output/Query (TRANSACTIONS): # of FTR1-4 DET5 -15 DET16+ DET 0 - 1 Low Low Ave 2 Low Avg High 3+ Avg High High

19. Example • Consider a requirement: ability or functionality to add a new employee to the “system.” • (Data): Employee information involves, say, 3 external file that each has a different Record Element Types (RET) • Employee Basic Information file has employee data records • Each employee record has 55 fields (1 RET and 55 DET) - AVERAGE • Employee Benefits records file • Each benefit record has 10 fields (1 RET and 10 DET) - LOW • Employee Tax records file • Each tax record has 5 fields ( 1 RET and 5 DET) - LOW • (Transaction): Adding a new employee involves 1 input transaction which involves 3 file types referenced (FTR) and a total of 70 fields (DET). So for the 1 input transaction the complexity is HIGH

20. Function Point (FP) Computation • Composed of 5 “Basic Entities” • input items • output items • inquiry • master and logical files • external interfaces • And a “complexity level index” matrix : Low Average High 3 4 6 Input 5 7 Output 4 3 Inquiry 4 6 Logical files 7 10 15 Ext. Interface 7 5 10

21. 3) Compute Initial Function Point • Initial Function Point : • Σ [Basic Entity x Complexity Level Index] all basic entities Continuing the Example of adding new employee: - 1 external interface (average) = 7 - 1 external interface (low) = 5 - 1 external interface (low) = 5 - 1 input (high) = 6 Initial Function Point = 1x7 + 1x5 + 1x5 + 1x6 = 23

22. 4) More to Consider • There are 14 more “Degree of Influences” (DI) on a scale of 0 - 5 : • data communications • distributed data processing • performance criteria • heavy hardware utilization • high transaction rate • online data entry • end user efficiency • on-line update • complex computation • reusability • ease of installation • ease of operation • portability (supports multiple sites) • maintainability (easy to change)

23. Function Point Computation (cont.) • Define Technical Complexity Factor (TCF): • TCF = .65 + (.01 x DI ) • where DI = SUM ( influence factor value) • So note that .65 ≤ TCF ≤ 1.35 Function Point (FP) = Initial FP x TCF Finishing the Example: Suppose after considering 14 DI’s, our TCF = 1.15, then: Function Point = Initial FP x TCF = 23 x 1.15 = 26.45

24. Defect Rate: Defects/FP by CMM Levels • C. Jones estimated defect rates by SEI’s CMM levels through the maintenance life of a software product: • CMM Level 1 organizations – 0.75 defect/FP • CMM Level 2 - 0.44 • CMM Level 3 – 0.27 • CMM Level 4 – 0.14 • CMM Level 5 – 0.05 Be careful of this type of claims – use it with caution

25. End Function Point Separate Segment

26. Pre-Release Process Quality Metrics • Most common one is from testing (Defect Discovery Rate) defects found (by severity) per time period ( per dev. phase) • Compare “defect arrivals” by time by test phase • looking for “stabilization” (**what would the curve look like?*) • looking for a decreasing pattern • Compare number of defects by products • those with high number of problems found during pre-release tend to be “buggy” after release (interesting phenomenon) • Other Pre-Release quality metric: Defect Removal Effectiveness (e.g. Via inspection) • defects removed / ( total latent defects) • latent defects are estimated : how estimated? --- go back later with defects found in the field

27. Post-Release Product and Process • Post-Release Product: • # of Problems per Usage-Month (# of PUM) • Post-Release “Fix Process”: • Fix Quality = Number of Fix bugs/ Total number of fixes • Very sensitive if fix quality is not close to zero • Post-Release Process Quality • Problem backlog= total # of problems unresolved • by severity • by arrival date • Problem Backlog Index = # of problems resolved / # of arrivals per some time period such as week or month • Average Fix Response Time ( from problem open to close ) • These metrics are usually compared with a goal: • average response time on severity 1 problem is 24 hours • problem backlog index is between 1.3 and .8 (.8 may be problem!)

28. Collecting Data • Decide on what Metrics are to be used • measuring what (validity of measure) • what’s the goal (validity of measure) • Decide on how to collect the data • clearly defining the data to be collected • assure the recording is accurate (reliability) • assure the classification is accurate (reliability/validity) • Decide on tools to help in the collection • source code count • problem tracking

29. Data Collection Methodology (Basili & Weiss) • Establish the goal of the data collection • Develop a list of questions of interest • Establish data categories • Design and test data collection mechanism (e.g. forms) • Collect and check the reliability data • Analyze the data