1 / 60

INFO 636 Software Engineering Process I Prof. Glenn Booker

INFO 636 Software Engineering Process I Prof. Glenn Booker. Week 7 – Measurement . Measurement . Measurements are critical, because the act of measuring something focuses people’s attention on it We have already looked at several basic measures in the PSP

darren
Download Presentation

INFO 636 Software Engineering Process I Prof. Glenn Booker

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. INFO 636Software Engineering Process IProf. Glenn Booker Week 7 – Measurement INFO636 Week 7

  2. Measurement • Measurements are critical, because the act of measuring something focuses people’s attention on it • We have already looked at several basic measures in the PSP • Now expand to a more general process for choosing measurements and how to present them INFO636 Week 7

  3. Measurement • The approach presented here is a variation on the GQM approach cited in the text, called GQ(I)M • The (I) adds Indicators, which are the means used to present the measurements graphically • This approach is also used in INFO 630 INFO636 Week 7

  4. Why Care About Measurement? • To quote Albert Einstein, "Not everything that counts can be counted and not everything that can be counted counts." • We are seeking to identify things • 1) which can be counted, and • 2) which count toward achieving our goals INFO636 Week 7

  5. Reasons for Measurement • Measurements are required by all major process models (CMM, ISO 9000, etc.) • To Characterize or understand the current status of activities or products • To compare that understanding to our objectives, and Evaluate whether the current status is “good” or “bad” INFO636 Week 7

  6. Reasons for Measurement • To Predict future performance, based on past trends • To form the basis for measuring Improvement • You won’t know if you improved, if you don’t know where you started! INFO636 Week 7

  7. Where do we get Information? • Metrics are often used to support decision making (the Evaluate step on a previous slide) • Decisions should be based on quantified information • To get that information, we calculate measures from raw data called data elements • The data elements each come from a data source INFO636 Week 7

  8. How do we Choose what to Measure? • A commonly used method for selecting measurements is called GQ(I)M for Goal, Question, Indicator, and Measurement • It is based on GQM work by Victor Basili (first reported circa 1988-89) • GQ(I)M was published by Robert Park et al INFO636 Week 7

  9. How do we Choose what to Measure? • The GQ(I)M method uses ten steps to describe measurements systematically • The steps don’t have to be followed in the order presented; their main purpose is to help ensure that measurements have been fully thought out • You can start at the top, or the bottom, or in the middle INFO636 Week 7

  10. 1. Identify Business Goals • Business goals are the big, vague, lofty desired accomplishments for an organization • Think of what you’d find bragged about in a company’s annual report • Reduce cycle time • Improve customer satisfaction INFO636 Week 7

  11. 1. Identify Business Goals • Develop detailed process history • Respond to changing business environment • Reduce overhead • Improve competitive position • Increase market share • Improve product quality INFO636 Week 7

  12. 2. Identify Desired Knowledge • Break down each goal into products, resources, and activities needed to meet those goals • Think of questions like: • What activities do I manage or execute? • What do I want to achieve or improve? • To do this, I will need to … INFO636 Week 7

  13. 3. Identify Subgoals • Set subgoals (objectives) for each of the areas you manage • What do you want to know about the results of step 2? What kind of information is important to you? • How big/fast/expensive/complex/much time will a process/product/tool/resource take? INFO636 Week 7

  14. 4. Identify Entities and Attributes • Formulate questions to identify entities (documents, work products) created by your process, and the attributes of them you are interested in (size, quality, duration, cost) • Don’t get too detailed at this point - just identify the type of information desired (what do you want to understand), and the subject of that information (what about it do you want to know) INFO636 Week 7

  15. 4. Identify Entities and Attributes • Entities that you want to measure can come from any of: • A process’ inputs • The activities within a process • The process’ outputs • Then decide what characteristics of each entity are of interest INFO636 Week 7

  16. 5. Define Measurement Goals • First choose the type of measurement goals • Active goals: reduce or improve something • Passive goals: identify, assess, understand • Lower maturity organizations start with passive goals, then work on active goals after some history is known INFO636 Week 7

  17. 5. Define Measurement Goals • Form structured statements of the measurement goals for each attribute • This step defines a metric in the form of a sentence INFO636 Week 7

  18. 5. Define Measurement Goals • The general format or syntax is: • <verb> <measure> <qualifier(s)> [objective] • where • <verb> is active or passive, as chosen earlier • <measure> is the actual measure to be collected • <qualifiers> indicate the scope or time frame of the measurements (a.k.a. independent variables), • [objective] is only given for active measurement goals – what value is desired? INFO636 Week 7

  19. 5. Define Measurement Goals • Passive measurement goal examples: • “Measure the number of requirements which changed each month.” • “Identify the voluntary turnover rate per month for programmers and software engineers.” • Active measurement goal examples: • “Reduce the defect rate of developed code over time to under 20 defects/KLOC” • “Improve the percent of satisfied customers after 30 days of product use to 95% or more” INFO636 Week 7

  20. 5. Define Measurement Goals • DO NOT report traits of individual people (for fear of judgment) • Unless that’s part of their job description (e.g. sales people) • Also need to balance how often measurements are made • More frequent measurement gives finer control, but excessive measurement wastes time and slows the process being measured INFO636 Week 7

  21. 6. Quantify Questions and Indicators • Pose questions to address your measurement goals (quantifiable ones, if addressing active goals) • Active: “Can we resolve customer emergencies, on average, in under 24 hours?” • Passive: “How many requirements do we have at the end of the Requirements Definition phase?” INFO636 Week 7

  22. 6. Quantify Questions and Indicators • Sometimes the search for a meaningful metric starts with a question, and that leads to filling out the GQ(I)M from the middle • Identify indicators to show the results effectively (see later this lecture), such as: • Pie charts • Bar graphs • Scatter plots, etc. INFO636 Week 7

  23. 7. Identify Data Elements • Identify the data elements needed to prepare (calculate) each indicator • e.g. “defect rate by module each month” needs a list of defects found, with the module each came from, for the last month, and the size of each module in kSLOC INFO636 Week 7

  24. 7. Identify Data Elements • Determine the source for each data element – from where do you get it? • E.g. defect data will come from our change request database • Size data will be generated by the development environment INFO636 Week 7

  25. 8. Define Measures • Define exactly what you mean by each measure • Use a checklist if needed to show what is and isn’t included in its definition (such as we used for definition of LOC) • Cite the source if an unusual measure is used; or if you made it up, explain why • New measures are allowed! INFO636 Week 7

  26. 8. Define Measures • Include any rules, assumptions, constraints, and environment which are part of the measure’s definition • E.g. Defect rate = (# of defects whose origin was in each module)/(# of kSLOC in that module) INFO636 Week 7

  27. 9. Identify Measurement Implementation • Deciding how to implement a measurement program is generally done in three steps • Analyze what measures are currently collected (if anything) and how they’re used • Diagnose how well the current measurements meet your goals – find what’s missing? INFO636 Week 7

  28. 9. Identify Measurement Implementation • Act on implementing new measurements, possibly in a phased approach based on project or organizational priorities • Implementing measurements is a cultural change, so like any other change, it’s best done in small steps • Add a few measurements now, wait six months, add a few more, etc. • Let people see benefits from the new measurements! INFO636 Week 7

  29. 10. Prepare Measurement Plan • Take all of the aforementioned information and create a complete plan to identify and implement measurement for your organization or project • This is generally called a Metrics Plan or a Measurement Plan INFO636 Week 7

  30. Summary of GQ(I)M Steps • Goal • Subgoal (objective; why collect this metric) • Question (answered by this metric) • Indicator (how display metric) • Measurement (the actual metric and its definition) • Data Elements (used to calculate metrics) • Source (of each data element) INFO636 Week 7

  31. Indicators • Indicators are the means used to present measures, such as charts, graphs, etc. • Consider how your data will be presented - in color or B/W, live or printed • Will your audience see pristine originals, or will it be copied a zillion times? INFO636 Week 7

  32. Indicators • To choose the right indicator, consider: • The Amount of Data to be presented for each interval (e.g. one measure at a time or five different survey responses at once) • The Number of Intervals to be shown, such as time units, modules of code, etc. INFO636 Week 7

  33. Indicators • Different indicators are better at different Amount or Number characteristics INFO636 Week 7

  34. Indicators • For most graphs: • The X-axis of a graph (the horizontal line) is the independent variable; often a <qualifier>, such as time, severity, etc. • If the X axis is Time, it should show how often measurements are made (weekly, monthly, etc). INFO636 Week 7

  35. Y Dependent variable (measure) x x x x x x x x X axis Independent variable (qualifier) Indicators • The Y-axis of a graph (the vertical line) is the dependent variable; the thing you are measuring (the Measure). INFO636 Week 7

  36. Pie Chart • The lowly pie chart is good for presenting a small amount of data • % of customers satisfied and not satisfied • % of defects by severity at this moment • Amount of Data: Shows a few data points (2-10) • Number of Intervals: One - it generally shows only one moment in time INFO636 Week 7

  37. Sample Pie Chart INFO636 Week 7

  38. Ishikawa’s Basic Tools • Developed circa 1989 for manufacturing production • Used widely in quality control • Focuses on project level concerns; is this batch good enough to accept? • Not very useful for research; has little theoretical basis INFO636 Week 7

  39. 1. Check sheet • Used to gather data easily, consistently, and in a standard format • A check sheet used to help the quality of a process or product is a “checklist” • Helps to define key parts of a process • Examples include code inspection checklist, detailed test procedures INFO636 Week 7

  40. 2. Pareto Diagram • Used to identify problem areas - where to fix first; what are the biggest fires to put out • Defects tend to cluster in buggy portions of code • Use Pareto to plot a column graph of the defect rate by the module it came from • Could add a line graph for the cumulative % of defects above it (not shown) • Pareto diagram must list X axis categories in descending order of value INFO636 Week 7

  41. 2. Sample Pareto Diagram INFO636 Week 7

  42. 3. Histogram • A bar chart is used to break down data by an ordered category (e.g. defect severity, satisfaction rating) • Can choose to put bars next to each other (clustered), or stack them. • Stack when they add to a constant (e.g. 100%), or when the total is also a useful measure (total number of defects) INFO636 Week 7

  43. 3. Histogram • Can show limited time spans, e.g. a few time intervals Simple bar graph  INFO636 Week 7

  44. 3. Histogram  Stacked Clustered  INFO636 Week 7

  45. 3. Histogram • A true histogram can group ranges of values in the X axis, and show counts or average Y values for each group • E.g. average salary (Y axis) for employees aged 18-29, 30-39, 40-49, etc. (X axis groups) • Good for hiding individual values, and looking at larger trends in the data INFO636 Week 7

  46. 4. Run Charts • Plots some measure versus time using a line graph • Often compare values to a desired or target value, especially at higher process maturity levels (CMM Level 3 and up) • Special case: The “S” curve plots (cumulative % completion of something) versus time INFO636 Week 7

  47. 4. Sample Run Chart INFO636 Week 7

  48. 5. Scatter Diagram • Plot two measures against each other to see if there’s a correlation between them • E.g. Defect rate per module versus module complexity, or productivity versus experience • When we say plot ‘Blah versus Ick,’ usually Blah is the Y axis, and Ick is the X axis INFO636 Week 7

  49. 5. Scatter Diagram INFO636 Week 7

  50. 5. Scatter Diagram • Once there’s enough data, can add curve fitted lines, and +/- variances • The scatter diagram is the basis for regression analysis • Curve fitting to the data, to help define a statistically significant connection between the two measures INFO636 Week 7

More Related