SMU CSE 8314 / NTU SE 762-N Software Metrics and Quality Engineering

1. SMU CSE 8314 / NTU SE 762-NSoftware Metrics and Quality Engineering Module 34 Statistical Process Control

2. Terminology • Upper Specification Limit • The highest value permitted for correct behavior • Lower Specification Limit • The lowest value permitted for correct behavior • Upper or Lower Control Limit or Action Threshold • The point at which action should be taken

3. Example - Specification Limit Upper Specification Limit (edge of road) Lower Specification Limit (edge of road)

4. Example - Control Limit Upper Control Limit (imaginary line) Lower Control Limit (imaginary line)

5. Control Chart - each x is a measured data point Upper Spec Limit Upper Control Limit (Action Threshold) 8 Goal x x x 7 x x x x x x x x x x x x x 6 x x x x x x x x x x x x x x x x 5 Factor x x x x x x 4 3 2 Lower Spec Limit Lower Control Limit (Action Threshold) Time Using Limits and Thresholds

6. Applications • Number of xxx’s per time period • xxx = objects designed, requirements specified, modules coded, modules tested, tests written, tests run, etc. • Projected or Actual value of yyy • yyy = cost, schedule, turnover rate, defect level, etc. • Anything you can measure and establish ranges for

7. How do you Determine Capability? 1) Collect data (truthful data!) from real projects over a period of time 2) Normalize the data 3) Analyze the data using statistical techniques to determine the mean (m) and standard deviation (s). 4) Establish specification limits and thresholds

8. Common Statistical Formulas Let N be the total size of the population and X be an individual data value Mean (“average”) m = SX N Variance s2 = S(X-m)2 N Standard Deviation _______ s = ÖS(X-m)2 N

9. How Do You Establish the Limits and Thresholds? • Before you have characterized the process, you need to make an initial guess • You will refine this later as you determine what actually happens • Usually, the mean or m is your initial “center line” or “goal”

10. Initial Limits and Thresholds • Often, 1.5 s (1.5 standard deviations) from the mean is a normal level of variation, so that could become your initial specification limit. But you might want to go higher, perhaps as much as 2 or 3 s. • Something smaller, such as 1.2 s, makes a good initial threshold

11. After You Characterize the Process • You will determine a level of variation that characterizes the behavior of your process • And you will set thresholds based on experience as to what levels of behavior tend to signify potential problems

12. Example - Actual Data

13. Example - Mean andStandard Deviation +1s m -1s

14. Upper Spec Limit Example - Spec Limits Lower Spec Limit

15. Example - Thresholds Upper Threshold Lower Threshold

16. Upper Spec Limit Example - All Together Upper Threshold +1s m -1s Lower Threshold Lower Spec Limit

17. Example -Spotting Trouble

18. Considerations forThresholds and Limits • What can your process do? • There is no point in forcing your process to perform beyond its capability • Process changes may be necessary if your process is not good enough • How stable is your process? • If the process is not in control, you cannot establish “narrow” limits • You may tend to overreact

19. More Considerations forThresholds and Limits • What levels of performance are acceptable in terms of final results achieved? • I.e., when is it necessary to take action? • There is no point in taking unneeded action Once your process is stable and well characterized, you are in a position to plan effective improvements to achieve desired performance goals

20. For Product Development Processes, Limits may Vary over Time

21. They Need Not be Balanced Around the Mean

22. In Short ... • Quantitative process management techniques allow you to control, predict, and manage the outcome of projects • By understanding and characterizing your true capability, you can • Predict outcomes • Spot problems • Make realistic improvement plans

23. Applying Statistical Process Control Techniques to Earned Value Metrics

24. Earned Value is a Way to Measure Cost and Schedule Performance Metric: • Amount of work actually performed • Projections for duration of project Measures: • Earned Value = the value of the total work performed • Performance indices for cost & schedule

25. This is the new measure that makes this a powerful technique Earned Value Techniques Use These Data • Your Plan - how much you expect or estimate you will spend • BCWS • Your Actual Expenses - how much it has actually cost you • ACWP • Your Earned Value - the value of the work you have performed • BCWP

26. Questions You Can Answer with Earned Value • Are we getting the work done on schedule? • Are we overspending? • Will we overrun or under-run the budget? • What is a realistic end date? • What performance level is required to meet the budget or the deadline?

27. Earned Value Provides Insight Early in the Project • You can tell if there are schedule or cost problems before they are obvious • You have time to make adjustments • You can communicate the problems and their scope using quantitative methods rather than guesswork

28. BCWS BCWS(Budgeted Cost of Work Scheduled) • The amount of effort you had planned to expend (i.e., the amount you budgeted) • “How much work should have been done”

29. BCWP BCWP(Budgeted Cost of Work Performed) • The value produced for the effort you have actually expended • What you have earned • “How much work was actually done”

30. ACWP ACWP(Actual Cost of Work Performed) • The amount of money you have spent so far • The “actual” cost • “What did your work cost?”

31. What Can We Tell About the Schedule From These Three Numbers? SV - Schedule Variance SV = BCWP - BCWS SPI - Schedule Performance Index SPI = BCWP / BCWS Negative means behind schedule Less than 1 means behind schedule

32. Ahead of Schedule Behind Schedule SPI Indicates Whether You are Ahead of or Behind Schedule

33. What Can We Tell About the Cost From These Three Numbers? CV - Cost Variance CV = BCWP - ACWP CPI - Cost Performance Index CPI = BCWP / ACWP Negative means over budget Less than 1 means over budget

34. Under Budget Over Budget CPI Indicates Whether You are Under or Over Budget

35. Two Useful Numbers BAC - Budget at Completion • The total budget for the project • BCWS at project completion SCHED - Schedule for the project • The total number of weeks or months or whatever

36. Estimated Budget At Completion EAC - Estimate at Completion • The estimated actual cost of the project EAC = BAC / CPI VAC - Variance at Completion VAC = BAC - EAC Negative means over budget by this much

37. Schedule At Completion SAC - Schedule at Completion • The estimated actual schedule of project SAC = SCHED / SPI SVAC - Variance at Completion SVAC = SCHED - SAC Negative means behind schedule by this much

38. Thus You Can Tell ... • How far behind or ahead of schedule you are • How far over or under budget you are • What your eventual schedule and cost are likely to be You also have an early warning of trouble that you can use to try to fix the underlying problems

39. You Can Set Control Limits and Thresholds for Earned Value • Just as you can for many other quantities • Usually based on history from previous programs • Acceptable variation is established by the behavior of programs that did not get into trouble This is typically done by organizations at CMM level 4 or 5. At lower levels of maturity, you may lack the data to set useful limits.

40. Control Limits on SPI Chart Upper Control Limit Lower Control Limit

41. What Are Your Chances of Meeting Your Budget? • If CPI is outside of the control limits, it means you are under or over performing to a degree that means corrective action is necessary • But also look at trends - they can indicate potential problems in the future

42. What If the Budget Changes? • Sometimes projects adjust budgets and goals • In light of clear evidence that previous budgets & goals cannot be met • Or due to changes in goals or requirements • Or due to changes in commitments • Or due to changes in resources • … • How does this effect earned value?

43. Responding to Budget Changes • You can continue to track against the original plan, but show new estimates on the same graphs • You can “replan” • Re-compute BCWS and other values affected by the change and compute earned value against the new plan • This is recommended if the new budget is significantly different

44. What If You Are Asked to Recommend The Adjusted Budget? • Suppose, in light of clear evidence that previous budgets & goals cannot be met, you are asked to estimate the likely cost and thus the recommended new budget? • How do you know if the change is enough? (next slide)

45. “To Complete”Performance Index Let BAC be your original budget and EAC be your estimate of actual cost Then TCPI = (BAC-BCWP) / (EAC-ACWP) (work left to do / money left to spend) • This tells you how you must perform in order to meet your estimate (EAC)

46. Using TCPI (To CompletePerformance Index) TCPI = (BAC-BCWP) / (EAC-ACWP) • If TCPI is outside the control limits (i.e., not very close to 1.0), it indicates that, if you continue to perform as you have in the past, you are unlikely to meet the EAC estimate • So you must change the goal • Or take other corrective action

47. How Do You Set Control Limits for TCPI? • Look at past history • Determine what values correspond to projects that turned out acceptably and what values correspond to projects that turned out unacceptably • If your process is stable, there should be a definite range of acceptable values

48. SummaryEarned Value Can ... … tell you your true status … give you an effective schedule / budget outlook … help you understand the reasons why if you are not meeting your plan If the estimates are developed by the workers, they are more likely to “buy in” to the results

49. Summary (continued) • Various indices can be computed to tell you things about your project, such as: • Projected cost or schedule • Performance level required to meet a specific goal (original budget or revised budget)

50. Summary (concluded) • You can set control limits on earned value (CPI, SPI) to manage programs more closely • You can also set and use control limits on TCPI when budget changes are proposed