March 6, 2012 Mauricio E. Peña

1. Mathematical Formulation and Validation of the Impact of Requirements Volatility on Systems Engineering Effort March 6, 2012 Mauricio E. Peña

2. Outline • Motivation and introduction • Research methods • Observations from the research • Model Development • Results • Model Calibration • Evaluation of Model Performance • Sensitivity Analysis • Cross-Validation • Conclusion

3. Importance of Understanding Requirements Volatility • Requirements volatility has been identified by numerous research studies as a risk factor and cost-driver of systems engineering projects1 • Requirements changes are costly, particularly in the later stages of the lifecycle process because the change may require rework of the design, verification and deployment plans2 • The Government Accountability Office (GAO) concluded in a 2004 report on the DoD’s acquisition of software-intensive weapons systems that missing, vague, or changing requirements are a major cause of project failure3 System developers often lack effective methods and tools to account for and manage requirements volatility Source: 1- Boehm (1991), 2- Kotonya and Sommerville (1995), 3- GAO-04-393

4. Requirements Volatility is Expected • Changes to requirements are a part of our increasingly complex systems & dynamic business environment • Stakeholders needs evolve rapidly • The customer may not be able to fully specify the system requirements up front • New requirements may emerge as knowledge of the system evolves • Requirements often change during the early phases of the project as a result of trades and negotiations Requirements volatility must be anticipated and managed Sources: Kotonya and Sommerville (1995); Reifer (2000)

5. CSSE Parametric Cost Models • The Constructive Systems Engineering Cost Model (COSYSMO) was developed by the USC Center for Software and Systems Engineering (CSSE) in collaboration with INCOSE and Industry affiliates • COSYSMO is the first generally-available parametric cost model designed to estimate Systems Engineering effort • Built on experience from COCOMO 1981, COCOMO II • During the development of COSYSMO, volatility was identified as a relevant adjustment factor to the model’s size drivers Source: 7th Annual Practical Software and Systems Measurement Conference. COSYSMO Workshop, Boehm

6. COSYSMO Operational Concept Volatility Factor 6 Source: Valerdi (2005)

7. Research Methods Data gathered from 25 industry projects Literature Review & 6 Workshops completed We are here Source: Boehm et al (2000)

8. Organizations that Participated in the Research • The Aerospace Corporation • Northrop Grumman Corporation • The Boeing Company • Raytheon • United Launch Alliance • BAE • TI Metricas Ltda. • IBM • Distributed Management • MIT • USC • Lockheed Martin • Ericsson España • Samsung SDS • Rolls Royce • Softstar • Texas Tech • The US Army • The US Navy • The US Air force • The Australian Department of Defense

9. Requirements Volatility Observations • Requirements volatility is caused by an identifiable set of project and organizational factors • The level of requirements volatility is a function of the system life cycle phase • Requirements volatility leads to an increase in project size and cost • The cost and effort impact of a requirements change increases the later the change occurs in the system life cycle • The impact of requirements volatility may vary depending on the type of change: added, deleted, or modified

10. Model Form • Incorporated volatility effects through a scale factor (SF) added to the diseconomies of scale exponent (E) • Similar approach used to model volatility effects in Ada COCOMO1 • Prior research points to the compounding or exponential effect of project factors with variable life cycle impact2 Source: 1: Boehm, B. and Royce, W. (1989); 2: Wang, G. et al., (2008)

11. Volatility Scale Factor Expected REVL is rated as Very Low, Low, Moderate, High and Very High • Where • REVL = The % of the baseline requirements that is expected to change over the system lifecycle • wvl= aggregate lifecycle phase volatility weighting factor • And: • wl= weighting factor for each life cycle phase1 • Θl= % of total requirements changes per life cycle phase 1Life Cycle Phases: Conceptualize, Development, Operational Test and Evaluation, and Transition to Operation

12. Data Collection • Collected expert judgment on REVL and weighting factors through surveys and workshops • Software and Systems Engineers with 20+ years of experience • Variety of industries represented with an emphasis on Aerospace and Defense • Historical Data Collection • Data were collected from 25 projects from the Aerospace and Defense application domain • Collected COSYSMO size, effort, and cost driver data • Collected volatility data: added, modified, and deleted requirements over time

13. Model Calibration A Priori Model Historical data Mean and Variance Data-determined Model Regression Analysis; OLS Initial parameter mean and variance Expert-judgment estimates A Posteriori Model Updated parameter mean and variance Bayesian Analysis Formally combines expert judgment With sample data Optimized combination of data sources to increase precision Source: Boehm et al. (2000); Nguyen (2010)

14. Requirements Volatility (REVL) Rating Levels Developed based on surveys of experienced S/W and Systems Engineers (N =38)

15. Requirements Volatility Profile Based on expert judgment collected through three workshops (N = 36) and historical project data (N = 25)

16. Regression and Bayesian Calibration • COSYSMO can be described as multiple regression model • The model is linearized using a logarithmic transformation • The data-determined coefficient β1is combined with the a-priori expert judgment to develop the Bayesian calibrated model A posteriori Bayesian Update 1.96 2.09 2.01 Data Analysis A priori Expert Judgment

17. Model performance evaluated using the baseline model and a model with local calibration The performance of COSYSMO improves by including the requirements volatility factor Model Performance Comparison

18. Coefficient of Determination (n=25) COSYSMO with Requirements Volatility Factor Academic COSYSMO Estimated Systems Engineering Size (with diseconomies of scale) Estimated Systems Engineering Size (with diseconomies of scale) * Due to proprietary reasons only the analysis of the model accuracy is shown, not the data itself

19. Sensitivity Analysis Scenario 1 • Evaluated the sensitivity of the model’s results to the variability of key parameters • Standard deviation in the volatility life cycle profile (Θl) • Scenarios 1 and 2 • Standard deviation in the requirements volatility weighting factor (wl) • Scenarios 3 and 4 • Combination of cases: • Scenario 1-3, 1-4 • Scenario 2-3, 2-4 % Volatility per Life Cycle Phase Operational Test & Eval Transition to Operations Conceptualize Development Scenario 2 % Volatility per Life Cycle Phase Operational Test & Eval Transition to Operations Conceptualize Development Source: Guidelines for the Economic Analysis of Projects (1997)

20. Cross-Validation Results • The 25 projects were randomly divided into K=6 subsets • One of the subsets is excluded from the data set from which the model is built • The resulting model is used to predict effort for the excluded cases Improvement holds across the scenarios

21. Conclusions • Observations from the literature and workshop surveys were used to develop a mathematical framework for quantifying the impact of requirements volatility on systems engineering effort • An evaluation of the model was performed by comparing its prediction accuracy against Academic COSYSMO • The results indicate an improvement in effort prediction accuracy and MMRE • Cross validation and sensitivity analysis were performed to demonstrate the accuracy of the model in predicting effort for new projects

22. Future Work • Obtain additional project data from other organizations • The external validity of this study will be limited by the engineering organizations that contribute data and the background of the industry experts that participate in the research • Evaluate the effect of reuse on the results and its potential interaction with requirements volatility • Further work is needed to complete the characterization of the model performance depending on the type of change: added, modified or deleted

23. References • B. Boehm and W. Royce. Ada COCOMO and the Ada Process Model, TRW Defense Systems Group, 1989 • B. Boehm, Software risk management: Principles and practices, IEEE Software 1 (1991), 32-41. • B. Boehm, C. Abts, A.W. Brown, S. Chulani, B. Clark, E., Horowitz, R. Madachy, D.J. Reifer, and B. Steece, Software Cost Estimation with COCOMO II, Prentice Hall, New York, NY, 2000 • Economics and Development Resource Center .1997. Guidelines for the Economic Analysis of Projects. • S. Conte, H. Dunsmore, and V. Shen. Software Engineering Metrics and Models. Benjamin/Cummings Publishing Company, 1986. • General Accounting Office, "Stronger management practices are needed to improve DoD’s software-intensive weapon acquisitions (GAO-04-393)," 2004. • ISO/IEC, "ISO/IEC 15288:2002 (e) systems engineering - system life cycle processes," 2002. • G. Kotonya and I. Sommerville, Requirements engineering: Processes and techniques, John Wiley & Sons, New York, NY, 1998. • MIL-STD-498, "Software development and documentation," 1994. • D. Reifer, Requirements management: The search for nirvana, IEEE Software 17(3) (2000), 45-47 • D. Rhodes, R. Valerdi and G. Roedler, Systems engineering leading indicators for assessing program and technical effectiveness, Systems Engineering 12(1) (2009), 21-35. • Valerdi, R. (2005). The constructive systems engineering cost model (COSYSMO). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department. • Wang, G., Boehm, B., Valerdi, R., and Shernoff, A. (2008). “Proposed Modification to COSYSMO Estimating Relationship.” Technical Report. University of Southern California, Center for Systems and Software Engineering.

24. Back-up

25. COSYSMO Cost Estimating Relationships (CERs) SE_Hrs = Systems Engineering effort (hours) A = calibration constant derived from historical project data SIZE = measure of functional size of the system (number of requirements, interfaces, algorithms, operational scenarios) n = number of cost drivers (14) EM = effort multiplier for the ith cost driver The exponent (E) accounts for diseconomies of scale Source: Valerdi (2005)

26. Expert JudgmentLife Cycle Weighting Factors (n = 27) Systems Engineering Effort Penalty Due to Volatility Operational Test & Evaluation Transition to Operation Development Conceptualize Data collected from two workshops: 25th Annual USC CSSE COCOMO and the 2011 USC-CSSE ARR

27. Evaluation of Model Performance • The cost estimation accuracy of COSYSMO was compared to the accuracy of the model with the requirements volatility factor • The model was evaluated using predictive accuracy levels, Mean Magnitude of Relative Error (MMRE) and the coefficient of determination (R2) • The prediction accuracy level is defined as: Where k = number of projects in the set whose Magnitude of Relative Error is ≤ l Source: Conte et al., (1986)

28. Model Comparison Test (F-ratio) • The F-test is typically used to compare regression models and determine whether the null hypothesis is supported or refuted • In this case, the null hypothesis is that the simpler model (without a requirements volatility factor) is correct • Values of the F ratio near one (1) support the null hypothesis, while larger values favor the alternative hypothesis • The F-value was calculated to be 7.62 with > 95% confidence level: Supports the alternative hypothesis