Quantifying the impact of requirements volatility on systems engineering effort
1 / 29

Quantifying the Impact of Requirements Volatility on Systems Engineering Effort - PowerPoint PPT Presentation

  • Uploaded on

Quantifying the Impact of Requirements Volatility on Systems Engineering Effort. October 18, 2012 Mauricio E. Peña Ricardo Valerdi. Outline. Motivation and Introduction Research Methods Model Development Results Cross-Validation and Sensitivity Analysis Conclusion Threats to Validity

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Quantifying the Impact of Requirements Volatility on Systems Engineering Effort' - charles-bailey

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Quantifying the impact of requirements volatility on systems engineering effort

Quantifying the Impact of Requirements Volatility on Systems Engineering Effort

October 18, 2012

Mauricio E. Peña

Ricardo Valerdi

Outline Engineering Effort

  • Motivation and Introduction

  • Research Methods

  • Model Development

  • Results

  • Cross-Validation and Sensitivity Analysis

  • Conclusion

  • Threats to Validity

  • Future Research

Importance of understanding requirements volatility
Importance of Understanding Requirements Volatility Engineering Effort

  • Requirements volatility has been identified by numerous research studies as a risk factor and cost-driver of systems engineering projects1

  • Requirements changes are costly, particularly in the later stages of the lifecycle process because the change may require rework of the design, verification and deployment plans2

  • The Government Accountability Office (GAO) concluded in a 2004 report on the DoD’s acquisition of software-intensive weapons systems that missing, vague, or changing requirements are a major cause of project failure3

System developers often lack effective methods and tools to account for and manage requirements volatility

Source: 1- Boehm (1991), 2- Kotonya and Sommerville (1995), 3- GAO-04-393

Requirements volatility is expected
Requirements Volatility is Expected Engineering Effort

  • Changes to requirements are a part of our increasingly complex systems & dynamic business environment

    • Stakeholders needs evolve rapidly

    • The customer may not be able to fully specify the system requirements up front

    • New requirements may emerge as knowledge of the system evolves

    • Requirements often change during the early phases of the project as a result of trades and negotiations

Requirements volatility must be anticipated and managed

Sources: Kotonya and Sommerville (1995); Reifer (2000)

Csse parametric cost models
CSSE Parametric Cost Models Engineering Effort

  • The Constructive Systems Engineering Cost Model (COSYSMO) was developed by the USC Center for Software and Systems Engineering (CSSE) in collaboration with INCOSE and Industry affiliates

  • COSYSMO is the first generally-available parametric cost model designed to estimate Systems Engineering effort

  • Built on experience from COCOMO 1981, COCOMO II

  • During the development of COSYSMO, volatility was identified as a relevant adjustment factor to the model’s size drivers

Source: 7th Annual Practical Software and Systems Measurement Conference. COSYSMO Workshop, Boehm

Research methods
Research Methods Engineering Effort

Data gathered from 25 industry projects

Literature Review & 6 Workshops completed

We are here

Source: Boehm et al (2000)

Organizations that participated in the research
Organizations that Participated in the Research Engineering Effort

  • The Aerospace Corporation

  • Northrop Grumman Corporation

  • The Boeing Company

  • Raytheon

  • United Launch Alliance

  • BAE

  • TI Metricas Ltda.

  • IBM

  • Distributed Management

  • MIT

  • USC

  • Lockheed Martin

  • Ericsson España

  • Samsung SDS

  • Rolls Royce

  • Softstar

  • Texas Tech

  • The US Army

  • The US Navy

  • The US Air force

  • The Australian Department of Defense

Observations from the literature and research workshops
Observations from the Literature and Research Workshops Engineering Effort

  • Requirements volatility is caused by an identifiable set of project and organizational factors

  • The level of requirements volatility is a function of the system life cycle phase

  • Requirements volatility leads to an increase in project size and cost

  • The cost and effort impact of a requirements change increases the later the change occurs in the system life cycle

  • The impact of requirements volatility may vary depending on the type of change: added, deleted, or modified

Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston (2000); Zowghi and Nurmuliani (2002); Ferreira (2002); Kulk and Verhoef 2008

Cosysmo cost estimating relationships cers
COSYSMO Cost Estimating Relationships (CERs) Engineering Effort

SE_Hrs = Systems Engineering effort (hours)

A = calibration constant derived from historical project data

SIZE = measure of functional size of the system (number of requirements, interfaces, algorithms, operational scenarios)

n = number of cost drivers (14)

EM = effort multiplier for the ith cost driver

The exponent (E) accounts for diseconomies of scale

Source: Valerdi (2005)


Model form
Model Form Engineering Effort

  • Incorporated volatility effects through a scale factor (SF) added to the diseconomies of scale exponent (E)

  • Similar approach used to model volatility effects in Ada COCOMO

  • Prior research points to the compounding or exponential effect of requirements volatility

  • Linearized form:

Source: Boehm, B. and Royce, W. (1989); Kulk and Verhoef (2008); Wang, G. et al., (2008)

Volatility scale factor
Volatility Scale Factor Engineering Effort

Expected REVL is rated as Very Low, Low, Moderate, High, and Very High

  • Where

    • REVL = The % of the baseline requirements that is expected to change over the system lifecycle

    • C = Scale factor constant = 0.05

    • wvl= aggregate lifecycle phase volatility weighting factor

  • And:

    • wl= weighting factor for each life cycle phase1

    • Θl= % of total requirements changes per life cycle phase

    • l = life cycle phases

1Life Cycle Phases: Conceptualize, Development, Operational Test and Evaluation, and Transition to Operation

Data collection
Data Collection Engineering Effort

  • Collected expert judgment on the causes of volatility, expected REVL, and life cycle weighting factors through surveys conducted at 6 workshops

    • Software and Systems Engineers with 20+ years of experience

    • Variety of industries represented with an emphasis on Aerospace and Defense

  • Historical Data Collection

    • Data were collected from 25 projects from the Aerospace and Defense application domain

    • Collected COSYSMO size, effort, and cost driver data

    • Collected volatility data: added, modified, and deleted requirements over time

Requirements volatility revl rating levels
Requirements Volatility (REVL) Engineering EffortRating Levels

Developed based on surveys of experienced S/W and Systems Engineers (N =38)

Requirements volatility profile
Requirements Volatility Profile Engineering Effort

Based on expert judgment collected through three workshops (N = 36) and historical project data (N = 25)

Model calibration
Model Calibration Engineering Effort

A Priori Model

Historical data

Mean and Variance

Data-determined Model

Regression Analysis; OLS

Initial parameter

mean and variance

Expert-judgment estimates

A Posteriori Model

Updated parameter mean and variance

Bayesian Analysis

Formally combines expert judgment

With sample data

Optimized combination

of data sources to increase precision

Source: Boehm et al. (2000); Nguyen (2010)

Regression and bayesian calibration
Regression and Bayesian Calibration Engineering Effort

The coefficient β1for the requirements volatility scale factor was calculated through linear regression analysis using data from the 25 projects

The data-determined coefficient β1was formally combined with the a-priori expert judgment to develop the Bayesian calibrated model

A posteriori Bayesian Update




Data Analysis


A priori Expert Judgment


Model performance comparison

Model performance evaluated using the baseline model and a model with local calibration

The performance of COSYSMO improves by including the requirements volatility factor

Model Performance Comparison

Coefficient of determination n 25
Coefficient of Determination (n=25) model with local calibration

COSYSMO with Requirements Volatility Factor

Academic COSYSMO

* Relative Scale

Cross validation results
Cross-Validation Results model with local calibration

  • The 25 projects were randomly divided into K=6 subsets

  • One of the subsets is excluded from the data set from which the model is built

  • The resulting model is used to predict effort for the excluded cases

  • This method is repeated for all subsets and the mean magnitude of relative errors (MMRE) and prediction accuracy PRED (l) across all K trials is calculated

Sensitivity analysis
Sensitivity Analysis model with local calibration

Scenario 1

  • Evaluated the sensitivity of the model’s results to the variability of key parameters

  • 2 Scenarios were developed based on the standard deviation in the volatility life cycle profile (Θl)

  • Scenario 1:

    • None of the requirements changes occur in the conceptualize phase

  • Scenario 2:

    • 66% of the changes occur in the conceptualize phase

% Volatility per Life Cycle Phase

Operational Test & Eval

Transition to Operations



Scenario 2

% Volatility per Life Cycle Phase

Operational Test & Eval

Transition to Operations



Source: Guidelines for the Economic Analysis of Projects (1997)

Sensitivity analysis results
Sensitivity Analysis Results model with local calibration

  • The performance of the model scenarios was calculated using the K-fold cross-validation method

  • The improvement in performance of the model with the requirements volatility scale factor over Academic COSYSMO remains largely unchanged

Conclusions model with local calibration

  • The volatility of requirements throughout a system’s life cycle was found to be statistically significant predictor (p = 0.03) of systems engineering effort

  • The prediction accuracy of Academic COSYSMO improved by incorporating a requirements volatility factor into the model

  • Cross validation and sensitivity analyses were performed to demonstrate the robustness of the model in predicting effort for new projects

Threats to validity
Threats to Validity model with local calibration

Internal validity

Data collected from an engineering organization is influenced by many factors and all the relevant variables cannot be fully controlled as they would be in a true experiment

External validity

Limited by the organizations that contribute data and the background of the industry experts that participate in the research

While a model calibrated to a single organization’s data can be useful, it is less than fully definitive for other organizations.


Potential inaccuracies in effort and requirements metrics due to changes in business systems and practices over the time span of the projects

Mitigated though:

The use of consistent counting rules and involvement from personnel familiar with the projects

Holding workshops to ensure survey respondents had a consistent understanding of the questions


Future research
Future Research model with local calibration

  • Increase the quantity and diversity of the data – calibrate the model using data from additional organizations

  • Evaluate the potential interaction between requirements volatility and reuse

  • Further investigation of the impact of systems engineering effort depending on the type of change: added, modified, and deleted

References model with local calibration

Boehm, B., Abts, C., Brown, A.W., Chulani, S., Clark, B., Horowitz, E., Madachy, R., Reifer, D.J., and Steece, B. (2000). Software Cost Estimation with COCOMO II. Prentice Hall

Department of Defense (2010). Quadrennial Defense Review Report

Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009). “Understanding the effects of requirements volatility in software engineering by using analytical modeling and software process simulation.” The Journal of Systems and Software. Vol. 82, pp 1568-1577.

Wang, G., Boehm, B., Valerdi, R., and Shernoff, A. (2008). “Proposed Modification to COSYSMO Estimating Relationship.” Technical Report. University of Southern California, Center for Systems and Software Engineering.

General Accounting Office (2004). Stronger Management Practices are Needed to Improve DOD’s Software-intensive Weapon Acquisitions (GAO-04-393). Defense Acquisitions.

Hammer, T., Huffman, L., and Rosenberg, L. (1998). “Doing requirements right the first time.” Crosstalk, the Journal of Defense Software Engineering. Pp 20-25.

Houston, Dan X. (2000). A Software Project Simulation Model for Risk Management, Ph.D. Dissertation, Arizona State University

ISO/IEC (2008). ISO/IEC 15288:2008 (E) Systems Engineering - System Life Cycle Processes.

Kotonya, G., Sommerville, I., (1998). Requirements Engineering: Processes and Techniques. John Wiley and Sons, Ltd.

Malaiya, Y., and Denton, J. (1999). “Requirements Volatility and Defect Density.” Proceedings of the International Symposium on Software Reliability Engineering.

MIL-STD-498. 1994. Software Development and Documentation. U.S. Department of Defense.

Nguyen, V. and Boehm, B. (2010). A COCOMO Extension for Software Maintenance. 25th International Forum on COCOMO and Systems/Software Cost Modeling

Valerdi, R. (2005). The constructive systems engineering cost model (COSYSMO). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department.

Zowghi, D. and Nurmuliani, N. (2002). A Study of the Impact of Requirements Volatility on Software Project Performance. Proceedings of the Ninth Asia-Pacific Software Engineering Conference


Back up

Back-up model with local calibration

Scale factor correlation matrix
Scale Factor Correlation Matrix model with local calibration

The correlation between the Requirements Volatility Scale factor and COSYSMO cost drivers that are potential sources of volatility was evaluated

  • The correlations between predictors were below the correlation threshold of 0.66 used for COCOMO II and COSYSMO


Cumulative volatility
Cumulative Volatility model with local calibration



REVL (% of baseline requirements changes)



Operational Test & Eval

Transition to Operations