Introduction Wilson Rosa, AFCAA wilson.rosa@pentagon.af.mil

1. IntroductionWilson Rosa, AFCAAwilson.rosa@pentagon.af.mil CSSE Annual Research Review March 8, 2010

2. Agenda Overview

3. Project Background Goal is to improve the quality and consistency of estimating methods across cost agencies and program offices through guidance, standardization, and knowledge sharing. Project led by the Air Force Cost Analysis Agency (AFCAA) working with service cost agencies, and assisted by University of Southern California and Naval Postgraduate School We will publish the AFCAASoftware Cost Estimation Metrics Manual to help analysts and decision makers develop accurate, easy and quick software cost estimates for avionics, space, ground, and shipboard platforms.

4. Stakeholder Communities SLIM-Estimate™ • Research is collaborative across heterogeneous stakeholder communities who have helped us in refining our data definition framework, domain taxonomy and providing us project data. • Government agencies • Tool Vendors • Industry • Academia TruePlanning® by PRICE Systems 4

5. Research Objectives Establish a robust and cost effective software metrics collection process and knowledge base that supports the data needs of the United States Department of Defense (DoD) Enhance the utility of the collected data to program oversight and management Support academic and commercial research into improved cost estimation of future DoD software-intensive systems

6. Software Cost Model Calibration • Most program offices and support contractors rely heavily on software cost models • May have not been calibrated with most recent DoD data • Calibration with recent data (2002-Present) will help increase program office estimating accuracy

7. AFCAA Software Cost Estimation Metrics ManualTable of Contents • Chapter 1: Software Estimation Principles • Chapter 2: Product Sizing • Chapter 3: Product Growth • Chapter 4: Effective SLOC • Chapter 5: Historical Productivity • Chapter 6: Model Calibration • Chapter 7: Calibrated SLIM-ESTIMATE • Chapter 8: Cost Risk and Uncertainty Metrics • Chapter 9: Data Normalization • Chapter 10: Software Resource Data Report • Chapter 11: Software Maintenance • Chapter 12: Lessons Learned

8. Manual Special Features • Augment NCCA/AFCAA Software Cost Handbook: • Default Equivalent Size Inputs (DM, CM, IM, SU, AA, UNFM) • Productivity Benchmarks by Operating Environment, Application Domain, and Software Size • Empirical Code, Effort, and Schedule Growth Measures derived from SRDRs • Empirically Based Cost Risk and Uncertainty Analysis Metrics • Calibrated SLIM-Estimate™ using most recent SRDR data • Mapping between COCOMO, SEER, True S cost drivers • Empirical Dataset for COCOMO, True S, and SEER Calibration • Software Maintenance Parameters

9. Manual Special Features (Cont.) • Guidelines for reconciling inconsistent data • Standard Definitions (Application Domain, SLOC, etc.) • Address issues related to incremental development (overlaps, early-increment breakage, integration complexity growth, deleted software, relations to maintenance) and version management (a form of product line development and evolution). • Impact of Next Generation Paradigms – Model Driven Architecture, Net-Centricity, Systems of Systems, etc.

10. Agenda Overview (Barry Boehm) Data Analysis Software Sizing Recent Workshop Results Conclusions

11. DoD Empirical Data • Data quality and standardization issues • No reporting of Equivalent Size Inputs – CM, DM, IM, SU, AA, UNFM, Type • No common SLOC reporting – logical, physical, etc. • No standard definitions – Application Domain, Build, Increment, Spiral,… • No common effort reporting – analysis, design, code, test, CM, QA,… • No common code counting tool • Product size only reported in lines of code • No reporting of quality measures – defect density, defect containment, etc. • Limited empirical research within DoD on other contributors to productivity besides effort and size: • Operating Environment, Application Domain, and Product Complexity • Personnel Capability • Required Reliability • Quality – Defect Density, Defect Containment • Integrating code from previous deliveries – Builds, Spirals, Increments, etc. • Converting to Equivalent SLOC • Categories like Modified, Reused, Adopted, Managed, and Used add no value unless they translate into single or unique narrow ranges of DM, CM, and IM parameter values. We have seen no empirical evidence that they do…

12. SRDR Data Source

13. Data Collection and Analysis Approach Be sensitive to the application domain Embrace the full life cycle and Incremental Commitment Model Be able to collect data by phase, project and/or build or increment Items to collect SLOC reporting – logical, physical, NCSS, etc. Requirements Volatility and Reuse Modified or Adopted using DM, CM, IM; SU, UNFM as appropriate Definitions for Application Types, Development Phase, Lifecycle Model,… Effort reporting – phase and activity Quality measures – defects, MTBF, etc.

14. Data Normalization Strategy Interview program offices and developers to obtain additional information not captured in SRDRs… Modification Type – auto generated, re-hosted, translated, modified Source – in-house, third party, Prior Build, Prior Spiral, etc. Degree-of-Modification – %DM, %CM, %IM; SU, UNFM as appropriate Requirements Volatility -- % of ESLOC reworked or deleted due to requirements volatility Method – Model Driven Architecture, Object-Oriented, Traditional Cost Model Parameters – True S, SEER, COCOMO, SLIM