System of Systems Engineering Cost Modeling: What Makes It Different from Traditional Systems Engineering Cost Modelin

1. System of Systems Engineering Cost Modeling: What Makes It Different from Traditional Systems Engineering Cost Modeling Jo Ann Lane USC CSSE jolane@usc.edu Ricardo Valerdi MIT rvalerdi@mit.edu COCOMO Forum October 2007

2. Overview • Overview of SoSs and SoSE • Summary of key comparative research and pilot studies • Comparison of traditional SE and SoSE cost models • Conclusions ©USC-CSSE

3. Relationships between Traditional Systems, SoSs, and Complex Systems [Kuras and White, INCOSE, 2005] ©USC-CSSE

4. What is a “System of Systems”? • Very large systems developed by creating a framework or architecture to integrate component systems • SoS component systems independently developed and managed • New or existing systems in various stages of development/evolution • Have their own purpose • Can dynamically come and go from SoS • SoS exhibits emergent behavior not otherwise achievable by component systems • Typical domains • Business: Enterprise-wide and cross-enterprise integration to support core business enterprise operations across functional and geographical areas • Military: Dynamic communications infrastructure to support operations in a constantly changing, sometimes adversarial, environment Based on Mark Maier’s SoS definition [Maier, 1998] For more on SoS definitions, see [Lane and Valerdi, 2007] ©USC-CSSE

5. TSE, SOSE, and Related Industry Standards • Current standards for TSE processes • ISO/IEC Standard 15288 – describes system processes from system conception through system retirement • ANSI/EIA Standard 632 – detailed processes for conceptualization, development and transition to operation (subset of ISO/IEC 15288) • SEI’s Capability Maturity Model Integrated – framework for defining an organization’s standard processes and procedures • Defense Acquisition Guidebook (DAG) – references above as examples of best practices, but defines own set of processes • Guidelines for SoSE • SoS SE Draft Guidebook (extension to DAG) ©USC-CSSE

6. SoSE Compared to TSE Activities • SoSs have their own characteristics and associated challenges that are different from traditional systems and large, complex systems • Reported areas of difference • Architecting • Prototypes/experimentation/tradeoffs • Scope of SoS • SoS performance • Maintenance and evolution • Key challenges for DoD SoSE • Business model and incentives to encourage working together (SoS level) • Doing the necessary tradeoffs at the SoS level • Human-system integration • Commonality of data, architecture, and business strategies (SoS level) • Removing multiple decision making layers • Requiring accountability at the enterprise level • Evolution management • Maturity of technology ©USC-CSSE

7. Recent Research Findings* • Many types of SoS • SoS Engineering Teams: Varying degrees of responsibility and authority • Incorporating many agile-like approaches to handle • Multiple constituent systems • Asynchronous activities and events • Quickly take advantage of opportunities as they appear • SoSE must • Support multiple purposes and visions • Requires significantly more negotiation • Is content to satisfice rather than optimize • SoSE activities map to traditional SE activities (e.g., DAG and EIA 632), but take on a different focus and scope * Based on USC CSSE SoSE cost model research, MIT Systems Engineering Advancement Research Initiative, and OSD SoS SE pilot studies ©USC-CSSE

8. Translating capability objectives Assessing (actual) performance to capability objectives Typically not the role of the SE but key to SoS [assume these are fixed] Developing, evolving and maintaining SoS design/arch Understanding systems & relationships (includes plans) Block upgrade process for SoS External Influences Persistent framework overlay on systems in SoS [architecture] Monitoring & assessing changes Large role of external influences Addressing new requirements & options Orchestrating upgrades to SoS Relationship Among Core Elements of SoS SE ©USC-CSSE

9. COSOSIMO Planning, Requirements Management, and Architecting (PRA) Source Selection and Supplier Oversight (SO) SoS Integration and Testing (I&T) SoSE Core Element Mapping to COSOSIMO Sub-models Translating capability objectives Understanding systems & relationships (includes plans) Developing, evolving and maintaining SoS design/arch Addressing new requirements & options Orchestrating upgrades to SoS Assessing (actual) performance to capability objectives Monitoring & assessing changes ©USC-CSSE

10. Comparison of System of Interest Focus Areas ©USC-CSSE

11. Comparison of System of Interest Focus Areas (continued) ©USC-CSSE

12. Areas of Emphasis in SE and SoS SE* * [Valerdi, et al., 2007] ©USC-CSSE

13. Engineering Cost and Schedule—What Do Engineering Cost Models Look At? • Engineering product characteristics • Processes used to develop the product • Skills and experience levels of the technical staff responsible for development of product • Size drivers used to compute nominal effort • Cost drivers used to adjust nominal effort up or down ©USC-CSSE

14. Comparison of Cost Model Parameters ©USC-CSSE

15. Impact of Differences to SE Cost Model • Elements of SE that are not of major concern/driver of effort or can be handled as a constant • Number of system algorithms • Number of recursion levels in the design • Migration complexity • Number/diversity of platforms/installations • Level of documentation • Multi-site coordination • Elements of SoS SE not adequately addressed in SE cost model • Analysis of capabilities to determine SoS and constituent system requirements • Impact of negotiations required to develop architecture and enhancements • Impact of number of constituent systems and organizations that own and manage the constituent systems • The asynchronous nature of constituent component changes and the accommodation of partial increment implementations operationally • Additional effort required to accommodate unplanned constituent component changes • Uncertainty with respect to funding sources and impact on required negotiations ©USC-CSSE

16. Conclusions to Date and Future Work • Recent research efforts articulate some significant differences between TSE and SoSE • At a basic level, many of the activities are the same • However, there are significant differences to the scope and focus of the engineering activities • Major focus of/impacts to SoSs • Legacy systems and the independent control of them • Focus on inter-relationships between relatively independent components vs. integration of tightly coupled components • Elements of SoS SE not adequately addressed in existing traditional SE cost models • COSOSIMO strives to address these SoS SE differences • We need industry support to better understand and calibrate these differences ©USC-CSSE

17. Backup Charts

18. SoSE Compared to Traditional SE Activities: Reported Differences • Architecting • Architecting composability vs. decomposition (Meilich 2006) • Net-friendly vs. hierarchical (Meilich 2006) • Prototypes/experimentation/tradeoffs • Early tradeoffs/evaluations of alternatives (Finley 2006) • Intense concept phase analysis followed by continuous anticipation; aided by ongoing experimentation (USAF 2005) • Modeling and simulation, in particular to better understand “emergent behaviors” (Finley 2006) • First order tradeoffs above the component systems level (e.g., optimization at the SoS level, instead of at the component system level) (Garber 2006) • Discovery and application of convergence protocols (USAF 2005) ©USC-CSSE

19. SoSE Compared to Traditional SE Activities: Reported Differences (continued) • Scope and performance • Added “ilities” such as flexibility, adaptability, composability (USAF 2005) • Human as part of the SoS (Siel 2006, Meilich 2006, USAF 2005) • Organizational scope defined at runtime instead of at system development time (Meilich 2006) • Dynamic reconfiguration of architecture as needs change (Meilich 2006) • Maintenance and evolution • Component systems separately acquired and continue to be managed as independent systems (USAF 2005) ©USC-CSSE

20. SoSE Core Element Descriptions • Translating capability objectives • Developing a basic understanding of the expectations of the SoS and the core requirements for meeting these expectations, independent of the systems that comprise the SoS • Understanding systems and relationships • In a SoS, the focus is on the systems which contribute to SoS SE capabilities and their interrelationships (as opposed to in a system, the focus is on boundaries and interfaces) • Assessing actual performance to capability objectives • Establishing SoS metrics and methods for assessing performance and conducting evaluations of actual performance using metrics and methods • Developing, evolving, and maintaining an SoS architecture/design • Establishing and maintaining a persistent framework for addressing the evolution of the SoS to meet user needs, including possible changes in systems functionality, performance or interfaces ©USC-CSSE

21. SoSE Core Element Descriptions (continued) • Monitoring and assessing changes • Monitoring proposed or potential changes and assessing their impacts to: • Identify opportunities for enhanced functionality & performance, and • Preclude or mitigate problems for the SoS and constituent systems (this may include negotiating with the constituent system over how the change is made, in order to preclude SoS impacts • Addressing new requirements and options • Reviewing, prioritizing, and determining which SoS requirements to implement next • Orchestrating upgrades to SoS • Planning, facilitating, integrating, testing changes in systems to meet SoS needs ©USC-CSSE

22. References Dahmann, J. (2007); “Systems of Systems Challenges for Systems Engineering”, Systems and Software Technology Conference, June 2007. DiMario, Mike (2006); “System of Systems Characteristics and Interoperability in Joint Command Control”, Proceedings of the 2nd Annual System of Systems Engineering Conference Electronic Industries Alliance (1999); EIA Standard 632: Processes for Engineering a System Finley, James (2006); “Keynote Address”, Proceedings of the 2nd Annual System of Systems Engineering Conference Garber, Vitalij (2006); “Keynote Presentation”, Proceedings of the 2nd Annual System of Systems Engineering Conference INCOSE (2006); Systems Engineering Handbook, Version 3, INCOSE-TP-2003-002-03 Krygiel, A. (1999); Behind the Wizard’s Curtain; CCRP Publication Series, July, 1999, p. 33 Kuras, M. L., White, B. E., Engineering Enterprises Using Complex-System Engineering, INCOSE Symposium 2005. Lane, J., Valerdi, R., “Synthesizing System-of-Systems Concepts for Use in Cost Modeling,” Systems Engineering, Vol. 10, No. 4, December 2007. Maier, M. (1998); “Architecting Principles for Systems-of-Systems”; Systems Engineering, Vol. 1, No. 4 (pp 267-284) Meilich, Abe (2006); “System of Systems Engineering (SoSE) and Architecture Challenges in a Net Centric Environment”, Proceedings of the 2nd Annual System of Systems Engineering Conference Pair, Major General Carlos (2006); “Keynote Presentation”, Proceedings of the 2nd Annual System of Systems Engineering Conference Proceedings of AFOSR SoSE Workshop, Sponsored by Purdue University, 17-18 May 2006 Proceedings of Society for Design and Process Science 9th World Conference on Integrated Design and Process Technology, San Diego, CA, 25-30 June 2006 Siel, Carl (2006); “Keynote Presentation”, Proceedings of the 2nd Annual System of Systems Engineering Conference Valerdi, R. (2005); Constructive Systems Engineering Cost Model. PhD. Dissertation, University of Southern California Valerdi, R., Ross, A., Rhodes, D., “A Framework for Evolving System of Systems Engineering,” CrossTalk - The Journal of Defense Software Engineering, October 2007. United States Air Force Scientific Advisory Board (2005); Report on System-of-Systems Engineering for Air Force Capability Development; Public Release SAB-TR-05-04 ©USC-CSSE