Revitalization of Systems Engineering

1. Revitalization of Systems Engineering Kristen Baldwin Acting Director, Systems and Software Engineering Office of the Deputy Under Secretary of Defense (Acquisition and Technology) presented to: NDIA Manufacturing Division Meeting August 13, 2008

2. AT&L Organization USD, Acquisition Technology & Logistics DUSD, Acquisition & Technology Dir, Joint Advanced Concepts Dir, Systems and Software Engineering Dir, Portfolio Systems Acquisition Defense Acquisition University Defense Procurement and Acquisition Policy Industrial Programs Small Business Programs Defense Contract Management Agency

3. Briefing Outline • Where is DoD Systems Engineering now? • SE Revitalization Status • Analysis of Failure • Key near-term initiatives • Where does SE need to be? • Strategic Direction • Enhanced SE Institutionalize SE Key Initiatives Revitalize SE 2003 2008 2010 2015

4. SSE Functions • Acquisition Program Support • Decision Support to Senior DoD Leadership • Mentoring Acquisition Programs • Technical Reviews of Key Program Documentation • Policy, Guidance, Best Practices • Acquisition Workforce Education and Training • SE, Test, Production Quality and Manufacturing, Software • Systemic Root Cause Analysis • “Discipline” Expertise • Risk  CMMI  Modeling/Simulation • Reliability  Software  Assurance/Cyber • Architectures  Energy  Comm/Networking • Test/Eval  Safety  System of Systems

5. Revitalizing DoD SE • Issued DoD-wide SE policy – focused effort on up front, sound technical planning • Issued guidance on SE, test and evaluation (T&E) and software • Revised SE & T&E curricula at Defense Acquisition University • Established SE Forum—senior-level focus across DoD • Instituted system-level assessments in support of OSD major acquisition program oversight role • Executed 83 reviews since March 2004 • Integrated, software, system assurance, energy, M&S and system of systems into SE revitalization efforts • Established closer working relationships with industry and academia Prescribing Good Practices Is Not Enough…

6. Analysis of Program Failure • Acquisition cost growth over 11 years (SAR data FY 1995–2005): • Estimation changes: $201B • Engineering changes: $147B • Schedule changes: $70B • Over the past 10 years, DoD systems have experienced a 33% cost growth due to “RDT&E mistakes” • DoD IOT&E results, FY2001-2007 • 35 systems; mix of ACAT II, 1C, 1D across 3 Services • 57% of programs found to be “Not Suitable or Effective” • Approx. 49% were deemed “Not Suitable” or partially NS • Approx. 34% were deemed “Not Effective” or partially NE

7. Analysis of Program Failure Specific Causes of Program Failure (Systemic Root Cause Analysis findings)

8. Key Initiatives2008-2009

9. SSE FY08/09 Thrust Areas • Systemic Root Cause Analysis – the collection and analysis of systemic program performance issues, determination of root causes, and development of corrective action • System Assurance – the reduction of vulnerability to malicious intent in our systems, considering the full spectrum security of information, technology and hardware/software components • Software Engineering Competency – the focus on software engineering as a critical element of complex systems acquisition, and strategic initiatives to ensure future Defense software demands can be met by government and industry • Systems of Systems – augmenting acquisition and engineering practices to better plan, develop, and manage interdependent systems of systems • Early Systems Engineering – the institution of technical management and engineering practices prior to program initiation (MS B) in order to enable risk informed, balanced acquisition and budgeting decisions

10. SSE FY08/09 Thrust Areas (cont’d) • Revitalizing Developmental Test & Evaluation – comprehensive strategy for early T&E involvement and integrated testing to manage technical risks across the acquisition system, sustaining systems and capabilities, and rebuilding government T&E expertise. • System Engineering Research UARC – establishment of a system engineering research program • Energy – reducing DoD energy consumption across our operations, to include force development, deployment, and support • Reliability – ensuring rigor, discipline and an integrated approach in systems engineering practices and T&E strategies for improving the reliability of systems • Safety – integrating safety process advances into Department acquisition policies • Modeling and Simulation – ensuring rigor in M&S policy and guidance to advance the state-of-practice; execute acquisition M&S Master Plan

11. Manufacturing Readiness Levels • MRL/EMRL Background: • MRLs documented by a “body of knowledge”, the MRA Deskbook and piloted in selected Air Force acquisition programs • EMRLs used as exit criteria on ACAT ID programs since 2002 • ODDR&E and OSD SSE collaborating to: • Transition from EMRL to robust MRL body of knowledge • Align metrics with new DoD life cycle management framework • Leverage acquisition milestones and SE technical reviews to assess manufacturing maturity • Draft policy to integrate manufacturing into acquisition decisions • Doubtful that it will go into the pending DoDI 5000.02 update • ACTION: OSD SSE has requested the NDIA Engineering and Manufacturing Committees to: • Review MRL criteria and its time phasing across the acquisition lifecycle • Better define production “relevant” and “representative” environments • Assess use of MRAs/Production Readiness Reviews on supplier base • Provide comments on the MRA Deskbook

12. Manufacturing Readiness Levels (cont.) • Goals: • Increase emphasis on manufacturing throughout the acquisition lifecycle • Ensure considerations of manufacturing and producibility through established service mechanisms • Revised DoDI 5000.02 mandates consideration of manufacturing maturity during each acquisition phase and at each milestone • Areas to continue to work: • Appropriate assessment of manufacturing at MS A and MS B • Ensuring manufacturing assessments are not stand-alone • Update engineering training & guidance accordingly

13. Integration Readiness Levels Background: • Systemic analysis has shown integration issues in ~ 45% of PSRs • Contributes to cost and schedule issues • DUSD(A&T) requested SSE study and develop integration readiness levels for use in acquisition • SSE initiating a collaborative effort to develop quantifiable integration metrics Product: Draft set of IRLs in 120 days ACTION: • Identify existing IRL activities/products and interest to support comprehensive study

14. Human Systems Integration Hardware Total System Approach Human Software Human Systems Integration (HSI). The interdisciplinary technical and management processes for integrating human considerations within and across all system elements; an essential enabler to systems engineering practice.

15. Recent HSI Guidance and Direction • H.R. 1585 National Defense Authorization Act for FY 2008: Section 231 • Directs the Secretary to: (1) coordinate and manage human systems integration activities throughout DOD acquisition programs; and (2) designate a senior DOD official to be responsible for such effort. • USD(AT&L) memorandum, Apr 3, 2008 • Designates DUSD(A&T) as the senior official responsible for coordination and management of HSI activities throughout DoD acquisition programs, with DUSD(S&T), Director, Biological Systems as the co-lead for S&T • FY09 HASC language • directs the USD(AT&L) to develop a comprehensive plan for funding and implementing HSI through all phases of science, research, and acquisition. Report due by March 15, 2009 ACTION: We are soliciting input from DoD and Industry partners to help respond to Congressional Direction. Specifically: • What are key HSI activities, initiatives, capabilities that exist today • Where gaps or issues exist

16. Our Challenge: Institutionalization2010-2015

17. SSE Strategic Direction • Cut cycle time to field capability by embedding systems engineering engagement with requirements and early phase acquisition • Change engineering culture, both vertically and horizontally, across Government and industry • Implement Human Capital Strategy to assure DoD workforce competency • Advance Engineering State-of-Practice through research transition • Implement comprehensive solution for system assurance/program protection/cyber security

18. Improve Knowledge through Technical Foundation Systems Engineering is effective when it informs, and is informed by, other Acquisition process owners MDD MS A MS B Business Decisions Agreement to pursue a materiel solution Selection of a preferred solution Engineering Support Preferred System Analysis National Research Council Uncertainty Formal Program Start Preferred System Concept PDR Technology Maturation And Prototyping “Pre-Milestone A and Early-Phase Systems Engineering” Jan 2008 System Level Specs AoA CDR Preliminary Design Completed Design Technology Development Material Solution Analysis

19. Questions?

20. Systems Engineering Research (SER) University Affiliated Research Center (UARC) • Overview: SER UARC implementation is intended to continue efforts to advance practice of SE across DoD enterprise • Approved by the DDR&E, October 2007 • Committed funding of $10M over 5 years • Mission: Research and analyze advanced and emerging systems engineering practices and relevant technologies to address the full spectrum of DoD systems across the Department, from capability areas, enterprise systems, systems of systems, and interoperability down to subsystems and configuration items with the goal of ensuring consistency and systems engineering excellence throughout the acquisition cycle. • Status: Request for Proposal nearing completion • All domestic institutions with engineering programs notified (>700) • Draft RFP reviewed by over 30 universities • Key Dates (planned): • RFP release – May 2008 • Contract award – July/August 2008

21. Initial SE Research Tasks • Examine and recommend areas for advancing current SE methods, processes, and tools as they are applied across the DoD acquisition life cycle focusing on three different development environments: individual weapons systems, SoS, and network-centric systems • Characterize SE effectiveness within the context of DoD acquisition and identify methods of measurement suitable for application to project execution organizations (e.g., defense contractors), program management organizations (e.g., project managers and program executive offices), and oversight organizations (e.g., OUSD(AT&L))

22. SoS SE Guide (SEG) v 1.0 Revision • Draft SoS SEG version 0.9 released December 2006 • March – September 2007 -- ‘Pilot Phase’ • Structured review of version 0.9 with SoS SE practitioners and researchers • Results offer ‘boots on the ground’ basis for guide revision • Draft SoS SEG version 1.0 released for comment December 10, 2007 • Available at http://www.acq.osd.mil/sse/ssa/guidance.html • Formal comment acceptance closed on February 14, 2008 • ~600 comments were adjudicated by April 1 • Final release of version 1.0 by June 2008

23. Developing & Evolving SoS Architecture Monitoring & Assessing Changes Understanding Systems & relationships Translating Capability Objectives Addressing new requirements & options Addressing new requirements & options Addressing requirements & solution options Orchestrating upgrades to SoS Orchestrating upgrades to SoS Orchestrating upgrades to SoS View of SoS Systems Engineering Assessing SoS Performance SoS Systems Multiple, possibly concurrent increments

24. DoD Software Core Competency • DUSD(A&T) vision for software engineering (SWE) leadership and world class capability • Issue: Software Growth • Software Requirements Growth: 80% of system functionality [CSIS] • Software Size Growth: 20 million ESLOC [CSIS] • Software Overruns: 50% of SW efforts [Standish Group] • Issue: SE and SW have not been well integrated on projects • Top Software Issues and Project Areas

25. System Assurance • We continue to be concerned with assurance of our critical DoD assets: • Critical information • Critical technologies • Critical systems • Observations: • Increasing numbers of network attacks (internal and external to DoD) • Broader attack space • Trends that exacerbate our concerns: • Globalization of our contracts, expanding the number of international participants in our system developments • Complex contracting arrangements that further decrease transparency below prime, and visibility into individual components These trends increase the opportunity for access to our critical assets and for tampering 25

26. Anti - Tamper Configuration Program Software Manager Protection Assurance Quality Engineer Software Trusted Program Protection Foundry Manager Initiative Center Reliability Safety For Information Engineer Engineer Assured Assurance Systems Software Engineer System Assurance for the Program Manager Lack of focused guidance to counter the threat

27. Acquisition Path Forward • Create a framework to integrate multiple security policies and oversight • Leverage Program Protection requirement for all acquisition programs as set by DoDD 5200.39 policy • Integrate all assurance oversight, planning, and risk mitigation activity at the system level • Develop Guidance on Engineering for System Assurance • Guidebook on Engineering for Assurance for program managers/engineers • Defines how assurance can be incorporated into system engineering and design: • e.g. Isolation, Redundancy, Quality, and Fault Analysis Raise the bar: Awareness - Knowledge of the supply chain - Who has access to our critical assets Protection - Protect critical assets through security practices - Design our systems for assurance

28. USD(AT&L) Direction DoD SE Leaders Deputy Secretary of Defense Leadership Direction for SE • AT&L Source Document Objectives: • Capabilities: Balance and rationalize requirements; Review key capability areas to seek greater efficiency; Evaluate mission capability gaps against technology opportunities • Programs: Accurately price programs and insist schedule and budget reflect realistic pricing, recognizing technical and integration risks; Build prototypes and conduct experiments to provide options; Arm the Program Manager with tools • People: Play an appropriate stewardship role for the science and engineering community • Strategic Objectives identified by the SSE Forum: • Enhance SE pre-MS B to ensure programs succeed • Transform Architecture to support Systems Engineering Enterprise • Establish a Human Capital Strategy for SE • Assess needs for additional SE Tools “We believe we have gone too far in reducing Systems Engineering capability … we need to re-build.” Honorable Secretary England, 7 November 2007, PEOSYSCOM Conference Keynote

29. SSE FY10-15 Program Outcomes (1/2) • Create enhanced Systems Engineering capability (pre-MS A and between MS A&B) • Provide SSE resources (support, tools) for Technology Development phase • Integrate SSE with Requirements, Programming and Management • Support pre-program and capability analyses (AoAs) • Establish AT&L matrix support to CPMs and SoSs • Develop a Human Capital Strategy for System and Software Engineering (SSE) • Advance Systems Engineering state of practice through research by leveraging and transitioning UARC recommendations • Revitalize Developmental Test and Evaluation to support faster fielding and reliability growth testing

30. SSE FY10-15 Program Outcomes (2/2) • Transition and promulgate SSE tools, practices • Institutionalize PSRs, SEPs, Software, Reliability and T&E practice, etc. among Services, Agencies, Industry Transition Partners • Transform lessons learned to predictive measures • Transparent data, correlate program findings with external databases (e.g., DAMIR, PoPs, PA&E) • Implement corrective actions • Value Energy in acquisition processes • Institutionalize energy in Milestone reviews and portfolio management • Ensure fuel demand considerations in AOA and all program analysis • Integrate assurance and security practice into SE • Transition from FY08/09 cyber and system assurance pilot activities