A Survey of Systems Engineering Effectiveness
Download
1 / 74

A Survey of Systems Engineering Effectiveness by: NDIA Systems Engineering - PowerPoint PPT Presentation


  • 106 Views
  • Uploaded on
  • Presentation posted in: General

A Survey of Systems Engineering Effectiveness by: NDIA Systems Engineering Effectiveness Committee INCOSE - Orlando Chapter Geoff Draper Harris Corporation gdraper@harris.com February 28, 2008. Agenda. Introduction – NDIA Systems Engineering Division (SED) Organization and Committees

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha

Download Presentation

A Survey of Systems Engineering Effectiveness by: NDIA Systems Engineering

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


A Survey of Systems Engineering Effectiveness

by: NDIA Systems Engineering

Effectiveness Committee

INCOSE - Orlando Chapter

Geoff Draper

Harris Corporation

gdraper@harris.com

February 28, 2008


Agenda

  • Introduction – NDIA Systems Engineering Division (SED)

    • Organization and Committees

  • NDIA Systems Effectiveness Committee:

  • A Survey of Systems Engineering Effectiveness

  • http://www.sei.cmu.edu/pub/documents/07.reports/07sr014.pdf

NDIA SE Division web page:

http://www.ndia.org/Template.cfm?Section=NDIA_Divisions_Page&Template=/TaggedPage/TaggedPageDisplay.cfm&TPLID=3&ContentID=677


NDIA SE Division – Org Chart


Key NDIA SE Division Initiatives

  • OSD (A&T) Initiatives

  • CMMI Co-Sponsor

  • Conferences

    • NDIA Systems Engineering Conference

    • CMMI Technology Conference

    • Net-Centric Operations Conference

  • Committees

  • Task Groups / Workshops

  • Awards

    • Ferguson Award for SE Excellence

    • Top 5 Government Programs


Effective Systems Engineering: What’s the Payoff for Program Performance?

NDIA Systems Engineering

Effectiveness Committee

CMMI Technology Conference

November 15, 2007


Does this sound familiar?

These are the ASSERTIONS, but what are the FACTS?


The Problem

  • It is difficult to justify the costs of SE in terms that program managers and corporate managers can relate to.

    • The costs of SE are evident

      • Time

      • Effort

    • The benefits are less obvious and less tangible

      • Cost avoidance (e.g., reduction of rework from interface mismatches

      • Risk avoidance (e.g., early risk identification and mitigation)

      • Improved efficiency (e.g., clearer organizational boundaries and interfaces)

      • Better products (e.g., better understanding and satisfaction of stakeholder needs)

How can we quantify the effectiveness and value of SE?How does SE benefit program performance?


Systems Engineering Effectiveness Survey(2004-2007)

  • Hypothesis: The effective performance of SE best practices on a development program yields quantifiable improvements in the program execution (e.g., improved cost performance, schedule performance, technical performance).

  • Objectives:

    • Characterize effective SE practices

    • Correlate SE practices with measures of program performance

  • Approach:

    • Distribute survey to NDIA companies

    • SEI analysis and correlation of responses

  • Survey Areas:

  • Process definitionTrade studiesProject reviews

  • Project planningInterfacesValidation

  • Risk managementProduct structureConfiguration mgmt

  • Requirements developmentProduct integrationMetrics

  • Requirements managementTest and verification


The ChallengePrevious Studies - Summary

Mink, 2007


The Challenge -Supporting Evidence

Gruhl, Werner (1992), Lessons Learned: Cost/Schedule Assessment, Internal Presentation, NASA Comptroller’s Office

Honour, Eric (2004), Understanding the Value of Systems Engineering, Proceedings of the 14th Annual INCOSE International Symposium


14 Process Areas

31 Goals

87 Practices

199 Work Products

CMMI-SE/SW/IPPD v1.1

  • 25 Process Areas

  • 179 Goals

  • 614 Practices

  • 476 Work Products

SystemsEngineering-related Filter

Size Constraint Filter

13 Process Areas

23 Goals

45 Practices

71 Work Products

Considered significant to Systems Engineering

Survey Development

Survey content is based on a recognized standard (CMMI)


Survey Methodology(Conducted: 2004-2007)


Analysis

  • Perf = f (PC, PE, SEC, AC)

  • where:Perf = Project PerformancePC = Project ChallengePE = Project EnvironmentAC = Acquirer Capability

  • SEC = Systems Engineering Capability

  • SEC can be further decomposed as:

    • Project Planning

    • Project Monitoring and Control

    • Risk Management

    • Requirements Development and Management

    • Technical Solution

      • Trade Studies

      • Product Architecture

    • Product Integration

    • Verification

    • Validation

    • Configuration Management

    • IPT-Based Capability

SE capabilities and analyses are fully defined by mappings of associated survey question responses


Project Challenge (PC)

Overall SE Capability (SEC)

Analysis -Validation of Survey Responses

Acquirer Capability (AC)

Project Performance (Perf)

Analyzed distributions, variability, relationships…

To ensure statistical rigor and relevance


Total SE Capability (SEC) vs. Project Performance (Perf)

Notation

Projects with better Systems Engineering Capabilities deliver better Project Performance (cost, schedule, functionality)


Relating Project Performance to Project Challenge and SE Capability

Project challenge factors:

  • Life cycle phases

  • Project characteristics (e.g., size, effort, duration, volatility)

  • Technical complexity

  • Teaming relationships

Projects with better Systems Engineering Capabilities are better able to overcome challenging environments


Results1. Product Architecture and Performance

Projects with better Product Architecture show a

“Moderately Strong / Strong” Positive Relationship with Performance


Results2. Trade Studies and Project Performance

Projects with better Trade Studies show a

“Moderately Strong / Strong” Positive Relationship with Performance


Results3. Technical Solution and Project Performance

Projects with better Technical Solution show a

“Moderately Strong” Positive Relationshipwith Performance


Results4. IPT-Related Capability and Performance

Projects with better IPTs show a

“Moderately Strong” Positive Relationshipwith Performance


Results5. Requirements and Performance

Projects with better Requirements Development and Management show a

“Moderately Strong” Positive Relationship with Performance


ResultsSummary of Process Relationships

Details

Moderately Strong

to Strong Relationship

Moderately Strong

Relationship

Strong Relationship

Weak Relationship


ResultsSummary of Relationships - Composite

Details

Composite Measures

Moderately Strong

to Strong Relationship

Moderately Strong

Relationship

Strong Relationship

Weak Relationship


Results - Reqts + Tech Solution controlled by Project Challenge

Project challenge factors:

  • Life cycle phases

  • Project characteristics (e.g., size, effort, duration, volatility)

  • Technical complexity

  • Teaming relationships

Projects with higher Requirements and Technical Solution capability are better able to achieve higher performance even in challenging programs


Summary

SE Effectiveness

  • Provides credible measured evidence about the value of disciplined Systems Engineering

  • Affects success of systems-development projects

    Specific Systems Engineering Best Practices

  • Highest relationships to activities on the “left side of SE Vee”

  • The environment (Project Challenge) affects performance too:

    • Some projects are more challenging than others ... and higher challenge affects performance negatively in spite of better SE

    • Yet good SE practices remain crucial for both high and low challenge projects


Potential Next Steps

  • Provide recommendations for action upon survey findings

  • Conduct additional follow-on surveys and analysis of collected data

    • IV&V

    • Broadened sample space

    • Trending

    • Improvements to survey instrument

  • Survey system acquirers


DoD Systemic Root Cause Analysis- Why do projects fail?

  • Root causes from DoD analysis of program performance issues appear consistent with NDIA SE survey findings.

  • Reference:

  • Systemic Root Cause Analysis,

  • Dave Castellano, Deputy Director Assessments & Support, OUSD(A&T)

  • NDIA Systems Engineering Conference, 2007

  • and NDIA SE Division Annual Planning Meeting


Acknowledgements


SE EffectivenessPoints of Contact

Al Brown alan.r.brown2@boeing.com

Geoff Draper gdraper@harris.com

Joe Elm jelm@sei.cmu.edu

Dennis Goldenson dg@sei.cmu.edu

Al MinkAl_Mink@SRA.com

Ken Ptack ken.ptack@ngc.com

Mike Ucchinomichael.ucchino@afit.edu


Backup

NDIA SE Effectiveness Survey

Analysis Slides


Conclusions & CaveatsConsistent with “Top 10 Reasons Projects Fail*”

  • Lack of user involvement

  • Changing requirements

  • Inadequate Specifications

  • Unrealistic project estimates

  • Poor project management

  • Management change control

  • Inexperienced personnel

  • Expectations not properly set

  • Subcontractor failure

  • Poor architectural design

Above Items Can Cause Overall Program Cost and Schedule to Overrun

* Project Management Institute Matching items noted in RED


Conclusions & CaveatsConsistent with “Top 5 SE Issues*” (2006)

  • Keysystems engineering practicesknown to be effective arenot consistently appliedacross all phases of the program life cycle.

  • Insufficient systems engineering is applied earlyin the program life cycle, compromising the foundation for initial requirements and architecture development.

  • Requirements are not always well-managed, including the effective translationfrom capabilities statementsinto executable requirements to achieve successful acquisition programs.

  • The quantity and quality ofsystems engineering expertise is insufficientto meet the demands of the government and the defense industry.

  • Collaborative environments, includingSE tools, are inadequateto effectively execute SE at the joint capability, system of systems, and system levels.

* OUSD AT&L Summit Matching items noted in RED


Summary SE Relationshipsto Project Performance

Details


Summary SE Relationshipsto Project Performance

Details

Highest scoring SE capability areas in Higher Performing Projects*:

Risk Management; Requirements Development and Management; IPTs

*Based on small partitioned sample size

Lowest scoring SE capability areas in Lower Performing Projects*:

Validation; Architecture; Requirements Development and Management


Terminology and NotationDistribution Graph

Histogram of

response

frequencies

Median

Interquartile

Range

Outliers

Sample size

(responses to corresponding survey questions)

Data

Range


Terminology and NotationMosaic Chart

Column width represents proportion of projects with this

level of capability

Relative performance distribution of the sample

Gamma: measures strength of relationship between two ordinal variables

p: probability that an associative relationship would be observed by chance alone

Projects exhibiting a given level of relative capability

(Lowest, Intermediate, Highest)

Sample size and distribution for associated survey responses

(capability + performance)

Measures of association and statistical test


SE Capability: Product Architecture (ARCH)


SE Capability: Product Architecture (ARCH)

Survey Questions


SE Capability: Configuration Management (CM)


SE Capability: Configuration Management (CM)

Survey Questions


SE Capability: IPT-Related Capability (IPT)


SE Capability: IPT-Related Capability (IPT)

Survey Questions


SE Capability: Product Integration (PI)


SE Capability: Product Integration (PI)

Survey Question


SE Capability: Project Monitoring and Control (PMC)


SE Capability: Project Monitoring and Control (PMC)

Survey Questions (Part 1)


SE Capability: Project Monitoring and Control (PMC)

Survey Questions (Part 2)


SE Capability: Project Planning (PP)


SE Capability: Project Planning (PP)

Survey Questions (Part 1)


SE Capability: Project Planning (PP)

Survey Questions (Part 2)


SE Capability: Project Planning (PP)

Survey Questions (Part 3)


SE Capability: Requirements Development & Mgmt (REQ)


SE Capability: Requirements Development & Mgmt (REQ)

Survey Questions (Part 1)


SE Capability: Requirements Development & Mgmt (REQ)

Survey Questions (Part 2)


SE Capability: Risk Management (RSKM)


SE Capability: Risk Management (RSKM)

Survey Questions


SE Capability: Trade Studies (TRADE)


SE Capability: Trade Studies (TRADE)

Survey Questions


SE Capability: Technical Solution (TS)

Note: TS is a composite measure equivalent to ARCH + TRADE.


SE Capability: Technical Solution (TS)

Survey Questions (Part 1)


SE Capability: Technical Solution (TS)

Survey Questions (Part 2)


SE Capability: Validation (VAL)


SE Capability: Validation (VAL)

Survey Questions


SE Capability: Verification (VER)


SE Capability: Verification (VER)

Survey Questions (Part 1)


SE Capability: Verification (VER)

Survey Questions (Part 2)


SE Capability: Combined Reqts+Tech Solution (REQ+TS)

(This is a higher order measure; see base measures for distribution)


SE Capability: Total Systems Engineering Capability


Project challenge factors:

Life cycle phases

Project characteristics (e.g., size, effort, duration, volatility)

Technical complexity

Teaming relationships

Project Challenge (PC)


SE Capability:Reqts+Tech Solution with Project Challenge

Project challenge factors:

  • Life cycle phases

  • Project characteristics (e.g., size, effort, duration, volatility)

  • Technical complexity

  • Teaming relationships


Relating Project Performance to Project Challenge and SE Capability


Reqts + Tech Solution + Project Challenge and Performance

Project challenge factors:

  • Life cycle phases

  • Project characteristics (e.g., size, effort, duration, volatility)

  • Technical complexity

  • Teaming relationships


SEEC Activities

Identify industry members’ focals

Contact focals, brief the survey process, solicit support

Provide Web access data to focals

Focal contact #1 to expedite response

Focal contact #2 to expedite response

Report* findings to NDIA and OSD

NDIA SED active roster

NDIA mg’t input

Company Focal

Activities

Solicit respondents and provide Web site access info

Identify respondents and report number to SEI

Respondent contact #1 to expedite response

Respondent contact #2 to expedite response

Report number of responses provided to SEI

Respondent Activities

Complete questionnaire and submit to SEI

Report completion to focal

SEI Activities

Collect responses and response rate data

Analyze data and report to SEEC

SE EffectivenessMethodology (In Detail)


Summary of Relationships


ad
  • Login