Toward an ontology for measuring systems engineering return on investment l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 29

Toward an Ontology for Measuring Systems Engineering Return on Investment PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Toward an Ontology for Measuring Systems Engineering Return on Investment. Advancing the knowledge of systems engineering. Eric Honour +1 (850) 479-1985 [email protected] Dr. Ricardo Valerdi +1 (617) 253-8583 [email protected] Version. Topics. SE-ROI Project “Ontology” concept

Download Presentation

Toward an Ontology for Measuring Systems Engineering Return on Investment

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Toward an Ontology for Measuring Systems Engineering Return on Investment

Advancing the knowledge of systems engineering

Eric Honour

+1 (850) 479-1985

[email protected]

Dr. Ricardo Valerdi

+1 (617) 253-8583

[email protected]

Toward Ontology for SE-ROI



  • SE-ROI Project

  • “Ontology” concept

  • COSYSMO work toward ontology

  • Categorization from current standards

  • Future directions

Toward Ontology for SE-ROI

Systems Engineering Return on Investment

Summary of the SE-ROI Project

Toward Ontology for SE-ROI

Traditional Design














“System Thinking” Design


Heuristic Claim of SE

  • Better systems engineering leads to

    • Better system quality/value

    • Lower cost

    • Shorter schedule

Toward Ontology for SE-ROI

Greater SE Effort led to better cost/schedule compliance and better predictability

Value of SE – 2004 Results


  • Quantitative data on SE not available in program databases

    • All data points were subjective

  • Detailed structure not available

Value = 1.0 if program met cost/schedule goals

Each dot is one program, with sizes between $1M and $6.5B

Source: SECOE 01-03INCOSE 2003

Toward Ontology for SE-ROI

SE-ROI Project

  • Interviews

  • Just-completed programs

  • Key PM/SE/Admin

  • Translate program data into project structure

  • Desired Results

  • Statistical correlation of SE methods with program success.

  • Leading SE indicators that can be used during a program.

  • Identification of good SE practices under different conditions.

  • Program characterization

  • Program success data

  • SE data (hours, quality, methods)

Statistical correlation

Toward Ontology for SE-ROI

“Ontology” Concept

What is this word and how does it relate to systems engineering?

Toward Ontology for SE-ROI

Current State of SE Definition

  • Fragmented by domain opinions

  • Military – DOD/MOD

  • Space - NASA/ESA

  • Commercial products

  • Aircraft

  • Automobiles

  • Nuclear waste

  • Process engineering

  • Tool vendors

  • Etc. Etc. Etc.

  • Fragmented by discipline opinions

  • Technical leaders

  • System architects

  • System analysts

  • Requirements engineers

  • Operations analysts

  • Design engineers

  • Fragmented by standards

  • ANSI/EIA-632

  • IEEE-1220

  • ISO-15288

  • CMMI

  • MIL-STD-499C

Toward Ontology for SE-ROI


“…a branch of metaphysics concerned with the nature and relations of being”











  • POSIWID – The Purpose of a Systems Is What It Does

  • Jack Ring

  • The purpose of systems engineering is different in the eyes of different people, because they perceive different actions/results from SE

Toward Ontology for SE-ROI

Purpose of this Paper

  • Explore the variety of what people see in SE

  • Formulate some general categories

  • Interpret historical SE effort data

  • Provide a structure for the data-gathering in the SE-ROI project.

Toward Ontology for SE-ROI

COSYSMO work toward ontology

An exploration of ontology as performed in the COSYSMO project

Toward Ontology for SE-ROI

COSYSMO Effort Profile

  • How is Systems Engineering effort distributed over time?

Toward Ontology for SE-ROI

Effort Distribution Across

ANSI/EIA 632 Fundamental Processes

Toward Ontology for SE-ROI

Systems Engineering Effort Profile

Toward Ontology for SE-ROI

Categorization from Current Standards

The start of an ontology, by identifying the widely-accepted categories.

Toward Ontology for SE-ROI

Categories in the Standards

  • Mission/Purpose Definition

  • Requirements Management

  • System Architecting

  • System Implementation

  • Technical Analysis

  • Technical Management/Leadership

  • Verification & Validation






Colored boxes on following slides show the terminology used by each standard

Toward Ontology for SE-ROI

Mission/Purpose Definition

  • Define the mission or purpose of the new/changed system.

    • Typically described in the language of the system users rather than in technical language

  • CMMI

  • Develop customer requirements

  • ANSI/EIA-632

  • Not included

  • MIL-STD-499C

  • Not included

  • IEEE-1220

  • Define customer expectations

  • ISO-15288

  • Stakeholder needs definition

Toward Ontology for SE-ROI

Requirements Management

  • Creation and management of requirements

    • Efforts to define, analyze, validate, and manage the requirements

  • CMMI

  • Requirements development

  • Requirements mgmt

  • ANSI/EIA-632

  • System design

  • Requirements definition

  • MIL-STD-499C

  • System requirements analysis and validation

  • IEEE-1220

  • Requirements analysis

  • ISO-15288

  • Requirements analysis

Toward Ontology for SE-ROI

System Architecting

  • Define the system in terms of its component elements and their relationships

    • Diagrams that depict the system, its environment, components, and relationships

  • CMMI

  • Technical solution

  • ANSI/EIA-632

  • System design

  • Solution definition

  • MIL-STD-499C

  • System product technical requirements analysis and validation

  • Design or physical solution representation

  • IEEE-1220

  • Synthesis

  • ISO-15288

  • Architectural design

  • System life cycle mgmt

Toward Ontology for SE-ROI

System Implementation

  • Development/completion of the system

    • Specific system-level efforts in the standards are system integration and transition to use

  • ANSI/EIA-632

  • Product realization

  • Implementation

  • Transition to use

  • CMMI

  • Product integration

  • MIL-STD-499C

  • Not included

  • IEEE-1220

  • Not included

  • ISO-15288

  • Implementation

  • Integration

  • Transition

Toward Ontology for SE-ROI

Technical Analysis

  • System-level technical analysis

    • Assessment of system performance

    • Trade-off analysis of alternatives

  • CMMI

  • Measurement and analysis

  • ANSI/EIA-632

  • Technical evaluation

  • System analysis

  • MIL-STD-499C

  • Functional analysis, allocations and validation

  • Assessments of system effectiveness, cost, schedule, and risk

  • Tradeoff analyses

  • IEEE-1220

  • Functional analysis

  • Requirements trade studies and assessments

  • Functional trade studies and assessments

  • Design trade studies and assessments

  • ISO-15288

  • Requirements analysis

Toward Ontology for SE-ROI

Technical Management/Leadership

  • Guiding the engineering teams involved in system design programs

    • Size/complexity of teams demands leadership

  • CMMI

  • Project planning

  • Project monitoring & control

  • Supplier agreement mgmt

  • Process/product quality assur.

  • Configuration mgmt

  • Integrated project mgmt

  • Decision analysis/resolution

  • Quantitative project mgmt

  • Risk mgmt

  • ANSI/EIA-632

  • Technical Mgmt

  • Planning

  • Assessment

  • Control

  • MIL-STD-499C

  • Planning

  • Monitoring

  • Decision making, control, and baseline maintenance

  • Risk mgmt

  • Baseline change control

  • Interface mgmt

  • Data mgmt

  • Subcontract mgmt

  • Technical reviews/audits

  • IEEE-1220

  • Technical mgmt

  • Track analysis data

  • Track requirements and design changes

  • Track performance

  • Track product metrics

  • Update specifications

  • Update architectures

  • Update plans

  • Maintain database

  • ISO-15288

  • Planning, Assessment, Control

  • Decision mgmt

  • Config mgmt

  • Acquisition, Supply

  • Resource mgmt

  • Risk mgmt

Toward Ontology for SE-ROI

Verification & Validation

  • Verification: comparison of the system with its requirements through objective evidence.

  • Validation: comparison of the system or requirements with the intended mission

  • ANSI/EIA-632

  • Technical Evaluation

  • Requirements validation

  • System verification

  • End products validation

  • CMMI

  • Verification

  • Validation

  • MIL-STD-499C

  • Design or physical solution verification and validation

  • IEEE-1220

  • Requirement verification

  • Functional verification

  • Design verification

  • ISO-15288

  • Verification

  • Validation

  • Quality mgmt

Toward Ontology for SE-ROI

Future Directions

Where is SE-ROI going?

Toward Ontology for SE-ROI

Project Advisory Group

  • Group of interested people/organizations

    • Communicate via web, telecon, meetings

    • Help define the data organization

    • Build public interest in the project

    • Provide access to real programs

    • View interim (protected) data as it develops

Current members come from:

AF Institute of Technology

Northrop Grumman

DOD Office of Secy of Defense


Johns Hopkins Univ


The Mitre Corp



Rand Corporation


Systems & Software Consortium

Univ of South Australia

USN Chief Engineer

For information, see

Toward Ontology for SE-ROI


  • Systems engineering current state of knowledge is fragmented

  • Broadly-accepted ontology is needed

  • SE-ROI project needs categorization now

    • Structure the data to be correlated

    • Discover leading SE indicators

    • Identify SE best practices and methods

  • Categorization across standards helps develop the needed ontology

Toward Ontology for SE-ROI


Eric Honour

+1 (850) 479-1985

[email protected]

Dr. Ricardo Valerdi

+1 (617) 253-8583

[email protected]

For information, see

Toward Ontology for SE-ROI

Survey of SE-ROI Knowledge

Toward Ontology for SE-ROI

Next Steps

  • Define interview data sheets

    • Use this categorization

  • Identify and interview trial projects

    • Obtain initial data

    • Evaluate the interview data sheets

  • Identify and interview projects

    • Ongoing effort for 2-3 years

  • Perform statistical correlation work

    • Ongoing effort for duration of project

    • Interim reports to participating organizations

  • Final report expected 2009

Toward Ontology for SE-ROI

  • Login