Developing Risk-Based Financial Analysis Tools and Techniques to Aid IV&V Decision-Making
Sponsored Links
This presentation is the property of its rightful owner.
1 / 30

FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL (CSIP) for the NASA Independent Verification and Validation Facility COTR: Kenneth McGill PI: Nancy Eickelmann PowerPoint PPT Presentation


  • 90 Views
  • Uploaded on
  • Presentation posted in: General

Developing Risk-Based Financial Analysis Tools and Techniques to Aid IV&V Decision-Making. FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL (CSIP) for the NASA Independent Verification and Validation Facility COTR: Kenneth McGill PI: Nancy Eickelmann Contract #S-54493-G September 4, 2002. Agenda.

Download Presentation

FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL (CSIP) for the NASA Independent Verification and Validation Facility COTR: Kenneth McGill PI: Nancy Eickelmann

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Developing Risk-Based Financial Analysis Tools and Techniques to Aid IV&V Decision-Making

FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL (CSIP)

for the

NASA Independent Verification and Validation Facility

COTR: Kenneth McGill

PI: Nancy Eickelmann

Contract #S-54493-G

September 4, 2002


Agenda

  • Why we need ASK IVEY

  • Consequences and Likelihood of Failure

  • IV&V Yield

  • Probability of IV&V Yield: Min, Max, Most Likely

  • ROI and Magnitude of Return of IV&V

  • What ASK IVEY can do


Why we need ASK IVEY

  • NASA program managers are asked to quantify the ROI and evaluate the cost/benefit of applying IV&V technologies.

  • This is a prediction of future events based on decisions and actions taken in the present.

  • A point estimate is likely to be inaccurate, whereas a probability of yield has a history of providing a scope of potential yield and an extent of likelihood of expected yield.


Financial Analysis Process Map

Yes

No

STOP

?

A

1

2

3

IV&V and IA analysis complete.

Create certification

Create development

Apply Financial models for

Level of IV&V or IA designated

cost framework

cost framework:

monetary quanitification

Total Cost

COQ

COPQ

STOP

A

4

Create probability

of yield structure

Calculating ROI a Financial Analysis Prompt Map


Consequences of Failure

NPG 2820 IV&V Criteria


Likelihood of Failure


IV&V YIELD

  • Ultimately, the yield of an IV&V program is based upon the difference between the net resource flow with IV&V and without IV&V.

  • If the resources saved (e.g., reduced rework) or returns gained (e.g., improved customer satisfaction or increased safety) are greater than the resources consumed to save/gain these resources, we have a net benefit.

  • Should the resources saved be less than the resources consumed, we have a net cost.


Cost of Poor Quality

  • Defect Leakage

    • If discovered internally

      • defect management

      • rework

      • retesting

    • If discovered externally

      • technical support

      • complaint investigation

      • defect notification


Stephen Knox“Modeling the Cost of Software Quality,”Digital Technical Journal, (Fall 1993)


Raytheon Cost of Poor QualityHaley and Dion


How Process Maturity Levels Affect IV&V

NON-TECH UP TO 3-4%

IV&V

MATURITY

NON-TECH UP TO 6-8%

LEVEL 5

OPTIMIZED

FOCUS ON PROCESS

IMPROVEMENT

IV&V

NON-TECH UP TO 15%

IV&V

LEVEL 4

MANAGED

PROCESS MEASURED

AND CONTROLLED

UNABLE TO ESTIMATE

NON-TECH %

NON-TECH UP TO 25%

LEVEL 3

DEFINED

PROCESS

CHARACTERIZED,

FAIRLY WELL

UNDERSTOOD

  • KEY PROBLEMS:

  • STILL HUMAN INTENSIVE PROCESS

  • DIFFICULT TO MAINTAIN OPTIMUM ORGANIZATION

  • DIFFICULT TO MAINTAIN TOOLS & PRACTICES AT STATE OF THE ART

IV&V

IV&V

SOMEWHAT

UNPREDICTABLE

  • KEY PROBLEMS:

  • CHANGING TECHNOLOGY

  • PROBLEM ANALYSIS

  • PROBLEM PREVENTION

  • ORGANIZATIONAL OPTIMIZATION

LEVEL 2

REPEATABLE

CAN REPEAT

PREVIOUSLY

MASTERED TASKS

LEVEL 1

INITIAL

UNPREDICTABLE &

POORLY CONTROLLED

  • KEY PROBLEMS:

  • ACCURATE PROJECT MEASUREMENT

  • OBJECTIVE PROCESS ANALYSIS

  • QUANTITATIVE QUALITY PLANS

  • PRIORITIES:

  • - DECIDING WHAT TO IMPROVE FIRST

  • - BALANCING THE PROCESS AND THE ORGANIZATION

  • KEY PROBLEMS:

  • PROPER TRAINING

  • DEVELOPMENT OF PRACTICE & PROCEDURES

  • STANDARDS

  • ORGANIZATION

  • KEY PROBLEMS:

  • CONFIG. MGMT.

  • PROJECT MGMT.

  • SOFTWARE QA

  • PROJECT PLANNING

  • EASTIMATING

  • COST

  • SCHEDULE


Cost of Leakage GrowsOver Time

  • Relative cost of fixing a problem found in design/coding, testing, or after release are:

    • 1:20:82 (Remus, 1983)

    • 1:13:92 (Kan, 1989)

    • 10:100:1000 (Coyle, 1999)


Cost of Rework in Each Phase

  • Reworkproduct design =

    • leakagerequirements * cost-to-fixnominal * 10

  • Reworkprogramming =

    • leakagerequirements * cost-to-fixnominal * 100

    • leakagedesign * cost-to-fixnominal * 10

  • Reworkintegration =

    • leakagerequirements * cost-to-fixnominal * 1000

    • leakagedesign * cost-to-fixnominal * 100

    • leakageprogramming * cost-to-fixnominal * 10

  • Reworkdeployment =

    • leakagerequirements * cost-to-fixnominal * 10000

    • leakagedesign * cost-to-fixnominal * 1000

    • leakageprogramming * cost-to-fixnominal * 100


Rework at Deployment

  • Tremendous cost

  • rework plus

    • product recall

    • technical support

    • field visits

    • cost factor may be over 10,000


Leakage: An Example

  • Reworkproduct design =

    • 49r * cost-to-fixnominal * 10 = 490

  • Reworkprogramming =

    • 39r * cost-to-fixnominal * 100 = 3,900

    • 113d * cost-to-fixnominal * 10 = 1,130

  • Reworkintegration =

    • 26r * cost-to-fixnominal * 1000 = 26,000

    • 49d * cost-to-fixnominal * 100 = 4,900

    • 418p * cost-to-fixnominal * 10 = 4,180

  • Reworkdeployment =

    • 8r * cost-to-fixnominal * 10000 = 80,000

    • 16d * cost-to-fixnominal * 1000 = 16,000

    • 56p * cost-to-fixnominal * 100 = 5,600142,200


CMM Maturity and Leakage

  • There is some evidence to suggest organizations with increased maturity have reduced rework costs

  • Knox: Percent of Budget to Rework:

    • Level 1: 55%

    • Level 2: 45%

    • Level 3: 35%

    • Level 4: 20%

    • Level 5: 6%


IV&V and Defect Leakage

  • Application of IV&V can reduce leakage to subsequent phases

  • The goal of the financial model is to propose a range of potential savings

  • Specific parameters will need to be established empirically


Timing of benefits for IV&V

  • Full In-Phase IV&V

    • prevention of errors starting at requirements - can potentially bar any errors from leaking through

  • Partial IV&V

    • prevention of errors at point of insertion - no errors from this phase will leak

  • Endgame IV&V

    • discovery of errors at the end of development - can potentially bar any errors from leaking to deployment

  • Audit Level IV&V


Rework and Return from IV&V By Maturity Level


Components to Return on Investment

  • Cost of IV&V

  • Expected Return

    • cost savings - measured as hours of rework

  • Likelihood of Returns

    • how effective is the organization at minimizing rework?

    • how effective will IV&V be?


Independence…

  • An organization independent from the developers study the artifacts of software production [IEEE Std. 1012-1998].

  • This requires:

    • Technical independence. Members of the IV&V team may not be personnel involved in the development of the software.

    • .Managerial independence. The responsibility for IV&V belongs to an organization outside the contractor and program organizations that develop the software.

    • Financial independence. Control of the IV&V budget is retained in an organization outside the contractor and program organization that develop the software.

  • IV&V is often perceived as testing the code after the development is completed …..NASA IV&V is full life cycle activities


IV&V is NOT SQA

  • IV&V is a full life cycle set of acivities that are applied to defect prevention, defect detection, and certification. NASA IV&V conforms to IEEE Standard 1012-1998.

  • IV&V and Software Quality Assurance (SQA) are not redundant activities. SQA as defined by DOD-Std 2168 defines 10 activities of SQA that are complemented by IV&V activities.There are 32 types of activities conducted by IV&V, of these 32, 22 are unique to IV&V and 10 are complemented by SQA.


Ask Ivey Prototype

What ASK IVEY can do…


Ask Ivey Input Screen


Ask Ivey Pull Down Menu


Ask Ivey Numeric Entry


Ask Ivey On-Line Report


Ask Ivey Printed Report


Ask Ivey On-Line Help


Questions?

Ask Ivey…


  • Login