slide1
Download
Skip this Video
Download Presentation
FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL (CSIP) for the NASA Independent Verification and Validation Facility COTR: Kenneth McGill PI: Nancy Eickelmann

Loading in 2 Seconds...

play fullscreen
1 / 30

FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL CSIP for the NASA Independent Verification and Validation Facility COTR: Ken - PowerPoint PPT Presentation


  • 115 Views
  • Uploaded on

Developing Risk-Based Financial Analysis Tools and Techniques to Aid IV&V Decision-Making. FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL (CSIP) for the NASA Independent Verification and Validation Facility COTR: Kenneth McGill PI: Nancy Eickelmann Contract #S-54493-G September 4, 2002. Agenda.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL CSIP for the NASA Independent Verification and Validation Facility COTR: Ken' - lynnea


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Developing Risk-Based Financial Analysis Tools and Techniques to Aid IV&V Decision-Making

FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL (CSIP)

for the

NASA Independent Verification and Validation Facility

COTR: Kenneth McGill

PI: Nancy Eickelmann

Contract #S-54493-G

September 4, 2002

agenda
Agenda
  • Why we need ASK IVEY
  • Consequences and Likelihood of Failure
  • IV&V Yield
  • Probability of IV&V Yield: Min, Max, Most Likely
  • ROI and Magnitude of Return of IV&V
  • What ASK IVEY can do
why we need ask ivey
Why we need ASK IVEY
  • NASA program managers are asked to quantify the ROI and evaluate the cost/benefit of applying IV&V technologies.
  • This is a prediction of future events based on decisions and actions taken in the present.
  • A point estimate is likely to be inaccurate, whereas a probability of yield has a history of providing a scope of potential yield and an extent of likelihood of expected yield.
calculating roi a financial analysis prompt map

Financial Analysis Process Map

Yes

No

STOP

?

A

1

2

3

IV&V and IA analysis complete.

Create certification

Create development

Apply Financial models for

Level of IV&V or IA designated

cost framework

cost framework:

monetary quanitification

Total Cost

COQ

COPQ

STOP

A

4

Create probability

of yield structure

Calculating ROI a Financial Analysis Prompt Map
consequences of failure
Consequences of Failure

NPG 2820 IV&V Criteria

iv v yield
IV&V YIELD
  • Ultimately, the yield of an IV&V program is based upon the difference between the net resource flow with IV&V and without IV&V.
  • If the resources saved (e.g., reduced rework) or returns gained (e.g., improved customer satisfaction or increased safety) are greater than the resources consumed to save/gain these resources, we have a net benefit.
  • Should the resources saved be less than the resources consumed, we have a net cost.
cost of poor quality
Cost of Poor Quality
  • Defect Leakage
    • If discovered internally
      • defect management
      • rework
      • retesting
    • If discovered externally
      • technical support
      • complaint investigation
      • defect notification
stephen knox modeling the cost of software quality digital technical journal fall 1993
Stephen Knox“Modeling the Cost of Software Quality,”Digital Technical Journal, (Fall 1993)
how process maturity levels affect iv v
How Process Maturity Levels Affect IV&V

NON-TECH UP TO 3-4%

IV&V

MATURITY

NON-TECH UP TO 6-8%

LEVEL 5

OPTIMIZED

FOCUS ON PROCESS

IMPROVEMENT

IV&V

NON-TECH UP TO 15%

IV&V

LEVEL 4

MANAGED

PROCESS MEASURED

AND CONTROLLED

UNABLE TO ESTIMATE

NON-TECH %

NON-TECH UP TO 25%

LEVEL 3

DEFINED

PROCESS

CHARACTERIZED,

FAIRLY WELL

UNDERSTOOD

  • KEY PROBLEMS:
  • STILL HUMAN INTENSIVE PROCESS
  • DIFFICULT TO MAINTAIN OPTIMUM ORGANIZATION
  • DIFFICULT TO MAINTAIN TOOLS & PRACTICES AT STATE OF THE ART

IV&V

IV&V

SOMEWHAT

UNPREDICTABLE

  • KEY PROBLEMS:
  • CHANGING TECHNOLOGY
  • PROBLEM ANALYSIS
  • PROBLEM PREVENTION
  • ORGANIZATIONAL OPTIMIZATION

LEVEL 2

REPEATABLE

CAN REPEAT

PREVIOUSLY

MASTERED TASKS

LEVEL 1

INITIAL

UNPREDICTABLE &

POORLY CONTROLLED

  • KEY PROBLEMS:
  • ACCURATE PROJECT MEASUREMENT
  • OBJECTIVE PROCESS ANALYSIS
  • QUANTITATIVE QUALITY PLANS
  • PRIORITIES:
  • - DECIDING WHAT TO IMPROVE FIRST
  • - BALANCING THE PROCESS AND THE ORGANIZATION
  • KEY PROBLEMS:
  • PROPER TRAINING
  • DEVELOPMENT OF PRACTICE & PROCEDURES
  • STANDARDS
  • ORGANIZATION
  • KEY PROBLEMS:
  • CONFIG. MGMT.
  • PROJECT MGMT.
  • SOFTWARE QA
  • PROJECT PLANNING
  • EASTIMATING
  • COST
  • SCHEDULE
cost of leakage grows over time
Cost of Leakage GrowsOver Time
  • Relative cost of fixing a problem found in design/coding, testing, or after release are:
    • 1:20:82 (Remus, 1983)
    • 1:13:92 (Kan, 1989)
    • 10:100:1000 (Coyle, 1999)
cost of rework in each phase
Cost of Rework in Each Phase
  • Reworkproduct design =
    • leakagerequirements * cost-to-fixnominal * 10
  • Reworkprogramming =
    • leakagerequirements * cost-to-fixnominal * 100
    • leakagedesign * cost-to-fixnominal * 10
  • Reworkintegration =
    • leakagerequirements * cost-to-fixnominal * 1000
    • leakagedesign * cost-to-fixnominal * 100
    • leakageprogramming * cost-to-fixnominal * 10
  • Reworkdeployment =
    • leakagerequirements * cost-to-fixnominal * 10000
    • leakagedesign * cost-to-fixnominal * 1000
    • leakageprogramming * cost-to-fixnominal * 100
rework at deployment
Rework at Deployment
  • Tremendous cost
  • rework plus
    • product recall
    • technical support
    • field visits
    • cost factor may be over 10,000
leakage an example
Leakage: An Example
  • Reworkproduct design =
    • 49r * cost-to-fixnominal * 10 = 490
  • Reworkprogramming =
    • 39r * cost-to-fixnominal * 100 = 3,900
    • 113d * cost-to-fixnominal * 10 = 1,130
  • Reworkintegration =
    • 26r * cost-to-fixnominal * 1000 = 26,000
    • 49d * cost-to-fixnominal * 100 = 4,900
    • 418p * cost-to-fixnominal * 10 = 4,180
  • Reworkdeployment =
    • 8r * cost-to-fixnominal * 10000 = 80,000
    • 16d * cost-to-fixnominal * 1000 = 16,000
    • 56p * cost-to-fixnominal * 100 = 5,600 142,200
cmm maturity and leakage
CMM Maturity and Leakage
  • There is some evidence to suggest organizations with increased maturity have reduced rework costs
  • Knox: Percent of Budget to Rework:
    • Level 1: 55%
    • Level 2: 45%
    • Level 3: 35%
    • Level 4: 20%
    • Level 5: 6%
iv v and defect leakage
IV&V and Defect Leakage
  • Application of IV&V can reduce leakage to subsequent phases
  • The goal of the financial model is to propose a range of potential savings
  • Specific parameters will need to be established empirically
timing of benefits for iv v
Timing of benefits for IV&V
  • Full In-Phase IV&V
    • prevention of errors starting at requirements - can potentially bar any errors from leaking through
  • Partial IV&V
    • prevention of errors at point of insertion - no errors from this phase will leak
  • Endgame IV&V
    • discovery of errors at the end of development - can potentially bar any errors from leaking to deployment
  • Audit Level IV&V
components to return on investment
Components to Return on Investment
  • Cost of IV&V
  • Expected Return
    • cost savings - measured as hours of rework
  • Likelihood of Returns
    • how effective is the organization at minimizing rework?
    • how effective will IV&V be?
independence
Independence…
  • An organization independent from the developers study the artifacts of software production [IEEE Std. 1012-1998].
  • This requires:
    • Technical independence. Members of the IV&V team may not be personnel involved in the development of the software.
    • .Managerial independence. The responsibility for IV&V belongs to an organization outside the contractor and program organizations that develop the software.
    • Financial independence. Control of the IV&V budget is retained in an organization outside the contractor and program organization that develop the software.
  • IV&V is often perceived as testing the code after the development is completed …..NASA IV&V is full life cycle activities
iv v is not sqa
IV&V is NOT SQA
  • IV&V is a full life cycle set of acivities that are applied to defect prevention, defect detection, and certification. NASA IV&V conforms to IEEE Standard 1012-1998.
  • IV&V and Software Quality Assurance (SQA) are not redundant activities. SQA as defined by DOD-Std 2168 defines 10 activities of SQA that are complemented by IV&V activities.There are 32 types of activities conducted by IV&V, of these 32, 22 are unique to IV&V and 10 are complemented by SQA.
ask ivey prototype
Ask Ivey Prototype

What ASK IVEY can do…

questions
Questions?

Ask Ivey…

ad