Risk-Based Financial Analysis Tools for IV&V Decision-Making

1. Nancy S. Eickelmann, PhDMotorola Labs1303 E. Algonquin Rd.Annex-2Schaumburg, IL 60196Phone: (847) 310-0785Fax: (847) 576-3280Nancy.Eickelmann@motorola.com

2. Developing Risk-Based Financial Analysis Tools and Techniques to Aid IV&V Decision-Making FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL (CSIP) for the NASA Independent Verification and Validation Facility COTR: Ken McGill PI: Nancy Eickelmann S-54493-G September 5, 2001

3. PROBLEM STATEMENT • This research addresses NASA’s need to evaluate the ROI and cost/benefit of applying IV&V technologies. • A prototype is to be developed that will provide financial valuation of IV&V for a given program. • The prototype will be developed using an iterative process that will incrementally implement the models and methodology researched and developed during prior years of this effort. • The tool will be evaluated for usability, accuracy, and consistency through limited use scenarios with NASA program managers.

4. Return on Investment - Status • This project was funded July 20, 2001 • Evaluation of data sets is in progress • Benchmarking for key factor target value ranges in progress • Model integration and interface to existing programs in progress, Ask Pete, ARRT

5. RESEARCH APPROACH Phase 1: • Reduce the models we developed earlier to actionable guidelines for practice Phase 2: • Introduce these models, processes and support tools to a small group of carefully selected pilot projects • Evaluate the results of applying the tools and methods Phase 3: • Use the feedback from step 3 to adapt the tools and methods for widespread dissemination, if warranted within the software project decision-making community at NASA.

6. HYPOTHESES/OBJECTIVE • The IV&V valuation methodology will be iteratively refined based on feedback from NASA program managers and statistical evaluation of the methodology and results. • Specific factors to be evaluated: Hypothesis 1: The cost relative to the potential benefits of IV&V is inversely proportional to key organizational factors, such as the capability maturity of the development organization.   Hypothesis 2: The realization of potential IV&V benefits is directly related to the development organizations’ acceptance of IV&V.   Hypothesis 3: The cost/benefit ratio for IV&V is directly related to the criticality of the application (and its individual subsystems).

7. IV&V YIELD • Ultimately, the yield of an IV&V program is based upon the difference between the net resource flow with IV&V and without IV&V. • If the resources saved (e.g., reduced rework) or returns gained (e.g., improved customer satisfaction or increased safety) are greater than the resources consumed to save/gain these resources, we have a net benefit. • Should the resources saved be less than the resources consumed, we have a net cost.

8. IV&V Yield • Cost of Quality • Key components… • Cost of Poor Quality • Key components…

9. What we already know…3 issues of empirical studies... June 5-6, 1986 the 1st Workshop on Empirical Studies of Programmers, Washington, D.C. • Need scientific rigor…“A Plan for Empirical Studies” Victor Basili • Need to look at real world variable values…“By the Way, Did Anyone Study Real Programmers” Bill Curtis • Need to study PITL…“Meeting the Challenge of Programming in the Large (PITL)” Elliot Soloway

10. Why is it Difficult to Apply Quantitative Management Principles for Software Engineering? • SE domain has a large number of key variables that have different degrees of significance depending on the environment • SE domain has key variables that have extreme variance within the same environment (i.e., programmer productivity 10:1) • SE domain variables in combination may create a “criticalmass” not present when variables are studied in isolation 1986 IEEE TSE, Basili, Selby and Hutchins, Surveyed software engineering empirical studies published to date. Cited 116 published studies.

11. Software IV&V SOW -Objectives - Requirements Program Milestones and Schedules Requirements Repositories Developer Documentation Software Test Plans & Procedures Software Development Folders Source Code Problem Reports Iterative IV&V Methodology Inputs Outputs Activities Software IV&V Plan IV&V Planning - Activities - Organization - CARA - Schedules - Tools - WBS Critical/High Risk Functions List IV&V Technical Reports Software Problem Reports Software Requirements Analysis Software Interface Analysis IV&V Traceability Matrix Iterative Per Software Release Software Design Analysis TRACEABILITYANALYSIS DELIVERABLES VALIDATION CHANGE IMPACT ANALYSIS TECHNICAL REVIEWS AND AUDITS SPECIAL STUDIES Software Code Analysis Findings and Recommendations Developer Test Analysis IV&V Metrics Monthly Progress/Status Reports Phase Dependent IV&V Tasks Phase Independent IV&V Tasks

12. IV&V Technologies - COQ

13. Experimental Simulation Qualitative and quantitative results based on non-deterministic or hybrid simulation model Math Modelingquantitative results based on a deterministic model Empirical Research Summary • Mirrors a segment of the real world, control of variables is high, supports testing of causal hypothesis, results can be replicated, high internal validity and generalizability • Captures real world context in which to isolate and control variables • Researcher bias can be introduced through selection of variables, parameters and assumptions concerning the model. Modeling requires high degree of analytical skill, and interdisciplinary knowledge • Results are not typically generalizable to other populations or environmental contexts, researcher bias is common,

14. Process Modeling and Simulation • Managed, measured, productivity • gains through: • process improvement • data driven decision-making • technological innovation • Quantitative valuation of • COQ vs COPQ

15. COQ versus COPQ

16. Process Simulation Models • Experimental Simulation Qualitative and quantitative results based on non-deterministic or hybrid simulation model • mirrors a segment of the real world • control of variables is high • supports testing of causal hypothesis • results can be replicated • high internal validity • high external validity, generalizability

17. IV&V Yield • Organizational context factors for cost • Key components

18. Independent Verification and Validation • An organization independent from the developers study the artifacts of software production. • This requires: • Technical independence. Members of the IV&V team may not be personnel involved in the development of the software. • .Managerial independence. The responsibility for IV&V belongs to an organization outside the contractor and program organizations that develop the software. • Financial independence. Control of the IV&V budget is retained in an organization outside the contractor and program organization that develop the software. • IV&V is often perceived as testing the code after the development is completed NASA IV&V is full life cycle activities

19. State of the Practice: Process Maturity Source: SEI Web Site SEMA Report for March 2000

20. IV&V Developed Tools EWatch Dist. System Analyzer Test Management (TMDB) Risk Management (RMS) Req. Analysis (ARDB) Criticality/Risk Analysis (CARA) Code and I/F Analysis (SIAT) IV&V Config. Testbed Req. Measurement (ARM) Flight Simulators IV&V Effort (IVVEE) Issue Tracking (PITS) IV&V Shuttle Toolset MATRIXx Utilities x x x Information Management AATT x x x x x CLCS x x x x x x EOS x x x x x x x x x ISS x Information Analysis NOAA x x x x x x Shuttle x x x x x x x x X33 x x x x x x x x Other Measuring IV&V Effectiveness

21. IndustryBenchmarking Source for US Data: Capers Jones (2000) Software Assessments, Benchmarks, and Best Practice, Addison-Wesley, p 339, System Software Baseline.

22. IV&V Yield • System factors for cost and gain

23. Prior Empirical ROI Studies ROI: Independent V&V Benefits IV&V applied early in the lifecycle has the greatest ROI. Source Jet Propulsion Laboratory TR.

24. Orion 3 Zenit 2 Delta 3 Flight 965 USAF STEP Titan 4B Airbus A320 Ariane 5 IMPACT of Major Air & Space Software Problems [DS-1] [Galileo] [Lewis] [Poseidon] [Pathfinder] [Galileo] [NEAR] ‘93 ‘96 ‘97 ‘98 ‘99 Aggregate Cost: $640 million $116.8 million $255 million $1.6 billion Loss of Life: 3 160 Loss of Data: [‘99] – NASA IV&V presentation

25. Tracing Impacts to Causes…Cause-Effect Graphing Mission Success at Reduced Cost Safety Objective Quality Objective Reliability Objective Cost Objective Identify and Manage Risks Identify and Eliminate Hazards Defect Prevention Defect Detection IT Infrastructure, Web-based reporting, DSS, ARM, PITS, RMS, Ask Pete, ARRT Communication Channels & Reporting Process Improvement Avoid Rework Eliminate Redundancy Efficient Resource Allocation Skilled Workforce Domain Experts Engineers V&V Experts PL Reuse Technologies Domain Engineering Knowledge Maintenance V&V Models and Methods Information Analysis & Information Management, Product Certification Skills training program

26. BSC Cause and Effect Graphing Strategic and Financial Goals Competitive Objective Quality Objective Reliability Objective Cost Objective Identify and Manage Risks Optimize resource allocation & utilization Defect Prevention Defect Detection IT Infrastructure, Web-based reporting, COMPASS, TIGERS, TeamPlay, Communication Channels & Reporting Process Improvement Avoid Rework Eliminate Redundancy Efficient Resource Allocation Skilled Workforce Black Belts Engineers Telecom Experts SIX SIGMA Performance Excellence Knowledge Maintenance Communications Models and Methods Information Analysis & Information Management Product Certification Skills training program - Motorola University

27. Filter Attributes

28. DTE – Rule Based

29. NEURAL NETWORK

30. Intelligent update of rule structure

31. STATISTICAL ANALYSIS

32. BENEFITS • The benefits of this proposed Center Initiative would be applicable to all NASA software development organizations for whom IV&V is an option. The formalization of an objective decision-making process, along with enabling support tools would provide key capabilities to make rational budgetary decisions that impact safety and mission critical aspects of all NASA software systems. This is significant in enabling NASA to engage in effective administrative and managerial control based on objective, quantified information. • The techniques proposed under this initiative will also provide NASA participants increased visibility into their process improvement efforts. The ISO-9001 certification requires that managers be able to document the benefits contributed to the organization by specific processes and process improvement effort [8]. A formalized, well-defined decision-making process would therefore make a significant contribution to NASA’s overall quality strategy.

33. MILESTONES Start=July 20, 2001 + 3 mo IV&V Process Description – Product Characterization Based on prior CSIP results Start=July 20, 2001 + 6 mo Information Analysis Data gathering for methodology Start=July 20, 2001 + 6 mo Initial Prototype Demonstration(s) & Iteration(s) Delivered GSFC IV&V interface required

34. Nancy S. Eickelmann, PhDMotorola Labs1303 E. Algonquin Rd.Annex-2Schaumburg, IL 60196Phone: (847) 310-0785Fax: (847) 576-3280Nancy.Eickelmann@motorola.com