Estimating Effort for PPQA Using the Raytheon 6  Process

1. Estimating Effort for PPQAUsing the Raytheon 6 Process Raytheon North Texas Software Engineering Donna Freed

2. Level 5 Level 5 RTIS Integrated Level 4 CMMI RTIS Integrated Level 4 CMMI Software Sigma Product Software Sigma IPI Product RTIS Policies & IPI & Cycle Time RTIS Policies & Development & Cycle Time Development Procedures Procedures Process Process Level 3 Level 4 RTIS Software RTIS Software Baseline 10x Operating Operating Validation Fault Density Baldridge Instructions Level 3 2003 2003 2003 2003 Instructions Improvement Award 2002 2002 2002 2002 2001 2001 2001 2001 2000 2000 2000 2000 Level 2 - - - - 1999 1999 1999 1999 1997 1997 1997 1997 1996 1996 1996 1996 1995 1995 1995 1995 1994 1994 1994 1994 1993 1993 1993 1993 1992 1992 1992 1992 Raytheon Raytheon Raytheon 1991 1991 1991 1991 1990 1990 1990 1990 Level 4/5 Level 4/5 Level 4/5 Acquisition Acquisition Acquisition 1989 1989 1989 1989 Managed/ Managed/ Managed/ Level 3 Level 3 Level 3 & Transition & Transition & Transition Level 2 Level 2 Level 2 Optimizing Optimizing Optimizing Defined Defined Defined Repeatable Repeatable Repeatable Self - assessment • Self - assessment • • Software Improvement • Software Improvement Team Formed Team Formed N TX Software Engineering Process Improvement Journey Level 5 Level 5 Level 5 Level 5 RTIS Integrated RTIS Integrated Level 4 Level 4 CMMI CMMI RTIS Integrated RTIS Integrated Level 4 Level 4 CMMI CMMI Software Sigma Software Sigma Product Product Software Sigma Software Sigma IPI IPI Product Product RTIS Policies & RTIS Policies & IPI IPI & Cycle Time & Cycle Time RTIS Policies & RTIS Policies & Development Development & Cycle Time & Cycle Time Development Development Procedures Procedures Procedures Procedures Process Process Process Process Level 3 Level 3 Level 4 Level 4 RTIS Software RTIS Software RTIS Software RTIS Software Baseline Baseline 10x 10x Operating Operating Operating Operating Validation Validation Fault Density Fault Density Baldridge Baldridge Instructions Instructions Level 3 Level 3 2003 2003 Instructions Instructions Improvement Improvement Award Award 2002 2002 2001 2001 2000 2000 Level 2 Level 2 - - 1999 1999 1997 1997 1996 1996 1995 1995 1994 1994 1993 1993 1992 1992 Raytheon Raytheon 1991 1991 1990 1990 Level 4/5 Level 4/5 Acquisition Acquisition 1989 1989 Managed/ Managed/ Level 3 Level 3 & Transition & Transition Level 2 Level 2 Optimizing Optimizing Defined Defined Repeatable Repeatable Self Self - - assessment assessment • • Self Self - - assessment assessment • • • • Software Improvement Software Improvement • • Software Improvement Software Improvement Team Formed Team Formed Team Formed Team Formed IPI CMM - Based Internal Process Improvement Assessment IPI CMM - based Internal Process Improvement Assessment RTIS Raytheon/TI Systems RTIS Raytheon/TI Systems CMMI CMM Integrated CMMI CMM Integrated

3. CMMI at Levels 4 & 5 Set Objective Performance & Quality Goals Measure Improvements & Re-baseline CPI Stabilize process & Establish Process capability Baseline Deploy Improvements Select & Prioritize Improvements Pilot Improvements

4. SQE Cost Estimating (Develop) SQE Cost Estimating (Deploy) R6s Acronyms / Abbreviations: Cal. = Calibrate C & S = Cost and Schedule Integ. = Integration R6s = Raytheon Six Sigma Project Rvw = Review SQE = SW Qualify Engineering SCM = SW Configuration Management Std = Standard SWEC = SW Engineering Center Data TS = Trade Study process WBD = Wideband Delphi Size Estimate nn = Org. Improvement Opportunity # SCM Cost Estimating (Develop) SCM Cost Estimating (Deploy) 36 Select Std C&S Model (Research) Use Std C&S Model (Develop) Use Std C&S Model (Pilot) Use Std C&S Model (Deploy) 24 TS Cost Estimating (Umbrella) Cost Rvw Package (Develop) Cost Rvw Package (Pilot) Cost Rvw Package (Deploy) Cal. SWEC & SEER (Develop) Improve WBD (Develop) Improve WBD (Pilot) Improve WBD (Deploy) 45 C & S Integ. Tool (Develop) C & S Integ. Tool (Pilot) C & S Integ. Tool (Deploy) Key: Completed Active Planned Not Yet Planned OID OE Data Better Data: Productivity(Develop) Better Data: Productivity(Deploy) 19 Cost Estimating Improvement Roadmap

5. Visualize the Future Visualize Celebrate & Plan for the Future Commit to Change Achieve Commit Determine Improvement Opportunities Design & Implement Changes Improve Prioritize Characterize Define Current State R6 Process

6. Visualize • "Imagine the Future" • Create a vision of the future with a clear and pressing need for change. Align sponsorship throughout the value stream to maximize our benefits. • In late 1999, Raytheon introduced a new methodology for software cost estimating that included the use of “adders” or percentages for support engineering. PPQA Support Minimum Selected Maximum Defined Process x x … North Texas ? Example is too low! This Sigma project determines North Texas Guidance for PPQA based on actuals for labor hours

7. Visualize • Confidence in the cost estimation process for PPQA support is low: • Proposals need to be more task based so managers can understand and explain PPQA functions on their programs when presenting proposals. • Johnny Barrett, SWEC Director, challenge for PPQA as input to December 1999 SQE Planning Session • PPQA took the challenge: • to verify and justify the PPQA cost percent estimate used in proposals. • to provide a process for PPQA cost estimation based on historical actuals and tasks required to support the project.

8. Commit • "Commit to Change" • Develop a committed sponsor and team aligned with the vision, accountable and energized to make change. • Commitment from sponsors (SWEC Director) • Commitment from subject matter expert (Software Cost Estimation) • Commitment from specialist (SQE Manager) • To establish commitment, PPQA prepared a detailed break out of the actual labor cost data from 2 historical sources: • SMCRS and the “old” SMDB • Two pareto charts were prepared and presented to the SWECLT, characterizing the actuals. The wide spread of percentages called into question whether there is a valid percentage.

9. Prioritize • “Determine Improvement Priorities" • Define goals and action plans. Commit resources to focused improvement project(s) in order to realize significant results • Goal/Scope: Is there a valid PPQA “adder” percentage? • Analyze actual labor hour ranges for PPQA for software engineering in North Texas • Limited to SQE costs, but approach can apply to other support. • Johnny Barrett, SWEC Director, sponsored this analysis as a R6 project

10. Characterize--Analyze Actuals • Combined SMCRS and SMDB into one file for analysis. • Further refined data by removing all projects without core labor hours. Distribution of Projects Total A B Business Area SMDB SMCRS C D E 0 5 10 15 20 25 30 35 40 45 50 55 60 Number of Projects

11. Characterize • “Define Existing Process and Plan Improvements" • Understand and document current state performance (metrics, process flow, and critical factors); translate the current state opportunities into a plan for improvement. • Validate current data • Break down cost actuals • Omitted projects with data collection in “bid” or “contract” • Selected projects with data collection: Final • Omitted projects with no data or inadequate data in development stages • Other factors to consider • Omitted any project with no SQE actuals Approach: validate then analyze actuals

12. Characterize--Analyze Actuals • Actuals were analyzed using both • linear (y = mx + b) and • polynomial (y = ax2 + bx + c) regression. • Note: y = SQE labor hours, and x = software development labor hours

13. Characterize-- Detailed Results • Coefficient of correlation: • Core: Linear = .85, Polynomial = .86 • Total: Linear = .83, Polynomial = .84 • Virtually no differences between core and total results--either can be used with the same confidence in the cost model • Residual (remaining base constant for x=0) • Core: Linear = 167, Polynomial = -64 • Total: Linear = 180, Polynomial = 6 • Polynomial correlation shows significant advantage over linear especially in model based on total hours (because residual is closer to 0) Our intuition was incorrect!

14. Characterize-- Predictions from the model • Model predicts declining percentage of SQE effort as size of project increases • Model works best with medium size projects • Smaller projects may be inaccurate due to fixed cost in core model • Not enough data points for larger projects

15. Improve – Design and Implement Improvements • “Design and Implement Improvements" • Design and implement integrated improvements and control systems to maximize value. • Improvement Plan • Publish guidance for PPQA cost estimating • Review and approve Cost SMR content for PPQA estimates (Heisenberg principle--the act of observing influences the object being observed) • Verify metrics data input for PPQA actuals • Enforce requirement to collect PPQA costs separately.

16. How do we compare? Northeast SQE Bidding Guidelines vs. N. TX model SQE Support Adders NE Guidance NTX Guidance Project Size Micro x+ 5.5% (+ / -1.5) x+ 1.1% (+ / - .9) Small x+ 2.55% (+ / - 1.5) x+ .7% (+ / - .7) Medium x% (+ / - 1.0) x+ .2% (+ / - .9) Large x- 1.75% (+ / - .75) x- .6% (+ / - 1.1) Very Large x- 3% (+ / - .5) x- 2.1% (+ / - 1) • As a percent of Total Software Development effort, including requirements, design, code and test activities.

17. Achieve – Celebrate Achievements, Build for Tomorrow • "Celebrate Achievements, Build for Tomorrow" • Deliver measurable results that change the way people think and act, create knowledge, build momentum for continuous improvement, celebrate success, and get people excited about doing Raytheon Six Sigma repeatedly. • SQE Cost estimating procedure provides consistency for SQE proposal preparation. • Cost Management review includes the SQE Manager to verify that: • Costs fall within guidelines • Tasks fall within guidelines for software quality process • Tasks outside the standard Raytheon software quality process are estimated separately

18. Are estimates consistent with the model? • Estimates were consistently below the model based on actuals, until 2003. Guidelines were published in mid 2002.

19. How consistent? • Based on 34 Software Cost Estimates over 4 years.

20. Summary • PPQA cost estimating process has been successfully used as guidance for SQE cost estimates on programs in North Texas. • Started with basic tasks for a CMM level 3 organization. • Cost estimates are refined as actuals are included in the model. • A tool for estimating essential PPQA tasks based on the size of projects, with estimates declining in percentage as the size of the project grows. • Reviewing estimates against guidelines results in estimates which are more consistent with actuals.