Using Process Improvement and Knowledge Management for Better Predictive Analysis Capability

1. Using Process Improvement and Knowledge Management for Better Predictive Analysis Capability Rick Hefner Marilee J. Wheaton 310.812.7290 310.813.6510 rick.hefner@trw.com marilee.wheaton@trw.com TRW One Space Park - R2/2130 Redondo Beach, CA 90278 Presented to Sixteenth International Forum On COCOMO and Software Cost Modeling

2. Motivation for Improvement CMMI Level 4 Requirements and Expectations Organizational Metrics Process Database Design and Implementation Lessons Learned Systems Architecture Vision Agenda 16th COCOMO Forum

3. Motivation for Improvement to Level 4 • Level 3 ensures well-defined, repeatable processes • Competition demands better quantitative understanding of the contributors to cost and schedule • Process productivity • Product quality (which drives rework) • Needed better measures, data, and analytic techniques for critical process and product characteristics: • Determine whether processes are behaving consistently or have stable trends (i.e., are predictable) • Identify improvement in TRW’s standard processes • Identify project practices which may be best practices • Understand the cost-quality-schedule tradeoffs 16th COCOMO Forum

4. How Does Six Sigma Fit with ISO 9001 and CMMI? Process Improvement Best-Practices Six Sigma Quality Mgmt. Business Measures Voice of the Customer ISO 9000 SW CMM Change Management DMAICDFSS Process Management ISO 9001 Methods & Tools CMMI • Capability Maturity Model Initiative (CMMI) and ISO 9001 establish the change management and process management framework needed for Six Sigma • Six Sigma methods and tools assist in the quantitative analysis needed at CMMI Levels 4/5 16th COCOMO Forum

5. Continuous process improvement 2 Managed CMMI-SE/SW Staged Representation Focus Level Process Areas Causal Analysis and Resolution Organizational Innovation and Deployment 5 Optimizing Quantitative management 4 Quantitatively Managed Quantitative Project Management Organizational Process Performance Organizational Process Focus Organizational Process Definition Organizational Training Integrated Project Management Risk Management Decision Analysis and Resolution Requirements Development Technical Solution Product Integration Verification Validation Process standardization 3 Defined Requirements Management Project Planning Project Monitoring and Control Supplier Agreement Management Measurement and Analysis Process and Product Quality Assurance Configuration Management Basic project management 1 Performed 16th COCOMO Forum

6. Level 4 Changes in the CMMI • SW-CMM process areas are split by process and product quality • Quantitative Process Mgmt Identify and correct special causes of process variation • Software Quality MgmtDevelop a quantitative understanding of the quality of the project's software products • CMMI process areas are split by organization and project • Organizational Process PerformanceMaintain a quantitative understanding of the performance of the organization’s set of standard processes, and provide the process performance data, baselines, and models to quantitatively manage the organization’s projects. • Quantitative Project ManagementQuantitatively manage the project’s defined process to achieve established quality and process performance objectives. 16th COCOMO Forum

7. Organizational Process Performance (CMMI) • Establish and maintain a quantitative understanding of the performance of the organization’s set of standard processes, and to provide the process performance data, baselines, and models to quantitatively manage the organization’s projects. • Required Goals • SG 1 Establish Performance Baselines and ModelsBaselines and models that characterize the expected process performance of the organization's set of standard processes are established and maintained. • GG 3 Institutionalize a Defined ProcessThe process is institutionalized as a defined process. ExpectedImplementation Practices ExpectedInstitutionalization Practices 16th COCOMO Forum

8. Expected Implementation Practices • SP 1.1 Select ProcessesSelect the processes or process elements in the organization's set of standard processes that are to be included in the organization's process performance analyses. • SP 1.2 Establish Process Performance MeasuresEstablish and maintain definitions of the measures that are to be included in the organization's process performance analyses. • SP 1.3 Establish Quality and Process Performance ObjectivesEstablish and maintain quantitative objectives for quality and process performance for the organization. • SP 1.4 Establish Process Performance BaselinesEstablish and maintain the organization's process performance baselines. • SP 1.5 Establish Process Performance ModelsEstablish and maintain the process performance models for the organization's set of standard processes. 16th COCOMO Forum

9. Approach • Our Level 3 process database supported cost estimation and process improvement • Surveyed management team to establish business drivers • Defined measures needed to characterize process performance and quality at the organizational level • Defined measures needed to characterize project satisfaction of organizational goals • Identified sub-processes amenable to quantitative management • Defined project measures needed for quantitative management of those sub-processes • Examined improvements in the organization standard process needed to stabilize the process or make measures meaningful • Examined improvements in the project’s defined processes 16th COCOMO Forum

10. Example Organizational Metrics Collected and Derived • Collect base measures • Size • Effort by activity • Cost by activity • Number of defects (By phase) • Derive other measures • Defect density (defects/size) • Productivity (size/effort) • Defect containment (defects saves/escapes from defects by phase) • Rework effort • SPI/CPI (planned vs. actual effort) 16th COCOMO Forum

11. How Projects Use theOrganizational Process Assets CMM Organizational Standard Process & Organizational Procedures Process Asset Library (PAL) OrganizationalDatabase OrganizationalTraining Office Organizational Policies Senior ManagementReview Organization Project Project Schedules & Budgets Project Plans Project Results Project Defined Process& Procedures 16th COCOMO Forum

12. Project Comparative Data Base (PCDB) • PCDB contains: • Cost data from a certified accounting system • Project validated technical characteristics data • PCDB relates: • Cost data to standardized (WBS) work elements • Cost data to the technical characteristics of work performed • The Office of Cost Estimation uses the PCDB to: • Calibrate the parametric models • Derive cost estimating relationships • Provide historical data for proposals, project planning/replanning • Define risk affordability/analysis • Characterize process performance and quality 16th COCOMO Forum

13. Accounting Data Project Inputs PCDB Standard WBS JN Mapping PCDB Standard WBS Alternate Hours/Cost Mapping Accounting Data Labor hours, dollar costs Non-labor costs Breakdown by PCDB WBS element, by labor category S/W Development Descriptive Data Examples SLOC, other size measures ESLOC (derived) Labor Required to Develop SLOC/ESLOC Software Development Other Project Disciplines Cost Model Parameters Accounting & Descriptive Data Project Management Systems Engineering Hardware Development Software Development Systems Integration & Test Site Activation Integrated Logistics Configuration Management Data/Documentation Management Quality Assurance Development Support Facility Operations and Maintenance Specialty Customer Services Other Activities What is the PCDB ? 16th COCOMO Forum

14. Parametric Cost Models CERs Estimation Notebook Guidelines PCDB Support to Cost Modeling and Proposals RFP Project and Proposal Team Support Functional Requirements Final Estimates First Order Estimates Advanced Pricing System (APS) Model Calibration Project Comparative Data Base (PCDB) Cost Volume Sanity Check Final Estimates Project Technical Description Data & Metrics Certified Accounting Data BOEs Sanity Check BOE Generation Support Actuals Management Information System (AMIS) Project Data 16th COCOMO Forum

15. Proj OK? Data Review and Validation Retrieve Accounting System Data Start End OCE Place Data in PCDB in the Review State Place Data in PCDB in the Input State Generate Derived Data and Summary Reports Project Staff Place Data in the Reportable State Update Input Data PCDB Data Submittal Process No Project Prepares and Submits Descriptive Data Inputs Yes Project Prepares and Submits Accounting Data Inputs Proj & OCE OK? No Yes Perform Parametric Validation if Applicable 16th COCOMO Forum

16. Parametric Validation/Calibration • Additional selective validation for software development History Data Points • Creation of parametric cost model baseline • Input from Project • Descriptive Data Forms: Software Development • OCE provides guidance and assistance • Provided to Project • Software Parametric Model (i.e., Costar (COCOMO II), SEER-SEM, Price S, or SLIM) cost estimation validation • Iteration with project to achieve validation within 10% - 20% of actual effort • Results in calibrated baseline for future estimates 16th COCOMO Forum

17. Lessons Learned - 1 • Determining a common set of metrics is tough because there are different needs (and frequency, granularity, accuracy, etc.) • Project management decisions • Customer insight • Senior management oversight • Organizational process performance characterization • Process improvement • Proper support for metrics collection requires changing the culture to “management-by-data” • Project managers must use the data to manage their projects • Senior managers must use the data to meet organizational business objective • Data collectors must be confident that data will not be used against them 16th COCOMO Forum

18. Lessons Learned - 2 • Understanding variation in process performance allows more insight into estimation • What’s the likely cost of this work? • What’s the probability we can perform the work for $____? • Project resistance to data collection is primarily due to the time and effort required to collect and report the data • Must be integrated and consistent with the process • Data collection mechanisms require clear instructions, to ensure the desired information is captured and validated • Collection should be a combination of automated and manual methods for cost and accuracy 16th COCOMO Forum

19. Systems Architecture Vision Current Situation Project Risk Assessment • Organizational Metrics Data What if Modeling Predictive Analysis • Financial Data Project Risk Assessment What if Modeling Risk Radar • Project Mgmt Data • Decision Support Layer • Organizational Metrics Data • Group Common Data Repository • Financial Data • Proposal Data • Project Mgmt Data The End Objective • Integrate Metrics with Financial, Project, proposal and other information to support trend and risk analysis • Proposal Data 16th COCOMO Forum

20. Conclusions • The process database developed at Level 3 is a key asset in achieving Level 4 • The many uses of metrics places additional emphasis on innovative database design and usage • Characterizing the organization’s process performance requires: • Definitizing your business goals • Selecting the right metrics • Stabilizing the organization and projects’ processes • Collecting and analyzing the metrics • Management and decisions that are data driven result in better predictive analysis capability 16th COCOMO Forum