NASA Langley Research Center Software Process Improvement Initiative (SPII) Findings Presentation

1. NASA Langley Research Center Software Process Improvement Initiative (SPII) Findings Presentation October 27, 1997

2. CornerStone Findings Presentation • CornerStone Overview • Findings • Next Steps

3. CornerStone Overview

4. CornerStone Goals • The CornerStone Phase is the foundation of LaRC’s overall Software Process Improvement Initiative • The goals of the CornerStone Phase are: • Develop a plan to improve LaRC’s software development practices • Identify current state of software development at LaRC • Identify current best practices used in software development at LaRC • Develop a High Performance Model for LaRC’s software development activities (incorporates the appropriate elements of the Capability Maturity Model, ISO 9000-3, Strategic and Quality Framework, and Baldrige Award Criteria) • Obtain management’s support, complete with resources, to implement LaRC’s Software Process Improvement Plan

5. Software Process Improvement Initiative (SPII) Goals • Improve the work environment for LaRC’s software community, leading to higher morale and increased productivity • Develop sustainable mechanisms for continuous improvement in the productivity and quality of software developed across LaRC • Increase customer satisfaction with LaRC software products

6. CornerStone Activities • Lay the Foundation • Establish Infrastructure • Plan for the Assessment • CornerStone Planning and Validation • Baseline • Workshop Training • Customer Workshops • Supplier Workshops • Follow-Up Interviews (as needed) • Best Practices Documentation • Analyze Workshop Results • Prepare and Present Findings • Plan • Define LaRC SEPG • Develop SPI Plan • Review SPI Plan with Sponsors • Present SPI Plan • Implement Improvements …

7. CornerStone Team Members Norma Campbell, RTG/FDCD Victoria Chung, IOG/ISSD Michael Holloway, RTG/FETD Chuck Niles, IOG/FSED Pam Rinsland, IOG/AESD Pat Schuler, IOG/ISSD Jim Townsend, RTG/FMAD Jim Watson, OSEMA/OMA Sue Voigt, SASPG/SSCD Consultant: Cindy Torpey, ChangeBridge Inc.

8. CornerStone Scope • Centerwide involvement • Organizations involved in software management, development, and maintenance • Customers of software projects • 101 civil servants and contractors interviewed

9. CornerStone Principles • Start with a process framework • Baseline current state (not audit) • Conduct interviews and discussions • Observe strict confidentiality • Involve senior management • Work as a Centerwide team • Focus on LaRC needs

10. Level Process Characteristics Predicted Performance Optimising Process improvement is institutionalised Probability Target N-2 Time/$/... Product and process are quantitatively controlled Probability Managed Target N-y Time/$/... Software engineering and management processes defined and integrated Defined Probability Target N-x Time/$/... Project management system in place; performance is repeatable Repeatable Probability Target N-1-a Time/$/... Process is informal and unpredictable Initial Probability Target N Time/$/... Capability Maturity Model

11. Capability Maturity Model • Repeatable / Level 2 (All Key Process Areas covered in interviews ) • Requirements Management • Software Project Planning • Software Project Tracking & Oversight • Software Quality Assurance • Software Subcontractor Management • Software Configuration Management • Defined Level (Selected Key Process covered in interviews) • Training Program • Software Product Engineering • Intergroup Coordination • Peer Reviews

12. CornerStone Approach…. • Baseline current state (snapshot in time) • …some improvements are already initiated • Baseline included all areas which impact the software development group • … some are under our control, others will require a coordinated effort • Not all points are at the same detail • …some are very specific, other are more general • Processes were assessed, not specific groups • …especially when we discuss SQA and Training Program • Identify areas for improvement • …not specify how to

13. CornerStone Findings • Key Process Area • Purpose and Description • Best Practices • Improvement Opportunities

14. Requirements Management • Purpose • Establish a common understanding of the customer’s requirements that will be addressed by the software project • Description • Establish and maintain an agreement with the customer on the requirements for the software project • The "customer" is a system engineering group, TAG, a project office, another internal organization, or an external customer • Agreement covers both the technical and nontechnical (e.g., delivery dates) requirements • Agreement forms the basis for estimating, planning, performing, and tracking the software project's activities throughout the software life cycle

15. Requirements Management • Best Practices • Requirements are prioritized and managed according to priority • Developer works with customer to interactively draw out requirements • Operational Concepts Documents help in understanding the requirements (some projects put on the web for increased visibility) • Test cases traceable back to requirements • Explicit contractor task to capture and manage requirements • Rapid prototyping to validate requirements with customer

16. Requirements Management • Improvement Opportunities • Importance of requirements is not realized • Requirements are neither defined early enough, nor refined far enough • Inadequate resources dedicated to requirements management • Software requirements do not track to system requirements • Don’t know how to adequate document requirements • Customers are not trained in requirements generation • Changes in requirements handled by overtime masks over-commitment • Requirements creep is not managed nor controlled (cost is not understood) • No established policy to guide projects in requirements management • Role and responsibility of systems analysis is not clearly understood • Requirements are not consistently documented and reviewed with the customers • Segmentation of responsibilities between researchers and software developers leads to uncertainty in product functionality

17. Requirements Management • Consequences • Project schedule and cost are significantly increased when requirements are inadequately defined and documented • Rework is common due to changing requirements • Customer satisfaction is decreased when their requirements are not met

18. Software Project Planning • Purpose • Establish reasonable plans for performing software engineering and managing the software project • Description • Develop estimates for the work to be performed, establish the necessary commitments, and define the plan to perform the work

19. Software Project Planning • Best Practices • Project cost, effort and duration estimates are developed and tracked • Software Work Breakdown Structure is generated at the same level as hardware WBS • Scheduling tools (MS Project, REVIC) are used • Statements of Work specifies creation of project plans • Software staged release at planned milestones • Software manager designated to oversee a software development project • Software personnel participate in project plans • SQA involved in project planning

20. Software Project Planning • Improvement Opportunities • Schedules are driven by externally set milestones, instead of actual work required • Software Management Plans are not consistently documented, if at all • No planning metrics are captured across the organization on which to base realistic future estimates • Commitments are not honestly negotiated • Inadequate project planning is compensated for by overtime • Over-commitment • No LaRC project policy for managing software projects - the approach to software project management is project dependent • The value of software project plans is not fully understood by developers, managers, or researchers • Personnel are not aware of available procedures and guidelines for software estimating • No guidelines are available for document software development plans

21. Software Project Planning • Consequences • Software projects do not meet schedules or budgets (could face cancelation) • Burn out civil servants and contractors

22. Software Project Tracking and Oversight • Purpose • Provide adequate visibility into actual progress so that management can take effective actions when the software project's performance deviates significantly from the software plans • Description • Track and review the software accomplishments and results against documented estimates, commitments, and plans • Adjust plans based on the actual accomplishments and results

23. Software Project Tracking and Oversight • Best Practices • Informal interchange meetings are held frequently for tracking • Automated tools (MS Project, PC Artemis) are used to track project progress • Milestones used as checklist • Formal reviews held at selected milestones • Dedicated business manager to track software project status

24. Software Project Tracking and Oversight • Improvement Opportunities • Corrective actions are not taken in a timely manner • Software costs are not accurately reflected in project charge structures • No measurements or lessons learned are captured and used for future projects • Software size is not estimated or tracked • Managers are not trained in software project management and tracking • Technical issues are not managed and communicated • Software Development Plans are not used to manage the project • Software project reviews are ad hoc and ineffective; results are not elevated to management (guidelines needed)

25. Software Project Tracking and Oversight • Consequences • Products not completed on expected dates • The real software project status is not known until it is too late to do anything about it

26. Software Subcontractor Management • Purpose • Select qualified software subcontractors and manage them effectively • Description • Select a software subcontractor • Establish commitments with the subcontractor • Track and review the subcontractor's performance and results

27. Software Subcontractor Management • Best Practices • Periodic reviews are conducted with contractors to review progress and communicate status • COTRs and Technical Monitors are designated to establish and manage the software contract • Source Evaluation Boards select contractors based on their ability to perform the work • Success has been demonstrated using Performance Based Contracting • Formal quarterly evaluations are required • Software services are acquired using standard NASA contracting regulations

28. Software Subcontractor Management • Improvement Opportunities • No accountability for poor contractor performance • COTRs and Technical Monitors are not trained in software engineering or managing software development efforts • Performance Based Contracting is misunderstood, and due to poor training it is ineffectively applied • Source Evaluation Board frequently recommends lowest bidder rather than best qualified • Required reviews are not consistently conducted • Contractor turnover is high and requires frequent re-orientation resulting in increased costs • No written LaRC policy or guidelines for managing software contracts • Contractors over-commit and rely on “free” overtime • Skill level and training on the contract does not match what is required

29. Software Subcontractor Management • Consequences • Frequent dissatisfaction with contractor’s products • PBC task generation is more time-consuming

30. Software Quality Assurance • Purpose • Provide management with appropriate visibility into the process being used by the software project and of the products being built • Description • Review and audit the software products and activities to verify that they comply with the applicable procedures and standards • Provide the software project and other appropriate managers with the results of these reviews and audits

31. Software Quality Assurance • Best Practices • SQA contractors advise and give consultation to projects on software engineering when requested (on configuration management and software management plan) • Infractions are reported to participating project teams

32. Software Quality Assurance • Improvement Opportunities • Need sufficient staff for uniform coverage of SQA on Langley software projects • Increase awareness of added value of SQA practices • Appropriate guidelines for SQA activities (current LHB not widely accepted) • Review SQA activities on a regular basis • Track cost and associated return on investments on SQA tasks

33. Software Configuration Management • Purpose • Establish and maintain the integrity of the products of the software project throughout the project's software life cycle • Description • Identify the configuration of the software (i.e., selected software work products and their descriptions) at given points in time • Systematically control changes to the configuration • Maintaining the integrity and traceability of the configuration throughout the software life cycle • Placed under software configuration management both the software products delivered to the customer (e.g., the software requirements document and the code) and the items identified with or required to create these software products (e.g., the compiler)

34. Software Configuration Management • Best Practices • Configuration Control Boards are established and used to control changes, assess impacts and communicate status • Automated tools (ClearCase, PVCS, RCS, LibLink, CVS, LabView) are used to control software work products (requirements, change rationale, code and supporting documentation) • Intent of SCM can be met without the use of automated tools for projects • Web-based change request system established (ClearDDTS) • SCM process documented and followed on some projects

35. Software Configuration Management • Improvement Opportunities • Rigor of formal SCM is not always appropriate for project’s size, phase and purpose • Change request status is not adequately communicated across all project personnel • Lack of awareness of existing guidelines and templates results in significant duplication of effort • Insufficient funding for SCM staff, tools and training • SCM typically applied too late in project phase • SCM plans are not documented and/or followed and changes are not controlled • Lack of understanding of the benefits of SCM • Contracts do not always require the appropriate level of SCM and there is inadequate CM on deliverables

36. Software Configuration Management • Consequences • Lack of SCM results in compromised mission certainty and validity of data and products

37. Training Program • Purpose • Develop the skills and knowledge of individuals so they can perform their roles effectively and efficiently • Description • Identify the training needed by the organization, projects, and individuals • Develop or procure training to address the identified needs • Evaluate current and future skill needs and determines how these skills will be obtained • Use informal vehicles when appropriate (e.g., on-the-job training and informal mentoring)

38. Training Program • Best Practices • Training Office exists to meet the needs of the LaRC including the software development community • Some projects and organizations successfully leverage the capabilities of the Training Office to meet their needs • Training Office annually surveys LaRC to establish training needs

39. Training Program • Improvement Opportunities • Increase is needed in training budget due to cross training and retraining of LaRC personnel • Inconsistent management support for training • Insufficient time is allocated for attending training • Training is not considered a priority • Need full implementation of Individual Development Plan (IDP) which ties to the SQF and the agency strategic plan • No effective mechanism for consistently training contractors and civil servants working on the same project • No project training plan • Travel funds shortage limits training opportunities and technical exchange

40. Software Product Engineering • Purpose • Consistently perform a well-defined engineering process that integrates all the software engineering activities to produce correct, consistent software products effectively and efficiently • Description • Perform the engineering tasks to build and maintain the software using the project's defined software process and appropriate methods and tools

41. Software Product Engineering • Best Practices • Operational Concept, Users Manual and Test Plans/Cases developed prior to code • Apply techniques such as Object Oriented Design (OOD), reverse engineering, structured programming, and modularity to reduce code complexity, increase reusability and improve maintainability • Automated software development tools (Purify, Quantify, PureCoverage, Rational Rose, ClearCase, ClearDDTS, Object Manual, LabView GUI) are used to design, test, CM, and document software • Provide realistic operational testing scenario for software • Office automation tools used in innovative ways for software development, testing and documentation (databases, spreadsheets)

42. Software Product Engineering • Best Practices (cont.) • A project has demonstrated LaRC’s ability to meet FAA standards for airworthy flight software • On line Web sites exist that describe software engineering guidebooks, formal methods and metrics collection database • Third party tests against requirements agreed to by users • Continuous rapid prototyping used to demonstrate feasibility and early progress • Requirements gathering techniques resulted in improved software product • Centralized facility provides access, guidance and expertise for domain-specific tools

43. Software Product Engineering • Improvement Opportunities • Insufficient time allocated for software engineering tasks (requirements definition, design, testing, integration, configuration management, and documentation) • Software engineering activities often hidden because their value is not recognized, perceived as overhead by projects, and researchers/projects do not want to pay for reuse (no corporate view for long-term investment) • No rewards for good software engineering • Single point failures on many projects created by one person and insufficient documentation • Testing address physics only, not software quality • No LaRC dissemination guidelines for software exist

44. Software Product Engineering • Improvement Opportunities (cont.) • Lack of in-house software product engineering expertise • Need point of contact where software engineering expertise and guidance can be obtained • Ineffective tool utilization due to poor communication and awareness • Web sites not advertised • Software engineering techniques not appropriately tailored for small projects • No incentive to take research code to production quality or make it reusable by other projects • Lack of expertise in project planning and scheduling leads to insufficient time for fundamental software engineering tasks

45. Software Product Maintenance • Best Practices • Use portable computer to set up, perform experimental test, and analyze results • Improvement Opportunities • Insufficient funding for sustaining and maintenance of software • Ineffective implementation of CM prohibits quick minor changes to software for experimental use • Insufficient verification of software prior to delivery leads to high maintenance cost • Due to insufficient manpower and maintenance procedures, staff is not adequately notified on software changes • Constant changes in COTS upgrade decrease productivity due to perpetual learning curve

46. Intergroup Coordination • Purpose • Establish means for the software engineering group to participate actively with the other engineering groups so the project is better able to satisfy the customer's needs • Description • Software engineering group participates with other project engineering groups to address system-level requirements, objectives, and issues • Representatives of the project's engineering groups participate in establishing the system-level requirements, objectives, and plans by working with the customer and end users, as appropriate • Requirements, objectives, and plans become the basis for all engineering activities

47. Intergroup Coordination • Best Practices • Workshops and shared facilities bring different groups together for improved information exchange • Interface Control Documents, frequent meetings and timely meeting notes ensure effective exchange of information • Web-based information and email exchange facilitates technical coordination • Physical co-location of discipline specialists promotes intergroup coordination during a project • Teamwork training is available • Communication between software programs used by different groups achieved by in-house developed interfaces (scripts, GUIs, wrappers, standard features, filters)

48. Intergroup Coordination • Improvement Opportunities • Need to have representatives of all technical disciplines (including software) involved from the start of a project (requirements, cost estimates, operational concepts, requirements allocation) • Due to the lack of systems engineering performed at LaRC, software and hardware staff are forced to fill that role in an ad-hoc manner • Poor communication among groups doing similar work within LaRC results in duplication of work • No effective mechanism to ensure all disciplines are represented on project teams • Software specialists not aware of overall project concept • Perception that software can fix anything, including poor hardware selection, leads wasted funds and staff hours • Lack of documented commitments between engineering groups • Changes are not effectively communicated across the engineering groups • Contractors resist communication due to proprietary fear

49. Peer Reviews • Purpose • Remove defects from software products early in the development process where it is more cost effective to remove them • Develop better understanding of software products and defects that might be prevented • Description • Peers methodically examine software work products to identify defects and areas where changes are needed • Identify and schedule specific products that will undergo peer review as part of the defined software process

50. Peer Reviews • Best Practices • Peer reviews identify problems early in the lifecycle resulting in saved time and decreased costs • Post-mortem review is effective mechanism to capture lessons learned • Informal review by an in-group peer works well on small projects • Peer reviews captures problems not otherwise identified • Completion of peer reviews is a phase exit requirement and indicates readiness to proceed • Peer reviews are effective mechanism to train staff and indoctrinate new hires • Peer review process is documented (NASA Formal Inspection Handbook, project documentation) • User requirements are basis for code reviews • Email and Key Activities are used to document peer review results