 Improving the Current State of Software within NASA

1. Improving the Current State of Software within NASA May 8, 2000 Presentation by CIO, CE, and Code Q SMC and NASA Administrator Version 5 Briefing to SMC and NASA Administrator CIO_Code Q_Chief Eng Talk_V5_March 28 V1.ppt

2. NASA Software Working Group Charge • Identify software challenges in NASA • What are our goals • What are our challenges in meeting those goals • Recommend plans to address the challenges and obtain the goals • Provide recommendations concerning the implementation of software best practices • Identify methodologies to evaluate and improve the quality of software across the Agency

3. Goals Primary Goal • Successfully Develop Software to support NASA Enterprises and Missions • High Quality • Meet all project requirements • Safely • Reliably • Within approved budget • Within approved schedule Secondary Goal • Continual Sustained Software Process and Product Improvement • Improve quality • Reduce cost • Increase productivity

4. Sources of Challenges Data • Survey data from individual NASA employees • Input received from Ames, GRC, GSFC, JPL, LaRC, MSFC, & HQ • Data provided from NASA organizations • NASA CIOs presentations (JPL and HQ) • Software Engineering Process Group (LaRC ) • Software Assurance Technology Center (GSFC) • Software Engineering Laboratory (GSFC) • Flight Software Group (MSFC) • Mishap reports • Mars Climate Orbiter Mishap Investigation Board, Phase I Report, NASA, November 10, 1999 • "Reporton the Loss of the Mars Climate Orbiter Mission", JPL D-18441, JPL Special Review Board, Nov. 11, 1999 • "Common Errors Afflicting Real Time Control System Software", Task 2 Interim Report, April 30, 1999, Prepared by JSC NX Technology Division, Approved by M. Himel/Chief, NX Technology Division

5. Findings Sorted into Categories of Challenges

6. Top 10 Software Challenges

7. Sorted List of Software Challenges

8. Findings The following give examples highlighting the seriousness of the top 10 priority challenges • Requirements Management • The flow down of requirements from the higher-level document to the software requirements document was not done resulting in loss of mission • Software Product Engineering • Software requirements are not controlled to establish a baseline for software engineering and management: software requirements are poorly documented (if at all) and are not always reviewed with the customer or kept up to date to reflect and communicate changes, and are not used as the basis for software management plans, products and activities • Software designs using effective methods are not being developed, maintained, documented, and verified to assure the requirements are fully covered and to form a framework for coding; the resulting code either does not work properly or if it works it has very high maintenance costs do to the poor design being hard to modify

9. Findings continued • Software Product Engineering - continued • Software code is not adequately documented and verified; the result is that the software does not meet the requirements • Software testing is performed in an ad hoc manner, test procedures do not fully cover the requirements if they are written at all; software products rarely work as expected, lack of sufficient testing prior to delivery leads to high maintenance costs • Integration testing is performed in an ad hoc manner, integration test procedures do not fully cover the requirements if interface requirements were written at all;partial or total system failure is experienced and hug overruns to pay for expensive error correction in the latter phases of the project • System and acceptance testing • Interface with navigation software not tested • Complete end to end testing including interfaces is not done • Acceptance criteria is either not defined or not well defined

10. Findings continued • Software Quality Assurance and I&V • Software quality assurance staff is 0 to 1 deep at some centers… • Testing until run out of budget or time vs. use of thorough testing methodologies • Systems Engineering • Due to the lack of systems engineering performed, software and hardware staff are forced to fill that role in an ad hoc manner • System engineers often have little software expertise • The role and responsibility of system analysis is not clearly understood or defined • The management of complexity is one of the most important issues facing NASA missions • Software Project Tracking and Oversight • Actual results and performance are not tracked against the plans, corrective actions are not taken and managed to closure … changes to commitments are not agreed upon … • Software Reliability, Safety • Making software safer and more reliable

11. Findings continued • Software Configuration Management • Lack of proper configuration management has contributed and caused mission failure • Human Resources • Software professionals are the first to leave; the hardest to replace • Obtaining and keeping Program/Project Managers sufficiently knowledgeable of software issues is a serious problem • Software Project Planning • The way project phases are currently structured, emphasis is not placed on software planning which leads to huge overruns in testing or high maintenance cost that all could be avoided if proper time was allotted up front • Finding knowledgeable and experienced software project managers who can apply disciplined engineering practices to software development • Management Support • Engineers perceive little or no management commitment to the software process • There are no investment, encouragement, or reward systems for improvements, best practices, or technology transfer

12. Summary • Conclusions • Our current software state is an Ad -hoc application of engineering • We must progress to a state of disciplined, consistent, repeatable application of software engineering principles • We will not achieve software quality without this • Need endorsement for the following to effect a holistic solution • Establish SEPGs at each Center • SWG define a NASA Software Engineering Implementation Plan to manage and coordinate software improvements efforts across the Agency • Headquarters review and approve plan and disposition SWG recommendations • Goals • Successfully Develop Software to support NASA Enterprises and Missions • Continual Sustained Software Process and Product Improvement • See additional findings on the next slides

13. Even Successful Missions may Experience Software Problems (Excerpt From Another Briefing) • “A few days after the July 4th, 1997 landing, the Mars Pathfinder began experiencing total system resets, each resulting in losses of data. The problem was a logical error in the real-time scheduling system---a classic priority-inversion problem. Fortunately, this problem was repairable from earth.” • “A malfunction in one of the on-board computers on Clementine on May 7, 1994 caused a thruster to fire until it had used up all of its fuel, leaving the spacecraft spinning at about 80 RPM with no spin control. This made the planned continuation of the mission, a flyby of the near-Earth asteroid Geographos, impossible.” • “The Magellan spacecraft broke Earth lock and lost communications several timesin August 1990 (soon after entering Venus orbit). It took over six months to identify the source of the problem, which was a timing error in the flight software.”

14. (Excerpt From Another NASA Findings Briefing) • Centers are almost universally weak in: • Project planning • Estimating cost, schedule, and resource requirements for project requirements fulfilled by software • Monitoring and control of software engineering products • i.e., tracking progress and taking effective corrective actions • Configuration management is not universally applied throughout the software development process • Interface between software and system engineering processes is not well defined so agreements, audits, and reviews are not well planned or performed to achieve the most benefit • Software Quality Assurance is generally not well understood nor is its value appreciated Findings by Raymond Kile, Authorized Lead Evaluator Center for Systems Management, Sept 2002

15. Findings(continued) • Software Contract Management • Qualified software contractors are not selected • Constantly receiving poor quality contract deliverables • Software Product Maintenance • The maintenance phase is rarely covered under the project initial budget estimates and plans • Insufficient maintenance staff are available to support ongoing programs • Peer Reviews • Required persons were not in attendance at the walkthroughs… • "…inadequate training of software and other personnel in the software walkthrough process" • Reuse • Software is re-written many times because the software is not developed and documented in such a way that it can be re-used. • Software Coping with Evolving Technology, Tools, Methods, COTS and Environment • Software architectures, testing, and validation for intelligent machines… • Tools that would help economically and accurately produce software are not funded (must compensate for this by manual methods that are prone to human error and much more time consuming and therefore more expensive) • Technology assessment and infusion • There is a huge lack of experience and knowledge in testing, & integration on projects using COTS/GOTS/ middleware/ glueware • Organization Process Focus • Software process development and improvement activities are not planned, funded, or coordinated across the organization • Strengths are not capitalized on, weaknesses go unchecked, those grass roots efforts that are trying to make progress are not coordinated resulting in duplication of effort • Organizational Process Definition • Software is developed using ad hoc approaches instead of sound engineering practices

16. Findings(continued) • Training and Education • Individuals in the software arena do not receive the training necessary to perform their role and training is not always provided even if sought • Integrated Software Management • A project’s defined software process is not a tailored version of the organization’s standard software process since one does not exist • Intergroup Coordination • Requirements and engineering roles and responsibilities are not clearly defined and agreed to • Need improved communication channels, awareness, and understanding between project management, hardware developers, and software developers • Institutional Support / Infrastructure • There is no place to get mentoring and training and advice on all aspects of software development… • Need to find mechanisms to penetrate projects, to introduce software engineering, and to work in partnership with the projects. • Baseline data / Metrics/Lessons Learned • Insufficient project tracking of size, cost, and error data to accurately do future planning, estimating, and process improvement • Security • … ensure that security measures have been implemented on all of NASA's computing systems • Policies and Standards • Currently there is no effective enforcement of the NASA Software Policies NPD 2820.1 • There are no firm guidance or requirements on which software standards should be used in the agency • General CMM related challenges • Flight Software Group was self-assessed at a level 1(of 5) on the CMM at two centers • Faster, Better, Cheaper • Issues related to development/testing/maintenance of real time embedded software for Faster-cheaper-better missions