Static Code Analysis

1. NASA Independent Verification and Validation Program Static Code Analysis Information and Resources NASA IV&V Program NASA/GSFC Code 180 100 University Drive Fairmont, WV 26554 www.nasa.gov/centers/ivv Jarrod Petersavage NASA Independent Verification and Validation Jarrod.M.Petersavage@NASA.gov 304-367-8388

2. Agenda • Introductions • History/Logistics • Static Analysis Overview • What is static code analysis? • Why perform static code analysis? • How do you get started? • What are some of the challenges? • What are some best practices? • What tools does IV&V support? • Available Resources • Next Steps • Questions and Further Discussion

3. NASA IV&V - Three Questions • What is the system-software supposed to do? • What is the system-software not supposed to do? • What is the system-software supposed to do under adverse conditions? These Three Questions are the Foundation of NASA IV&V Efforts

4. NASA IV&V • Full Lifecycle Analysis– provides assurances to the developer based on the 3 questions • Verify and Validate Concept Documentation - validate the selected solution and ensure that no false assumptions have been incorporated in the solution. • Verify and Validate Requirements - ensure the system’s software requirements are high quality (correct, consistent, complete, accurate, unambiguous , and verifiable), and will adequately meet the needs of the system and expectations of its customers and users, considering its operational environment under nominal and off-nominal conditions, and that no unintended features are introduced • Verify and Validate Design - ensure that the design is a correct, accurate, and complete transformation of the software requirements that will meet the operational need under nominal and off-nominal conditions and that no unintended features are introduced • Verify and Validate Implementation – ensure that the transformation from design to code, database structures, and related machine executable representations is correct, accurate, and complete, yielding source code that correctly implements requirements, meets the operational need under nominal and off-nominal conditions, and introduces no unintended features. • Static Code Analysis is one part of implementation analysis • Verify and Validate Test - ensure that the collection of test related content will serve as a sufficient means to verify and validate that the implementation meets the requirements and operational need under nominal and off-nominal conditions • Verify and Validate Operations and Maintenance - ensure operating plans and procedures are correct and usable, ensure that new constraints, changes in the operating environment, proposed software system changes, and their impact on the software are understood and appropriately addressed, and to ensure that anomalies that are discovered during operation are understood and appropriately addressed

5. NASA IV&V Services Software Engineering Tools Laboratory Laboratory for commonly used software engineering tools that enables all missions to share engineering tools and best practices Independent Testing Reusable emulated components, common user interfaces for the rapid instantiation of test and simulation environments enabling our teams to fully analyze the software in an executable environment Cyber Security Laboratory for conducting vulnerability assessments and penetration testing and advancing the tooling and knowledge for ensuring critical systems and information stay secured The Program’s Software Testing & Research Lab enables advanced tooling and testing to be effectively and efficiently applied across all of our customer’s projects

6. NASA IV&V Support Services Static Code Analysis Tool Support Support all of the static code analysis tools that are used within the program (setup, analysis build configuration, running of tools and training) Other COTS Tool Support Support COTS tools that are used within the program to support IV&V work (e.g. MATLAB, DOORS, Rhapsody) Custom Tool Development Develop custom tools to support IV&V business processes, e.g. Risk Management and IV&V Catalog of Methods management The Program’s Software Assurance Tools (SWAT) team provides development and maintenance support for all tools that are used within the program to support all of our customer’s projects

7. NASA IV&V – IA Independent Assessments (IA) • Anything less than lifecycle analysis • Can provide assurances which are coordinated with the customer • Can provide an assessment based on a snapshot in time • Can focus on particular parts of a system • Can include various combinations of IV&V Lifecycle Services, JSTAR services, and SWAT services Some Examples • Provide assurance that XXX will always restart upon soft reset • Provide assurance that the systems at ZZZ facility are secure • Help project YYY meet the Static Analysis requirements in NPR 7150

8. NASA Static Analysis Requirements • NPR 7150.2A SWE-135 The project shall ensure that results from static analysis tool(s) are used in verifying and validating software code. • Note: Modern static code analysis tools can identify a variety of issues and problems, including but not limited to dead code, non-compliances with coding standards, security vulnerabilities, race conditions, memory leaks, and redundant code. Typically, static analysis tools are used to help verify adherence with coding methods, standards, and/or criteria. While false positives are an acknowledged shortcoming of static analysis tools, users can calibrate, tune, and filter results to make effective use of these tools. Software peer reviews/inspections of code items can include reviewing the results from static code analysis tools. • Note: Static analysis tools may not be readily available for some platforms or computing languages. If static analysis tools are determined to not be available, the project can document the alternate manual methods and procedures to be used to verify and validate the software code. These manual methods and procedures will be addressed or referenced in the project's compliance matrix against this requirement.

9. NASA Static Analysis Requirements • NPR 7150.2A SWE-117 The software metrics report provides data to the project for the assessment of software cost, technical, and schedule progress. The software metrics report shall contain as a minimum the following information tracked on a CSCI basis:…c. Software quality measures …i. Results from static code analysis tools. Note: The next revision of NPR 7150 is currently underway andSWE-117 is potentially being removed.

10. Software Truths Software often determines the degree of achieved success in the fielded capabilities of our systems. Software often consumes the majority of costs for system development. Software is on the critical path of every development effort and often determines when and if a system is fielded. Software is the entity changed most often in the acquisition lifecycle of a system to meet the changing needs of the users.

11. Static Analysis Overview • What is static code analysis? • Why perform static code analysis? • How do you get started? • What are some of the challenges? • What are some best practices? • What tools does IV&V support?

12. What is static code analysis? • Static code analysis of software is performed without actually executing the programs (as opposed to dynamic analysis which is performed on executing programs) • Analysis is usually performed on some version of the source code, although some static analysis can also be performed against some form of the compiled object code (e.g. some security focused analyzers) • Automated static code analysis works hand in hand with manual analysis, which is required to review reported tool results and the relevant source code to determine if a true defect is present or not

13. Identifying Types of Code Defects Categories of code defects Syntax Logic Run-time • Static Code Analysis can contribute to identifying each of these categories of defects

14. Identifying Types of Code Defects Categories of code defects • Caught during compile time, e.g. code is attempting to use an undeclared variable • May include style or certain coding standards violations that are caught through basic parsing and “pattern matching” Syntax Logic Run-time

15. Identifying Types of Code Defects Categories of code defects Logic • May be caught during functional requirements based testing, i.e. does the code do what it was supposed to? • High value findings when able to identify or confirm logic defects through either manual analysis or use of dynamic analysis resources Syntax Run-time • IV&V analysts have found that digging deeper into understanding the code when reviewing static code analysis results can provide a side benefit of identifying other unrelated logic defects

16. Identifying Types of Code Defects Categories of code defects Logic • High value findings, as these defects may result in program crash or undefined behavior that could ultimately result in the loss of the mission or impact the accomplishment of a mission objective • Underlying tool technology (e.g. formal methods) and project configuration impacts the effectiveness of identifying certain defects • Occurs during program execution, e.g. out of bounds array access (“buffer overflow”), division by zero, non-initialized local variables, integer overflows, illegal dereferenced pointer Syntax Run-time • Static code analysis tools can help detect dangerous run-time defects in the code or even prove the absence of certain run-time defects when using formal methods-based approaches

17. Why perform static code analysis? • Complements testing and simulation (dynamic analysis) which cannot cover all paths of the program with an exhaustive set of test cases that address all possible ranges of input values to identify many errors, e.g. out of bounds array access • Improves code quality and reliability through identification of specific defects or “bugs” early in the development cycle, which can significantly reduce the cost of eliminating defects found only during test or even after release • Improves code consistency and quality through automated identification of coding standards violations (industry standards such as MISRA or custom NASA development project standards) • Scales well to large amounts of source code and can be integrated and automated as part of the build process

18. Why perform static code analysis? • Supports gathering of software metrics to help measure and track quality over time • Enables finding important defects that would be difficult to identify through other methods, e.g. IV&V analysts have identified the following using static code analysis on NASA missions • Use of formal methods based static analyzer (Polyspace) able to prove that developer provided guard code in a time critical method was unnecessary • Divide by zero problem in instrument flight software • Numerous out of bounds array access (“buffer overflow”) issues • Many other code defects that were identified and fixed by the developer on various missions

19. How do you get started? • Identify which software will be analyzed • Mission criticality • Level of assurance and verification required • Determine the scope of the static analysis • Identify any coding standards or other requirements (GOLDen rules) that are relevant to the target software • Identify any specific focus areas for analysis, e.g. “prove absence of mathematical overflow” or “remove possibility of SQL injection attacks” • Secure tool(s) or services that will best support the planned efforts

20. How do you get started? • Setup and Configure tool for usage • Install tool(s) in appropriate environment, address user access and authentication approach (e.g. could leverage LDAP) • Integrate tool within development build environment if desired, e.g. automate running of tool within the IDE or as part of a regular build process • Identify the default checks that are enabled by the tool prior to running an initial analysis build • Determine which additional checks you might want to enable for your analysis build, e.g. checking for MISRA standards compliance • Determine which checks you might want to disable for your analysis build, e.g. ignoring a standards type check that produces a significant volume of results for review • Run the tool, propagating results to support analysis build comparison where the tool provides this capability • Analyze results

21. Selecting Appropriate Tools Source code language Project software characteristics Size of code base Ground vs. Flight software Business software Characteristics of tools available Project development software quality objectives Targeted defect types Experience of the team with specific tools

22. Selecting Appropriate Tools

23. Selecting Appropriate Services • Some considerations when securing outside services or support for static code analysis • What tools does the provider intend to use? • What specific version of the tools does the provider intend to use? (this can impact the reported results) • Confirm tool version meets the needs of your project • For example, an earlier Klocwork version does not handle a particular version of the Java libraries very well • What checkers will be enabled or disabled (which defects targeted) for the planned analysis builds? • What level of review will be applied to each category of results? • Will you get the raw output results in addition to the vetted findings?

24. What are some of the challenges? • Initial learning curve and ongoing expertise required to configure and run the tools most effectively • Static analysis concepts (abstract code representation) and detailed compiler knowledge useful • Tools are updated on a regular basis, updating current checks • Customization options vary across different tools • Tools may vary significantly in their specific capabilities, with different strengths and weaknesses in finding certain types of defects • False Positive – tool identifies a possible defect that is not actually a problem when reviewed • False Negative – tool fails to identify a defect even though it is present in the source code

25. What are some of the challenges? • Maintenance overhead for multiple tools • Tools can be unique in system requirements, setup, and typical usage • Licensing varies across tools and can impact usage scenarios • Different versions of the same tool may be used across projects • Object-oriented source code can pose other challenges for static code analysis tools and the analysts due to its additional language complexity, e.g. C++, Java • Comparison of different tools against a representative set of common source code can be time consuming and difficult • Requires common source code to be created or gathered • Expertise in both the tools and the target source code is necessary to attempt any meaningful comparison of the tools • Thousands of messages can result, requiring a strategy for prioritizing manual analysis of these messages

26. What are some best practices? • Employ a multiple tool approach if resources are available, as tools are tuned in different ways and will have some unique findings • Set specific target software quality objectives and measure improvement in code quality over time • Determine which defects are of greatest importance, enabling or disabling the specific related rule checkers within the tools • Customize tools as desired to help meet the target objectives, e.g. custom coding standards

27. Maximizing Tool Effectiveness • Consider project coding standards, code constructs that are not relevant, and code constructs that should not be utilized • Consider also documenting what is NOT being checked for, if applicable • Consider running multiple builds with different sets of checks enabled • Consider enabling MISRA checks for critical flight software, but separate builds may be warranted in this instance due to the potential for a significant number of resulting messages Review the checks enabled within your tool(s) to ensure that the checks of interest for the development project have been selected

28. What tools does IV&V support? • FlexeLint (C, C++) • Full builds can be run for analysis in reasonable times • Small vendor does not update the analysis capabilities regularly • Klocwork (C, C++, Java) • Well established history of use and support at IV&V • No built-in support for important mathematical errors (e.g., divide by zero, arithmetical overflow/underflow) • CodeSonar (C, C++, Java) • Tool heritage based on JPL safety critical related work • Incomplete support for mathematical errors (ex: divide by zero, arithmetical over/under flow) • Polyspace Bug Finder (C, C++) • New tool from Mathworks vendor, separated from formal prover • Unproven tool in comparison to similar tools, e.g. Klocwork • Polyspace Code Prover (C, C++) • Provides formal verification of source code that can prove the absence of mathematical overflow, divide-by-zero, out-of-bounds array access (“buffer overflow”) and other run-time errors, as it explores all possible execution paths of the source code and tracks value ranges throughout • Steep learning curve, difficult to run against entire source code build, may need to break up build for analysis • Forcheck (Fortran) • Used in targeted instances for legacy Fortran code analysis • AdaCoreCodePeer (Ada) • Not in use currently, evaluated for possible use within the IV&V Program • Understand (C, C++, Java, Ada, Fortran, Python etc.) – Does Not Satisfy NPR • Source code browsing, basic capability to verify coding standards and conventions

29. IV&V Static Analysis Breakdown • Values are roughly representative. Every project is different. • Some tools require more preparation than others. • Not all checkers are equal, some are better at only producing valid findings • Some findings are quick/easy fixes, some require significant time to evaluate, fix, and test.

30. IV&V Static Analysis Services (1/2) • 1) Run Tool(s) and Provide Raw Results • Leverages IV&V experience setting up builds • Leverages IV&V tool licenses, servers, etc • Some tools may require a license to view the data • Still requires significant effort to triage the data set • 2) Run Tool(s) and Eliminate False Positives • Leverages IV&V experience setting up builds and reviewing results from tools • Leverages IV&V tool licenses, servers, etc. • Provides a much smaller data set to the project for analysis • Still requires some effort to verify the remaining findings

31. IV&V Static Analysis Services (2/2) • 3) Run Tool(s), Eliminate False Positives, Review True Positives, Author Findings • Leverages IV&V experience setting up builds, reviewing results from tools, IV&V licenses, servers, etc. • Does not require any tool licenses! • Provides actionable findings that are valid (very high acceptance rate) • 4) Run Tool(s), Eliminate False Positives, Review True Positives, Author Findings and Verify Closure of Findings • Leverages IV&V experience and Licenses • Provides actionable findings • Independent verification of corrective action and NASA compliance

32. Additional Resources • Additional resources • LaRS – Laboratory for Reliable Software - http://lars-lab.jpl.nasa.gov/ • Power of Ten – Gerard Holzmann - http://spinroot.com/gerard/pdf/P10.pdf • FlexeLint – www.gimpel.com • Klocwork – www.klocwork.com • CodeSonar – www.grammatech.com/codesonar • Polyspace Bug Finder - www.mathworks.com/products/polyspace-bug-finder • Polyspace Code Prover - www.mathworks.com/products/polyspace-code-prover • Forcheck - http://www.forcheck.nl/ • AdaCoreCodePeer- http://www.adacore.com/codepeer • Understand - http://www.scitools.com/

33. Next Steps • We will be available in this room from 1pm-3:30pm for detailed discussions pertaining to your projects Current Schedule • 12:00pm – 1:00pm Lunch • 1:00pm – 2:00pm SFS Demo • 2:00pm – 3:00pm EVA • 3:00pm – 3:30pm General Q&A • Please don’t hesitate to stop in! We can likely support more than one project per timeslot.

34. Questions and Further Discussion • Questions?