Use of a Technical Reference in NASA IV&V

1. Use of a Technical Referencein NASA IV&V T. Dawson, TASC 9/11/13

2. Abstract Providing mission assurance for NASA systems, and other customer systems under development, requires documentation of the system under evaluation and the standards and criteria against which the system is evaluated. These two categories taken together are called the Technical Reference, and are being used at NASA IV&V to support evidence-based assurance of customer systems. This presentation describes the intent of the Technical Reference, describes typical contents and usage, and present example products in use. Use of a Technical Reference in NASA IV&V

3. Motivations for the Technical Reference

4. What is an IV&V Technical Reference? * • “IV&V’s understanding of the system needs is developed in parallel with other IV&V activities and referred to as IV&V’s Technical Reference” • “… These two sources, IV&V’s Technical Reference and developer artifacts, are ultimately used in combination with the analytical processes to establish the needed evidence to determine whether or not the system’s software is going to satisfy the goals of the system.” • “IV&V Technical Reference is the collection of data and knowledge regarding IV&V’s independent understanding of the system’s software. The Technical Reference serves as the basis for IV&V analysis. This information includes but is not limited to system goals and needs, software interactions amongst system design elements, normal and abnormal behaviors and conditions of the system’s software and the operational environment.” * Excerpts from the IV&V Services Contract Statement of Work

5. Why do We Need One? • As “objective evidence to either confirm or deny that the software artifacts are correct and complete” • “The IV&V’s Technical Reference is one source for identifying how the system’s software should behave or perform.” • IV&V uses “the IV&V Technical Reference in conducting analysis and documenting the evidence needed to conclude whether the IV&V Project’s artifacts are sufficient or where there are limitations/risks/issues.” • In short: as one component of providing evidence-based software assurance, a primary goal of the NASA IV&V Facility

6. Additional Benefits of the Technical Reference • Enhances communications: as the foundation for IV&V analysis, it is natural and desirable for the Technical Reference to be used in communications of IV&V results with the developers, IV&V management, and other stakeholders • The Technical Reference also captures IV&V understanding of particular aspects of the system/software that can be reused for later missions that are inheriting or reusing systems, architectures, technologies, software, etc. • Part of the IVV& project documentation that lives on beyond the project

7. Characteristics of the Technical Reference

8. Characteristics of Technical Reference Content • Is a knowledge repository that contains IV&V’s understanding of the system • This understanding is synthesized in part from developer artifacts, but does not contain project the artifacts themselves • Contains specific, project-relevant criteria for performing the IV&V analyses • Contains IV&V-developed and outside (not development project) standards, evaluation criteria, and references

9. Characteristics of Technical Reference Content (cont.) • Documents the materials IV&V used in performing analysis and making assurance claims • Has a strong tie to Evidence-Based Assurance, but does not contain the evidence itself • IV&V issues and risks are not part of the Technical Reference. These items are two types of IV&V products that the Technical Reference supports producing

10. Technical Reference Over Time • Content is elaborated and updated as the IV&V project progresses • Individual components should be elaborated or updated as appropriate based on project needs and current understanding • “Creating the Technical Reference” is not a standalone task in its own right at the beginning of the project • The Technical Reference is defined for a given project, but content varies from activity to activity • A target of one analysis could form part of the basis for the Technical Reference for a follow-on activity

11. Specific Technical Reference Content Examples • IV&V’s understanding of the system as contained in IV&V-developed products: • Heritage Report • System Goals/System-Level Behaviors • IV&V Scoping Products • Context diagrams created by IV&V • Any Diagram generated to capture IV&V’s understanding of requirements/ behaviors • Use Cases generated by IV&V to capture understanding of the system/software capabilities and behaviors • Scenarios developed by IV&V to support analysis

12. Specific Technical Reference Content Examples (cont.) • Project-specific criteria for performing the IV&V analyses: • List of Adverse Conditions • List of Undesired Behaviors • Required interface characteristics derived from an ICD • IV&V-developed references/criteria: • Architecture analysis assessment criteria • Domain specific questions derived by IV&V • Outside references/criteria • MIL-STD-1553 standard for use in evaluating the project’s 1553 flight software

13. Technical Reference in the IV&V Process

14. Technical Reference’s Role in Performing IV&V Analysis Other Project Artifacts IV&V Analysis References to project artifacts Evidence Target Artifacts Technical Reference IV&V Sources Supplemental Representations Outside Sources Criteria Methods

15. Components of the Technical Reference Process • Project artifacts (synonymous with developer artifacts) consist of Target Artifacts and Other Artifacts • Target artifacts are the artifacts being evaluated in a given IV&V activity and should not form part of the basis for the Technical Reference for that IV&V activity. • Target artifacts can serve as one source for Technical Reference content. One example is a flight software requirements document that is used to build a representation of the system design that is then used to confirm consistency and other aspects of the system • Other project artifacts support the IV&V activity and provide one source for IV&V’s understanding of the system. The Technical Reference contains references to elements of these other artifacts as appropriate • IV&V and Other Sources comprises two sets of inputs to the technical reference. IV&V sources include whites papers, prior IV&V project findings, and anything created by IV&V as a reference for performing IV&V that applies to the given project but was not generated solely for use by the given project. Other Sources include similar inputs sourced from outside IV&V, to include IEEE and other standards and technical societies, and sources of scientific, engineering, software, aerospace, subsystem and other domain knowledge

16. Components of the Technical Reference Process (cont) • Supplemental representations are created by IV&V as necessary to capture IV&V’s understanding of the system • Criteria are used for the IV&V analysis. In order to determine sufficiency, some scale must be applied. This can vary dramatically from target to target and by IV&V Method being used, and these criteria form part of the Technical Reference • IV&V analysis consists of the IV&V analysis activities that comprise the majority of the project team’s work. A number of considerations, including the activity goals and the IV&V Catalog of Methods, are used to select the specific IV&V activities to be performed. The purpose of the IV&V activities is to produce evidence • Evidence is part of the Assurance Case approach to providing Evidence-Based Assurance and supports the Assurance Case Argument. Evidence is collected during the IV&V analysis activities

17. Documenting the Technical Reference • The needed content should be documented in IV&V process descriptions (currently underway) • A given project’s Technical Reference is defined and cataloged in the Master List • Content is activity-specific

18. Master List Fields • Filename • Location (path) • Activity for which intended • FY • Activity Number • Activity Name • Role in Activity • Content Description • Source of Content • Version/Date • Comments

19. Impact on IV&V Analysis

20. Impact on IV&V analysis • All of the inputs are used in “traditional” NASA IV&V • As such, there is very little impact on IV&V analysis with the introduction of Technical Reference concepts • Any impact on the analyst or specific analysis task should primarily be a positive (beneficial) by providing reference materials and analysis criteria

21. Impact on IV&V analysis (cont.) • Q: If it doesn’t change how we do things, why do we need it? • A: This is a structured approach to IV&V planning and execution • Provides us a way to explicitly account for these aspects during our project/task planning process

22. Going Forward

23. Technical Reference Evolution Recent Near-Term Mid-Term Long-Term Technical Reference Mode Umbrella Concept ECM-Based Project-Profile Based IVVO Data Mgmt-Based • Partially complete • Varies by project • Not fully integrated in processes • Single, defined location • Document-based • Potential for redundant products • Contained within Project Profile • Allows various views of the information • Mix of data elements and documents • Ultimate goal for IVVO data • Multi-views of same info • Fully dynamic content • Project Profile is a client application Technical Reference Characteristics Parallel Relevant Efforts IVVO Data Management Concept → Implementation SWAT ODS → Data Warehouse AMF CD → Implementation Now