Model-Based Requirements Validation

1. Model-Based Requirements Validation Validation Workshop September 8, 2008 Lisa Nicklow

2. Agenda • Introduction • Project Overviews • General Validation Approach • Project Approaches • IV&V Tool Usage • Results • Lessons-Learned

3. Introduction • Validation is… the process of evaluating artifacts to ensure that the right behaviors have been defined in the artifacts. The right behaviors are those that adequately describe what the system is supposed to do, what the system is not supposed to do, and what the system is supposed to do under adverse conditions. Validation ensures that the software system performs to the user’s needs under operational conditions. (IVV 09-1- http://www.nasa.gov/ivv/ims/slps/index.html)

4. WBS 1.2 Validate System Requirements

5. Project Overviews - JUNO • Salient Features • 1st solar-powered mission to Jupiter • 8 instrument payload to conduct gravity, magnetic and atmospheric investigations • Single polar orbiter (simple spinner) launches in August 2011 • 5 year cruise to Jupiter, JOI in August 2016 • 1 year operations, EOM via de-orbit into Jupiter in 2017 • Elliptical 11 day orbit swings below radiation belts to minimize radiation exposure • Key Juno partners: SwRI, JPL, ASI, LM-Denver andGSFC • Science • To improve our understanding of the solar system by understanding the origin and evolution of Jupiter, Juno will: • Determine the global O/H ratio (water abundance) in Jupiter’s atmosphere • Measure latitudinal variations in Jupiter’s deep atmosphere (composition, temperature, cloud opacity, and dynamics) • Map Jupiter’s magnetic and gravitational fields • Characterize Jupiter’s polar magnetosphere and aurorae

6. Project Overviews -JWST • Salient Features • JWST is a large infrared telescope with a 6.5-meter primary mirror.  • Launch is planned for summer 2013. • JWST includes four science instruments: Mid-Infrared Instrument (MIRI), Near-Infrared Spectrograph (NIRSpec), Near-Infrared Camera (NIRCam) and the Fine Guidance Sensor (FGS). • JWST's instruments will be designed to work primarily in the infrared range of the electromagnetic spectrum, with some capability in the visible range. It will be sensitive to light from 0.6 to 27 micrometers in wavelength. NASA is partnering with the European Space Agency (ESA) and the Canadian Space Agency (CSA). • Northrop Grumman Space Technologies is the prime contractor and the Space Telescope Science Institute will operate JWST after launch. • Science • First Light & Reionization seeks to identify the first bright objects that formed in the early Universe, and follow the ionization history. • Assembly of Galaxies will determine how galaxies and dark matter, including gas, stars, metals, physical structures (like spiral arms) and active nuclei evolved to the present day. • Birth of Stars & Proto-planetary Systems focuses on the birth and early development of stars and the formation of planets. • Planetary Systems & the Origins of Life studies the physical and chemical properties of solar systems (including our own) and where the building blocks of life may be present.

7. Project Overviews - Orion • Crew Exploration Vehicle to replace Shuttle • Capable of ISS Missions • Transport 6 crew members, max • Stay time of 210 days • Orion returns crew to earth • First Mission: 2014 • Lunar Mission • Transport 4 crew members, max • Uninhabited during lunar surface operations (up to 7 days) • Orion returns crew to earth • First Mission: 2020 • Orion System Elements • Launch Abort System: emergency escape during launch • Crew Module: Crew and cargo transport • Service Module: propulsion, electrical power, fluids storage • Spacecraft Adapter: structural transition to launch vehicle • IVV Key Partners: • Lockheed Martin, Honeywell, KSC, JSC • IVV: Started Summer of 2007 and Project PDR TBD

8. General Validation Approach

9. Validation Approach - JUNO

10. Validation Approach - JWST

11. Validation Approach - Orion

12. IV&V Tool Usage - Non-Integrated Environment

13. IV&V Tool Usage - Non-Integrated Environment Project Issue Tracking System (PITS) Model Reference Notes Use Case Notes Validation Database

14. IV&V Tool Usage - Non-Integrated Environment Observations against requirements Correlate Requirements Classify the behavior Validate / High quality - traceability

15. IV&V Tool Usage – Integrated Environment

16. Results • Incomplete requirements at Mission and Segment level addressing: • Behaviors associated with Mission Critical Events • Adherence to operational constraints and restrictions • Continuous transmission of telemetry during all mission phases • Operational constraints for parallel operations • Incomplete and inconsistent traceability of Mission, Segment, and Element level requirements • Ambiguous mission requirements • Conflicting requirements • Inconsistent requirements • Ambiguous Element requirements • Incomplete interface requirements specifying status of executed commands and fault notification • Inconsistent and incomplete specification of Element level requirements • Incomplete/incorrect tracing/flowdown of requirements

17. Lessons Learned • Centralized repository will facilitate analysis • Integrated Environment: Together & Integrity • Workflow Mgmt: distributed teams, multiple contractors • Establish Relationship with Modeling Team • Greater understanding of SRM • Establish correct behaviors in SRM • Training in the basics of UML and practical examples was extremely helpful and improved the team’s efficiency and productivity. • Early involvement in analysis of requirements. • Identification of system level issues: concepts, requirements, design/architecture • Identification of problematic sub-systems • Overall better understanding of the system • Peer Reviews • Large team peer reviews were not productive. Smaller group peer reviews with some consistent reviewers proved to be the best approach. • Important for reviewers to provide comments before group session. • Model for a purpose • SRM should be elaborated to the extent possible and used to validate complete, correct, and consistent requirements have been defined from top to bottom for each thread and the associated behaviors. • At higher levels of abstraction, formalism is unnecessary • At lower levels of abstraction, more formalism may be necessary • Model just what you need to model • Tendency to get bogged with detail • Generally model a half level less abstract than target

18. Lessons Learned • Model a little, validate a little • Avoid backlogs of un-validated models • Correspondence theory – avoid model divergence • Modeling for Validation • Source documents (e.g., OpsCons) should be parsed at the start to facilitate identification of Ops Threads, behaviors, constraints, etc. • Traceability links should be established from the source documents to the specific SRM products. • Essential to have a standardized Use Case template and guidelines for developing Activity Diagrams • Identify Operational Environment conditions early and assess impacts on system behaviors to ensure protections and constraints are captured in the SRM. • Use Case Diagrams should be developed following identification of the Ops Threads • Develop Use Cases for Contingency Ops and global behaviors first • Apply a prioritzation/focusing assessment process early to ensure the appropriate IV&V tasks are performed and completed for the most critical areas of the system while staying in phase with the Project's development lifecycle. • Configuration Management of SRM products is a must and a status report some be maintained on SRM product.

19. Back-Up Slides

20. What the System is Not Supposed to Do Considers Bad Actor Perspective Goal is to Invalidate Misuse Cases Parallels Fault Tree Concept JUNO - Misuse / Failure CasesAnswering Question #2

21. JWST – Validation Process • Step 1: SRM Development Approach/Process • Review source materials and extract the Mission/System Needs, Goals, and Objectives • Identify and Define Operational Threads and associated goals for each thread • Parse source materials and extract information on the following: • Behaviors: desired, preventative, and responsive • Behavioral Constraints • Undesired Behaviors • Adverse Conditions • Map behaviors, behavioral constraints, undesired behaviors, and adverse conditions to the set of Operational Threads • Develop the appropriate UML products to form the SRM at the applicable level of abstraction/decomposition

22. JWST – Validation Process • Step 2: Classify Behaviors • Review the behaviors and identify whether the behaviors require/drive System Safety or Dependability and if so, then identify the specific aspect of Safety (e.g., fault avoidance, fault warning, fault correction, fault tolerance, etc. ) or Dependability (e.g., recoverability, availability, robustness, reliability, etc.) • Review the behaviors and identify whether the behaviors are: • Desired:Behaviors that must be allowed/realized to satisfy mission goals/objectives. • Preventive: Behaviors that are specified/required to prevent undesired behaviors from occurring. The preventative behaviors will/can be modeled as Main Scenario Steps (MSS). • Responsive: behaviors that must exist to address/respond to possible, anticipated, or expected faults. • Undesired: A behavior that you never want to be realized. System must be developed to prevent the undesired behavior from ever happening. IV&V must ensure that requirements are defined to prevent the undesired behaviors from occurring.

23. JWST – Validation Process • Step 3: Correlate Requirements • Using the SRM products, review the target requirements set for requirements that specifically address the modeled behaviors and constraints • if parent requirements were used as a source for SRM development then the Project’s requirements traceability matrix can be used to assist in the identification of applicable requirements • Create a correlation matrix showing the relationship of the requirements to the SRM elements • Step 4: Validate Correct, Complete, and Consistent Requirements • Ensure the complete, correct, and consistent set of requirements have been defined to address each behavior or constraint • Compare correlated set of requirements to total set • Review uncorrelated requirements to determine if there are any behavioral requirements that were not correlated (could show inconsistencies between source materials and requirements or deficiency in SRM) • Ensure the correlated requirements address all dependability and safety aspects that were identified

24. JWST – Validation Process • Step 5: Validate Traceability of Requirements • Performed in support of validating that the correct, complete, and consistent set of requirements have been defined at each level • Using the Project’s traceability matrix, ensure that the correlated requirements are traceable to the source materials used in developing the SRM • Ensure the parent requirements are completely, correctly, and consistently decomposed at the target requirements level to drive implementation of the critical and high risk System “behaviors” required to meet the mission needs, goals, and objectives. • Step 6: Validate High Quality Requirements • Ensure each requirements set correlated to a behavior meets the criteria for a High Quality requirement • Unambiguous • Correct • Complete • Consistent • Verifiable

25. JWST Screen shot #1SRM Dev: Parse source requirements

26. JWST Screen shot #2Validation step1-Classify Behaviors

27. JWST Screen shot #3Validation step1-Classify Behaviors

28. JWST Screen shot #4Validation step1-Classify Behaviors

29. JWST Screen shot #5Validation step1-Classify Behaviors

30. JWST Screen shot #6Validation step3-Correlate Requirements

31. JWST Screen shot #7Validation step3-Correlate Requirements

32. JWST Screen shot #8 Observations against requirements Correlate Requirements Classify the behavior Validate / High quality - traceability

33. Issues Observations Resulting from validation 3 Questions For validation JWST Screen shot #9RTAT

34. JWST Screen shot #10RTAT MSS steps Correlated requirements

35. Orion - Manual Requirements Analysis • Requirements evaluated using pre-determined criteria

36. Orion - Manual Requirements Analysis • Individual requirements were classified according to type: • Functional:Represent the way in which the internal system components interact with their environment. Present a precise set of properties or operational needs that the system must satisfy. • Product:Result from the need for the delivered product to behave in a particular way. May be derived directly from user needs. • Organizational:Consequence of organizational policies and procedures. May be derived from the customer’s and developer’s organizations. • External:Arise from factors external to the system and its development process. This includes interoperability requirements, legislative requirements, and ethical requirements.

37. Orion - Requirements Validation with SRM • Validation performed using Orion SRM • Model was independently developed by the IV&V Orion Modeling Group • Model represents the ISS Crewed Design Reference Mission • Behavior Requirements: Inherent capability for mapping to SRM (Behavior) • Behavior requirements were mapped to 1+ model elements that represent the behavior described in the requirement • Evaluation between model elements and mapped requirements • Model elements missing requirements identify behavior that is not addressed by the requirements (potential missing requirement issue) • Requirements with no model element identify behavior not addressed by the model • Model is incomplete (possible model issue) • Extraneous functionality not contemplated by higher-level Orion documents (possible requirements issue)