Balancing Practices: Inspections, Testing, and Others JAXA scenario (formal method)

1. Balancing Practices: Inspections, Testing, and OthersJAXA scenario (formal method) Masa Katahira Japanese Space Agency

2. Strategy to select methods • We realize that • the correct use of particular methods, • the combination of several methods are very important. • But how? • Quality Goals • Budget Limitation • System Characteristics • Data availability, Development Phase • Methods Compliment • Inspection/Review • Testing • Formal Method • Theorem Proving • Auto Model Checking • Inspections/Reviews • Hard to cover all aspects • Testing • Not complete, too late in some case • Formal Method • TP: Complex for practical use • MC: State explosion possible

3. REA Risk Analysis (Robustness) Hazard Analysis/SFMEA Static analysis (Problem Reports) Auto Equivalency Check Compliance/Traceability Manual Check (Tools Support) Process & Quality In line Process Monitor (SMIP) Selection and Scalability of Methodologies(sample) Phases Full set Selection (Depth) Simulation Light Modeling/Model Checking Inspection/Review (Check List) Completeness/Consistency Auto Test Case Generation & Robustness Evaluation Design Coverage & Timing Interface Validation Assessment Attributes (Sample) Test Case & Test Test Result Review Verification Coverage

4. Still we don’t know how to decide methods’ selection!

5. Consideration • For projects A,B,C, there is not enough time to perform model checking. Review with check list instead. • For Project D, a checklist is useful for the data correctness and consistency of data handling system. • For Project E, the effective of model checking is confirmed due to having enough time. • Review also shows important role in erroneous description in the specification • For design phase such as C,E, model checking and a checklist help finding errors which can not be found by review. • For projects A,B,C, there is not enough time, and the review with check list shows efficiency. • For Project D, Review as well as review with checklist shows the efficiency for data handling system. • For such as Project E case, when enough time is assigned, the model checking shows good results. Fig.1 Each methods’ effectiveness among all significant issues Fig.2 Each methods’ effectiveness among all Editorial Errors Fig.3 Each methods’ effectiveness among all Significant issues and Editorial errors

6. Method Advantage Disadvantage Formal Model/ Model Checking ●It is useful to find the problem concerning the complicated state/mode transition and processing timing issues which is hard to be found by manual. ●Erroneous description in the spec. at modeling which is more effective than normal review. ●A certain mount of time are necessary to modeling and model checking. ●Need to know modeling and model checking knowledge. ●Low cost effectiveness for software which does not have complicated logic such as data handing, or transformation Review with Checklist (Inspection) ●High cost effectiveness even if there is very short time to access it. ●Erroneous descriptions are covered by check items in the list. ●It does not depend on the skill of evaluators than model methods. ●There are limited based on the items in the checklist. ●It is hard to check the detailed behaviors in the complex system and to cover the possible combination. Lessons Learned Summaryin JAXA case study

7. Boundary of Formal Method Application Safety Critical System Characteristics Complexity Available man-month Important Border

8. JAXA formal method activity

9. Needs and remaining issues of formal method [Problem Statement] • Need to assure the high reliability of spacecraft • Facing to the difficulty to prove the goodness only by test and inspection because of system complexity and safety requirements such “must not work” • Large number of defects are introduced mostly at the Requirement Phase or the Early Design Phase. Unintended or Unexpected system/software behavior is difficult to be found at the inspection/review by manual. [Challenge] • Knowledge base inspection/review is still very important, but model checking gives a chance to detect important findings which are easy to miss by the reviewer. • Modeling task itself gives a chance to think enough deeply what the specification really says as if the reviewers build the software by themselves. [Remaining Issues] • Quality of Model and model checking task • Large amount of time is spent to correct the erroneous model • Abstraction and partitioning techniques • To avoid state explosion, and missing the scenarios • Better Productivity • Hard to find real problems from thousands of auto checking results • Personnel skill

10. Modeling Natural Language Input Tools Direct Modeling Flow Diagram Tool Equivalency SMV, SPIN Uppaal Findings Findings Executable Code Operation Model Simulation (Dynamic Analysis) Test Case Proposal Consistency Task Model Tool Robustness Procedure Ope. Scenario Behavior Analysis Findings Modeling and model checking in JAXA Model Checking (Static Analysis) Requirement Model (SpecTRM*1, Uppaal) Completeness Consistency Reachability Req. Spec. ICD Hazard Report Design Model (SpecTRM, Uppaal) Design Spec. ICD Hazard Report *1:SpecTRM: Specification Tools and Requirements Methodology

11. Productivity improvement Modeling Task Cost

12. Real issues from the results of the modeling and model checking Identified issues in the specification

13. Lessons Learned from industry use of modeling and model checking • Modeling • Can organize information and execute the modeling in the brain • Identify lots of basic problems in the specification (ambiguous descriptions, inconsistency of the contents, unclear data definition) as to make the accurate model of the specification in the formal language • Automated Consistency analysis • Effective to identify the inconsistency in the requirement specification • Identify the inconsistency among the procedures in case that multiple tasks executions are allowed simultaneously in the design level. • Automated Completeness analysis • Check whether the nodes after the transition at the branch in the flowchart meet the number of the transition conditions and its contents, and whether all error handling and exceptional procedure are covered in the design level. • Formal Validation of the functional behavior using SPIN • Effective to validate whether the procedures are executed without stagnation and those behavior meet the requirements for the procedure flow in the detailed design • Effective to verify whether hazard control function/failure recovery functions are working without unintended stop in the real time

14. Questions? (Formal Method) • What is the role of formal method (Theorem Proving, Model checking etc.) in many quality practices? • When is a Formal Method necessary or efficient? • What is a Formal Method useful for? Specific Aspects? • What are most important research issues to deploy the method into real projects? Industry Needs? • What empirical data gathered at the industry will be useful to future research? • What is an expected benefit from use of formal method?

15. Backup Slide

16. Type of system Findings type and number from each methods Review Formal Modeling and Model Checking Review with Checklist System A (Controller) /Req. 0 1 0 1 1 1 0 2 2 2 1 5 Significant Editorial No Issue Sum Significant Editorial No Issue Sum Significant Editorial No Issue Sum System B (Controller) /Req. 8 4 4 16 4 2 1 7 9 2 1 12 System C (Controller)/Design 0 0 2 2 1 0 1 2 1 5 4 10 System D (DataHandling), Design 0 24 34 58 2 0 3 5 4 0 5 9 System E (Controller)/Design 2 0 27 29 3 13 8 24 2 0 32 34 Findings from each methods(Spacecraft’s Projects) Significant: Signification Issues to be modified such as incorrect or missing functions/logic/data Editorial: Editorial Errors in the specification No Issue: Non real problem (misunderstanding/modeling mistake)

17. SpecTRM (Model) Based Robustness Test Environment (SpecRobusT) Outline: • By using specification models, the important test cases are generated for full software simulation during development contractor’s test phase automatically and comparing results. • Especially, all inputs are verified in the model to generate the test cases. • Auto tests are performed at 10,000 – 100,000 cases / sec. Results of Project application: • # of Test Case ：550,870,000,000 • Benefits: • Verification at very early phase • Introduction to automated test environment • Introduce “Test Before Development” paradigm into development process Implementation Procedure