1 / 30

Formal Methods Tool Qualification for DO178B & DO178C

Formal Methods Tool Qualification for DO178B & DO178C. Nick Tudor tel: +44 1684 894489 email: njtudor@qinetiq.com. Agenda. Tool Overview and Qualification Approach The Current Guidance - DO178B Qualification Considerations – DO178B Applicant’s Claims – DO178B

thane
Download Presentation

Formal Methods Tool Qualification for DO178B & DO178C

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Formal Methods Tool Qualification for DO178B & DO178C Nick Tudor tel: +44 1684 894489 email: njtudor@qinetiq.com

  2. Agenda • Tool Overview and Qualification Approach • The Current Guidance - DO178B • Qualification Considerations – DO178B • Applicant’s Claims – DO178B • Qualification Considerations – DO178C • Applicant’s Claims – DO178C and Supplements • Summary

  3. Tool Overview and Qualification Approach

  4. Overview of the tool • The tool is a Formal Methods based software verification tool • Automates the verification of automatically generated code • Constraints include a subset of Simulink and a subset of the Ada programming language • Exploits the Z formal specification language • Automatic generation of formal specification of the Simulink in Z • Gives required independence • Uses Carroll Morgan’s Theory of Refinement, also automatic • Built in automated use of an off-the-shelf Theorem Prover (ProofPower) • Having used the tool manually for independent code verification, we aimed to: • Automatically prove automatically generated code for productivity reasons • Understand the tool qualification issues

  5. Development Ada Verification Z Verification Conditions Discharge proof Overview of the CLawZ Process B4S Simulink Z Producer User Interface Refinement Script Generator Compliance Notation Tool Supertac ProofPower

  6. Importing Simulink & Ada files & defining units The GUI and some functions (PID example) Creating Interface for each unit Defining variables Creating Z specification, doing refinement and proof Proof results

  7. The ‘pre-qualification’ approach • As part of the final development steps, we took advice from CAA • This is new because: • No-one has previously approached them to do any form of “pre-qualification” of a tool • NB This does not constitute certification authority pre-approval of this tool • It’s a formal methods tool • A formal methods tool which is intended to only automate code verification [and will be qualified as such] • We needed to examine the guidance from DO178B to ensure that we knew what we were aiming at and to check likely impact of DO178C • Particularly in light of both the Formal Methods and Tool Qualification Supplements

  8. How high is the bar? RTCA DO178C/EUROCAE ED12C Tool Qualification Supplement RTCA DO178B/EUROCAE ED12B “Plus” RTCA DO178B/EUROCAE ED12B Minimum QinetiQ Internal Processes

  9. Theoretical Applicant Theoretical DO178C A virtual, parallel approach Tool Development Theoretical System DO178B Includes (theoretically): Core text, Tool Qualification, FM Supplement MBD Supplement

  10. The Current Guidance - DO178B

  11. Not Shooting for the Minimum • RTCA DO178B states that for a Verification tool: • “The qualification criteria for software verification tools should be achieved by demonstration that the tool complies with its Tool Operational Requirements under normal operational conditions” • Production of Tool Operational Requirements, the Verification Plan, the Verification Results and the Tool Accomplishment Summary • NB FAA-8110-49 says you could also have a TQP and TAS, but specifically: • “Tool Operational Requirements satisfies the same objectives as the Software Requirements Data of the airborne software”. …ie (from 12.2.3.2): • “Tool Operational Requirements describe the tool's operational functionality. This data should include: • a. A description of the tool's functions and technical features. [For software development tools, it includes the software development process activities performed by the tool]. • b. User information, such as installation guides and user manuals. • c. A description of the tool's operational environment. • d. [For software development tools, the expected responses of the tool under abnormal operating conditions.] ”

  12. Adapted from FAA-8110-49 Fig 9-2

  13. Qualification considerations – DO178B

  14. An Applicant’s perspective • We needed to consider how an Applicant would use the tool and subsequently be able to claim credit • Despite confused guidance, what artefacts would need to be made available • Given that this is a tool based upon FM, what “language” to use (this is non-trivial!) • We needed to be cognisant of context, ie the overall software development life cycle • We have defined CLawZ Simulink Subset (CSS) used for Independent Verification and used a well known subset of code (Ada) • This constrains the possible input space to something that is well defined, but also very flexible • Also constrained the output space (code) • By doing this we set the Tool Operational Requirements and hence the scope of the qualification

  15. Upgrade/production of Plans Configuration Control Tool OperationalRequirements Configuration Index Verification Plan Verification Cases and Procedures Tool Accomplishment Summary Verification Results Applicants SAS QA and Technical Audit Available information includes: Development Plan Configuration Management Plan QA Plan/records (not for 178B) Design Description Configuration Records Development Environment Index User Guide Proposed approach History Tool Qualification Plan Applicants PSAC

  16. Applicant’s Claims – DO178B

  17. System Requirements A-3.1 Compliance A-3.6 Traceability A-3.2 Accuracy & Consistency A-3.3 HW Compatibility A-3.4 Verifiability A-3.5 Conformance A-3.7 Algorithm Accuracy (A-2: 1, 2) High-Level Requirements A-4.1 Compliance A-4.6 Traceability A-4. 8 Architecture Compatibility (A-2: 3, 4, 5) A-4.9 Consistency A-4.10 HW Compatibility A-4.11 Verifiability A-4.12 Conformance A-4.13 Partition Integrity A-4.2 Accuracy & Consistency A-4.3 HW Compatibility A-4.4 Verifiability A-4.5 Conformance A-4.7 Algorithm Accuracy Software Architecture Low-Level Requirements A-5.1 Compliance A-5.5 Traceability A-5.2 Compliance (A-2: 6) A-5.3 Verifiability A-5.4 Conformance A-5.6 Accuracy & Consistency A-6.3 Compliance A-6.4 Robustness Source Code (A-2: 7) A-5. 7 Complete & Correct A-6.5 Compatible With Target Executable Object Code Verification Objectives – DO178B NB Verification of the Verification, QA, DER liaison and other documentation alsorequired A-6.1 Compliance A-6.2 Robustness Compliance: with requirements Conformance: with standards

  18. Claims relating to source code verification Table A-5 Partial We can show that eg there are no uninitialised or unused variables or constants. There are no claims re stack usage, resource contention, WCET, etc

  19. Major Contribution Simulink [Continuous] Simulink [Discrete] Simulink for CLawZ Autocode, proof Verification Objectives System Requirements (A-2: 1, 2) High-Level Requirements (A-2: 3, 4, 5) Software Architecture Low-Level Requirements A-5.2 Compliance (A-2: 6) A-5.3 Verifiability A-5.4 Conformance A-5.6 Accuracy & Consistency A-5.1 Compliance A-5.5 Traceability Source Code (A-2: 7) A-5. 7 Complete & Correct Executable Object Code

  20. Qualification considerations – DO178C

  21. DO178B DO178C Criteria 1 ? Criteria 2 Criteria 3 DO178C introduces 3 categories of tools (see section12.2.2) Development tool Verification tool Criteria 2: A tool that automates verification process(es) and thus could fail to detect an error, and whose output is used to justify the elimination or reduction of: 1. Verification process(es) other than that automated by the tool, or 2. Development process(es) that could have an impact on the airborne software. Criteria 1: A tool whose output is part of the airborne software and thus could insert an error. Criteria 3: A tool that, within the scope of its intended use, could fail to detect an error.

  22. DO178C Tool Qualification Supplement • We wanted to know what we might be under ED12C/DO178C we we’re either • A verification tool (Criteria 3) and hence TQL5 would be required, or • A super verification tool (Criteria 2) and hence TQL5 or even TQL4 • We believe CLawZ would be a TQL5 tool, but this depends largely on what the Applicant will wish claim as a result of using the tool • And whether this is acceptable for the certifier • More on this at the end… • We will also examine TQL4 …just in case!

  23. DO178C TQL 4 – Tables T-1 & T-2 • We needed to keep and eye on what TQL4 required at the same time • Referring to Table T-1 and T2, there is considerable extra material required over DO178B • No requirement for Tool Operational Requirements, but is implicit • Interesting that QA Plan appears to be required, but not QA records • Also, that Tool Plans do not have to comply with the Supplement (T-1-6)

  24. TQL4 – DO178C – Tables T-3, T-4 and T-5 • There is large concentration of effort in Table T-3 (requirements) which is not “needed” under DO178B • It is implied that the Tool Operational Requirements are the “requirements”. • Suspect that in practice, for all but trivial tools under qualified under DO178B, there is a more clearly defined set of requirements contained within the TOR • It was not immediately clear why the only requirement for Table T4 is T-4-10 (protection mechanisms). Sought guidance from FAQ #2 Tool Qualification Supplement : • Seeks to show evidence for separation of functions, essentially through architecture – fine, but… • There are no other architecture requirements for TQL 4 (ie compatible with requirements, consistency, standards), so does this objective help safety? • No objectives for Table T-5 verification of the coding process…

  25. TQL4 - Tables T-4 and T-5 - A closer examination • From DP #2: Rationale for Tool Qualification Criteria Definition, para 2.2.3.2 Rationale for introducing Criteria 2 “These additional uses of tools raise some concerns about the rigor required for qualification. For example: • From a safety perspective, the impact of these tools on the final software can vary. • The required confidence may not be able to be assessed by considering only the Tool Operational Requirements. • The maturity and soundness of a mathematical theory and its implementation may be necessary to provide the required confidence • The major concern would appear to be the implementation, so no examination of low level requirements or the source code? • Reliance is therefore left to Table T-6… • NB No objectives are required to be met for TQL5

  26. TQL4 – ED12C/DO178C – Tables T-6, T-7, T-8 and T-9 • Focus here is on how the tool functions when actually undertaking verification • Examines object code with respect to the Tool Requirements • Checks compliance and robustness • Also looks at verification results • Checks that results were correct/discrepancies explained and • Coverage of Tool Requirements • Ensures that the configuration was identified

  27. Applicant’s Claims – DO178C and Supplements

  28. Possible claims relating to source code verification – DO178C FM Supplement • Specific claims for formal verification could be made in accordance with the Formal Methods Supplement (see table FM A-5) • Specific claims can be made FM A5-1 to 6 (FM7 refers to core text as above) • Also need to examine possible to claims to Extra Objectives FM8-11

  29. Perspective of OTS tools for DO178C vs in-house tool development for DO178B • Number of views from Tool Qualification Supplement • Re-use – see section 11.2 • OTS - see section 11.3 • Service History – see section 11.4 (not applicable) • Alternative methods – see section 11.5 (not applicable) • Section 11.3.2 shows activities required by the tool developer • Which are to produce: TOR, TQP, TCI, TAS • Also outlines which objectives are modified • Section 11.3.3 shows activities for the tool user • To assess the TOR, TQP, TCI and TAS and plan extra activities such as division of responsibilities

  30. Summary • We believe we can support an Applicant in system certification to DO178B • We have gone further than was required because this tool was “pre-developed” and is a formal methods tool • We believe that it would be a TQL5 tool under DO178C Tool Qualification Supplement, but… • Could be TQL4, depending on development processes of the Applicant • Interesting that our approach has mirrored the COTS advice to tool qualification • So does TQL4 achieve anything extra in terms of system safety?

More Related