Considerations in the Preference for and Application of RTCA/DO-178B in the Australian Military Avionics Context

1. Considerations in the Preference for and Application of RTCA/DO-178B in the Australian Military Avionics Context SQNLDR Derek Reinhardt Systems Certification and Integrity (SCI) Directorate of Aircraft Engineering (DAIRENG)

2. Overview • Introduction to criticisms of RTCA/DO-178B • Background and structure of DO-178B • Criticisms that RTCA/DO-178B is insufficient • Criticisms that RTCA/DO-178B is too onerous • ADF’s preference for RTCA/DO-178B • Application issues of RTCA/DO-178B in the Australian Military Avionics Context

3. Introduction • RTCA/DO-178B is the centre of much debate or criticism – insufficient, too onerous, etc • Avoidable software failures have already been responsible for aircraft mishaps • cockpit displays go blank • engines throttle back during takeoff • contradictory airspeed readings • This presentation/paper • examines these criticisms • how do they influence the ADF’s preference for and application of RTCA/DO-178B

4. ADF Preference for RTCA/DO-178B • RTCA/DO-178B is the ADF’s preferred software assurance standard for safety critical and safety related airborne software development • Refer AAP7001.054 Airworthiness Design Requirements Manual • ADF recognises the FAA processes and standards • widely used and accepted by many international airworthiness authorities • Many military aviation systems have a civil heritage with software developed under RTCA/DO-178B • AEW&C B737, MRTT A330, etc

5. Background of RTCA/DO-178B • 14 CFR 25.1309 • often referred to as FAR 25.1309 • SAE ARP 4754 and SAE ARP 4761 • aircraft level FHA, FTA • system level FMEA, FTA, etc. • common cause analysis – PRA, ZSA, CMA • RTCA/DO-178B for software design assurance • RTCA/DO-254 for hardware design assurance • Five failure condition severities are assigned design assurance levels (DALs) • Catastrophic (A), Hazardous / Severe Major (B), Major (C), Minor (D), No Effect (E)

6. Structure of RTCA/DO-178B • 66 objectives in 10 tables • + several implicit objectives • satisfaction tailored by software level • several prescriptive objectives • statement coverage, decisions coverage, modified condition decision coverage • Lifecycle phases • planning, development, requirements, design, coding and integration, verification • Integral processes • configuration management, quality assurance • Certification Authority liaison • RTCA/DO-248B, FAA Order 8110.49, CAST5

7. Intricacies of DO-178B • Common misconception that RTCA/DO-178B completely specifies the process and activities • Yes – objectives are coupled to the software lifecycle • Yes – they don’t distinguish between requirements • functional, software safety, etc • Fidelity of the objectives presents challenges for COTS and PDS • With exception of 3 objectives – flexibility is permitted in how the objectives are satisfied • Objectives can be broadly classified as contributing to requirements validity, requirements satisfaction, and requirements traceability

8. Criticisms of RTCA/DO-178B • Divided into several positions • those that believe RTCA/DO-178B is insufficient • academics, researchers, or consultants • those that believe RTCA/DO-178B is too onerous • development contractors with cost and schedule driven imperative • Criticisms are at odds with each other • central to the criticisms, then is it about right? • Widely accepted by FAA, EASA • NTSB reports that it is effective in those contexts • How does the ADF address the criticisms?

9. RTCA/DO-178B is Insufficient • Absence of mandatory formal methods • Absence of mandatory static code analysis • Ineffectiveness of MC/DC testing • Absence of specific requirements or objectives relating to software safety analysis and software safety requirements • Assumptions underlying the design assurance level definition are questionable

10. Absence of Mandatory Formal Methods • Does not prohibit formal methods • acceptable method to satisfy objectives • Application to problem domain • not universally applicable to problem domains and technologies used in critical systems • only partially applicable to problem scope • are closed languages, no inherent real world meaning, natural language is still required • Formal methods and safety • does not address significant sources of error WRT the safety of systems • little evidence that it would prevent a number of mishaps attributable to software • Complementary to testing • testing is required to demonstrate real world behaviours, on real hardware, in the target environment • Formal methods is not the ‘silver bullet’ for safety software

11. Absence of Mandatory Static Code Analysis • Does not prohibit static code analysis • it is an acceptable method used to satisfy objectives • Static analysis won’t find all the faults (requirements and implementation) most related to the safety of the software • Permits greater focus on those activities related to identifying requirements validity and satisfaction problems affecting safety • prevents developers being overwhelmed in code reviews and testing identifying inadvertent implementation problems, which static code analysis tools readily detect • Limitations to applying static analysis retrospectively

12. Ineffectiveness of MC/DC Testing • Exercise each data flow that directly affects a control flow to identify as many faults as possible • Widely debated objective • studies confirm that MCDC does find faults that other DO-178B testing approaches do not find • other studies found “no significant difference” • RCDC address some of these limitations • MCDC not finding additional faults is not the concern • if normal and robustness testing has been comprehensive, then it is possible that the gap in MCDC will be small, and NOT safety related • adequacy of normal and robustness testing

13. Absence of Specific Requirements or Objectives Relating to Software Safety Analysis and Software Safety Requirements • Is not explicit in objectives relating to software safety analysis and software safety requirements • but they are not absent! • number of objectives contribute to requirements validity, including that of safety requirements • However, systematic software safety analysis are not always proposed to show that the identified and allocated set of software safety requirements is complete and correct • DGTA recommends an IEEE1228 Software Safety Plan be used to document the planned software safety activities – and their outcomes • or the RTCA/DO-178B PSAC can be used

14. Assumptions Underlying the Design Assurance Level Definition are Questionable • Issues with integrity/assurance levels • little evidence that software of different integrity levels does have failure rates of integrity level order • debate regarding the philosophy and rules for allocating integrity levels • significant differences in the processes recommended by standards • Inconsistent application • misunderstanding of the differences between reviews versus analyses • some objectives being presumed to be satisfied solely by reviews • intent is combination of analysis and reviews of the outputs • variable • normal and robustness testing • software architecture is appropriate • avoids design constructs that would not comply with system safety objectives • software safety requirements are identified/allocated • Apportionment and adequacy of objectives • objectives fundamental to the validity and satisfaction – from Level C • Level A and B provide additional evidence - trustworthiness

15. RTCA/DO-178B is Too Onerous • Excessive requirements specification and traceability fidelity requirements • Excessive verification requirements

16. Excessive Requirements Specification and Traceability Fidelity Requirements • RTCA/DO-178B motivation for fidelity • each behaviour that constitutes the requirements at level of abstraction is systematically accounted for • design tool to assist developer produce a good design • Why should all the behaviours be accounted for? • evidence that all behaviours of the software are acceptable • do not lead to unacceptable failure conditions • all the behaviours of the software should be disclosed • permits reasoning about their suitability for the safety of the system • arguing non-interference with the behaviours that are important to the safety of the system • Questionable disagreements • Intellectual Property constraints • software development is a creative process, not itself compatible with such rigour requirements

17. Excessive Verification Requirements • Testing will always be required to gain confidence in the behaviour of software on the target hardware in the intended operating environment • Completion of testing • defensible engineering argument as to why testing is complete • with evidence to support it • Not based on the following factors: • cost and schedule • consensus of program managers • broad consensus of the programmers and testers • any other non-engineering based arguments • RTCA/DO-178B provides a defensible argument as to when testing is complete by specifying: • requirements completeness criteria • requirements coverage criteria • extent of normal and robustness tests • extensiveness of requirements based low level testing • coupled with additional implementation related coverage criteria (structural coverage) to elicit certain properties

18. Application of RTCA/DO-178B • RTCA/DO-178B is not applied in isolation • Test coverage objectives • Use of RTCA/DO-178B as a benchmark for assessing software assurance practices • COTS with RTCA/DO-178B • Migrating to RTCA/DO-178B

19. Not Applied in Isolation • System Safety Program - MIL-STD-882C or FAR 25.1309 • Key Issues • A Software Safety Program (SwSP) should be established to coordinate hazard identification and mitigation efforts for hazards with software-related causal factors. • IEEE1228 Software Safety Plan • Software Safety Analysis, Generic Software Safety Requirements • A Software Assurance standard is required for the development of all software that is safety related • Software process standards should be applied to the development of software for airborne and related systems • An assessment of the applicant’s software development capability, including domain knowledge, should be conducted as part of the tender evaluation and contract negotiation process • examines the safety culture • determine if non-systematic (experience) activities can be trusted

20. Test coverage objectives • Some organisations believe that DGTA does not require structural code analysis – by default – THIS IS NOT TRUE! • In cases where DGTA has not required it • additional activities to ensure that the coverage is comprehensive • fidelity of requirements • explicitness of traceability • extent of normal and robustness testing • additional activity to identify dead and deactivated code • often exception related code • mission systems, with no series hazards • extenuating circumstances – legacy constraints • Negotiate through the PSAC –expect to have a compelling argument if you are going to proposed not doing it

21. Software Assurance Benchmark • Assess software products and their development practices • software agencies that do not use RTCA/DO-178B • confused that DGTA is applying DO-178B where not contracted • RTCA/DO-178B objectives help with the assessment of • requirements validity, satisfaction, and traceability • RTCA/DO-178B provide clear measures of • fidelity in the specification and traceability of requirements • no gap between required behaviours and executable code • all software behaviours are appropriate with respect to safety • extent of test based verification of requirements and implementation on the target computer in the intended environment • development and review rigour • independence and oversight • assures that the evidence presented contains an acceptably small number of faults

22. COTS with RTCA/DO-178B • COTS fall under the scope of PDS • PDS criteria are demanding, but not excessive • but often the evidence is just not available • Alternate proposals for use of COTS • Provide evidence to demonstrate the following types of properties • Partitioning (containment and/or mediation) • Non-Interference • Acceptable Behaviours

23. Migrating to DO-178B • Acquired where no software assurance standard applied • DOD-STD-2167A, MIL-STD-498, IEEE12207 • Two options – transition to RTCA/DO-178B or develop a Software Assurance Task Matrix • FAA guidelines • Notice 8110.49 Chapter 10 • Intended for systems developed to RTCA/DO-178 and RTCA/DO-178A • Careful management of the scope of retrospective production of software assurance evidence is required • DGTA has assessed the approach as acceptable

24. Summary • Introduction to criticisms of RTCA/DO-178B • Background and Structure of DO-178B • Criticisms that RTCA/DO-178B is insufficient • Criticisms that RTCA/DO-178B is too onerous • ADF’s preference for RTCA/DO-178B • Application of RTCA/DO-178B in the Australian Military Avionics Context • RTCA/DO-178B applied within the ADF framework addresses relevant criteria for producing safety software systems

25. Questions ?