Background & source of vision

1. Technological infrastructural needs to support third party certificationCertification of Safety-Critical Software-Intensive SystemsFirst Public WorkshopNovember 11, 2011Sushil BirlaDivision of EngineeringOffice of Nuclear Regulatory Research(301-251-7660, Sushil.Birla@nrc.gov)

2. Background & source of vision Context: U.S. Govt. Inter-agency coordination activities • NITRD (Networking & IT R&D) • HCSS (High Confidence Software & Systems) • Cyber-physical systems • Focus area: Safety critical systems Sectors Health Energy Defense Transportation National Security Commonalities

3. Current state – some commonalities • Safety-critical CPSs are typically too complex to be completely verified and validated. Remaining uncertainties are significant, but not well understood. • Safety analysis and evaluation require high competence and judgment, but these capabilities are very scarce. • Cyber adversaries’ ability to develop and launch new attack tools and techniques outpaces the ability to develop and deploy countermeasures. • The competencecomplexity gap is widening rapidly. • Similar problems exist in most safety-critical, mission-critical application domains, but there is little synergy to find a common core set of underlying solution capabilities. • The requisite knowledge is not well-systematized • Commercially available tools, driven by non-critical consumer applications, are being used in critical applications, but their commensurate verification Is not feasible economically.

4. Current state: Some complexity issues • A single defect can make logic wrong, potentially leading to serious consequences, but the capability to engineer defect-free systems does not exist. • Networking (wired or wireless) introduces new vulnerabilities that are not well understood • Hidden dependencies and couplings • Latent defects could combine in many scenarios • Latent defects could cause a high consequence failure • The more complex a system the more exposure to defects • Verification of a high-integrity system or component, e.g. operating system, takes more effort and time than its initial development.

5. Vision state: Some commonalities • Systems can be routinely developed with built-in assurance of safety and security • “Do it right the first time” becomes the cheapest and fastest way to realize a system • Accredited third party services are commercially available for verification & validation (V&V) • Accredited third party services are commercially available for review, attestation, and certification • Requisite tools are certified • Requisite competence (knowledge, skills) is certified • Requisite competence becomes readily available • Requisite body of knowledge is mature and readily accessible • Educational and training institutions have mature curricula to produce and certify the requisite competence

6. 5.5 certification ISO 17000 definitions - 1 Third-partyattestationrelated to products, processes, systems or persons 5.2 attestation Issue of a statement, based on a decision following review,that fulfillment of specified requirements has been demonstrated 5.1 review Verification of the suitability, adequacy and effectiveness of selection and determination activities, and the results of these activities, with regard to fulfillment of specified requirementsby an object of conformity assessment

7. 3.1 specified requirement ISO 17000 definitions - 2 Need or expectation that is stated. NOTE: Specified requirements may be stated in normative documents such as regulations.... 2.1 conformity assessment Demonstration that specified requirements relating to a product, process, system, person or body are fulfilled 2.4 third party A person or body that is independent of the person or organization that provides the object, and of user interestsin that object

8. 2.5 conformity assessment body ISO 17000 definitions - 3 Body that performs conformity assessment services 5.6 accreditation Third-party attestation related to a conformity assessment bodyconveying formal demonstrationof its competence to carry out specific conformity assessment tasks 2.6 accreditation body Authoritative body that performs accreditation NOTE … authority … generally derived from government

9. Someexpectations &gaps Enable certification of safety-critical software 3rd party conformity assessment bodies Formally demonstrate competence Competence criteria Accreditation bodies

10. Some more gaps Standards Regulatory requirements are abstract Interpret Regulatory guides Concrete derived requirements missing/incomplete Expert Judgment needed Review SW in safety system Scarce!

11. Research needs identified Questions posed to expert group • What are sources of uncertainties? • What evidence do we need to reduce these uncertainties? • What are the areas that need more research?

12. Uncertainties even after best practices Residual Uncertainties? Focus of group Appendix A in RIL-1001 Assume conformity NRC’s regulatory guidance framework “Good” design practice Uncertainties and resulting size of potential fault space

13. Some sources of uncertainties • Validation of Requirements • Architecture: Complexity • Verification: Adequacy of coverage • Impact of change: Hidden/obscure dependencies • Transformation tools • Integrating/Combining evidence

14. Current review practice Perform thread audits of several requirements • Check for conformance clause-by-clause Is clause-by-clause review enough?

15. Combined effects of deviations Charles Perrow in “Normal Accidents- Living with High Risk technologies” 1984: • A major failure of a complex system is typically caused by a combination of relatively small incidents: Three Mile Island

16. Combined effects in SW • A single defect can make logic wrong • Hidden dependencies and couplings • Latent defects • Could combine in many scenarios • Could cause a high consequence failure- • The more complex a system the more exposure to defects

17. Combined effects of seemingly insignificant deviations High consequence failure of a complex system  Operators’ Action Faulty Equipment Incorrect indicator Inadequate Procedures Inadequate Design

18. Example of evidence gaps Demonstrate Uncertainties cannot combine to produce more complex uncertainties Demonstrate Independence and decoupling Demonstrate Compliance with architecture principles & constraints Inadequate criteria

19. Architecture: Complexity issues New I&C architectures overly complex 1. High degree of connectivity between two systems which are suppose to be independent 2. Safety to non-safety interconnectivity http://www.hse.gov.uk/newreactors/ri-ukepr-0002.pdf

20. Transformation tool issues • New (unknown) ways of introducing defects • Preservation of semantics

21. Integrating effect of uncertainties in software assurance V&Vresults Tools Auto test gen Auto code gen system software system Integr Test Unit Test FAT Arch Arch I Reqmts Reqmts D ? ? ? ? ? ? ? ? ? Safety demonstration in the presence of uncertainties Change Impact Analysis ? Each anomaly or uncertainty by itself seems to be small

22. V&V: Adequacy of coverage Some major sources of uncertainties • Environment • Assumptions • Input validity • Requirements • Correct? • Complete? • Consistent? Incomplete coverage Interference Proof of non- interference Coverage evidence (Diverse complementary) Analysis Safety Demonstration (e.g. assurance case) Model checking  Testing - Coverage based Evidence about other uncertainties …

23. Safety demonstration: Adaptation of Toulmin’s model Backing, e.g., theoretical or causal model basis for Inference rule used in Assertion/ Belief/Claim Evidence/ Grounds Argument Qualifiers (Strength; Condition) Factors influencing validity of argument affects Challenges; rebuttals; inconsistencies

24. Recap: NRC areas of interest Certification infrastructure needed • Accreditation bodies • Competence criteria • 3rd party conformity assessment bodies Some gaps in assurance technology infrastructure • Validation of Requirements • Architecture: Complexity • Verification: Adequacy of coverage • Impact of change: Hidden/obscure dependencies • Transformation tools • Integrating/Combining evidence