Enhancing Software Certification for Efficient Modeling, Analysis, and Testing

1. Topic 4: Effects of software certification on the current balance between software/system modeling, analysis and testing

2. Context • Traditional approach for certified software development has been to implement the system and then perform rigorous testing and analysis (post-creation) • Safety critical systems are designed to reduce this testing/analysis burden • Newer model-based approaches are intended to reduce errors and support analysis (pre-creation), but… • Certification processes require the same degree of confidence in all evidence used therein • E.g., when less direct methods are to be used for certification: • Any models, modeling tools, analysis tools, etc, must be certified to the same level as the underlying system, and • Fidelity of any system representations must be similarly certified for all applicable aspects (e.g. functionality, timing, concurrency) Certify?

3. Responses Concerned with • Applying certification criteria to larger scope of behavior • E.g. more dynamic behavior • Certification of “traditional” designs using non-system testing methods

4. Question 1 • …What techniques are available which compellingly reduce the aggregate certification effort? • (Topic 2) • Formal Methods? • (Sylvester Fernandez) • Commoditized components may help • Issues of trust and confidence, rather than technology alone, must be addressed • (Richard Payne) • Modeling and simulation (discrete event simulation) could be used to represent the behavior of the system and this simulation could be certified in relation to requirements • (Fredrick Sheldon) • SPN/SAN Models

5. Question 2 • …What unique approaches, if any, can more efficiently certify tools and representations than the system itself? • (Richard Payne) • Model-based testing – construct models of intended system behavior in the form of a sequence-based specification to validate system requirements are complete and correct. • Markov chain usage models- to generate a large sample of system software test cases of mission-based test scenarios • Model-based testing can be implemented at a relatively small cost compared to the overall software development and test budget as well as rework costs.

6. Question 3 • …What aspects are most amenable to these approaches and techniques? • Can techniques address unique challenges resulting from mobile ad hoc networking or other system of system dynamics? • (Topic 2) • Concurrency – formal methods “proof of properties” • (Joe Loyall) • Increase the ability to certify dynamic systems by constraining their dynamism • Dynamically choosing from a set of certified static choices • If we can’t certify exactly correct behavior for highly dynamic systems, perhaps we can certify their limits • Can we certify the adaptive mechanisms that delimit behavior, recover, protect, or keep software operating within a “safe” subset of possibilities • (Richard Payne) • Usage modeling process aid in identifying possible combinations and configurations of the system operating in a variety of ways and documenting them explicitly in behavioral models • It can help not only to verify that the system functions correctly under dynamic conditions but also help to determine what those conditions are • (Chris Gill) • Reliable message delivery in the face of transient connectivity by exploiting motion and availability profiles

7. Question 4 • …Are these approaches and techniques scalable to the size of FCS? • (Richard Payne) • The model-based testing approach has not been used on a program the scope of the UoA but from a modeling standpoint, it should be feasible • 105 - 106 usage states can be captured in usage models with the latest available techniques • Major challenge is providing required test capabilities for large scale, automated testing • Without a robust, semi-automated test capability, it will be very difficult to ever measure the software reliability with any level of confidence • (Chris Gill) • If we can find ways to decompose the problem