Expert System Presentation On….

1. Expert System Presentation On…. Software Certification for Industry - Verification and Validation Issues in Expert Systems By Anca I. Vermesan Presented by Wilford Morrison

2. Software Certification for Industry - Topics • Introduction • KBS Characteristics applicable to V&V • Task Related • Architecture Related • Software Certification • Product vs Process Certification • Components of Certification • Quality Characteristics of KBS • V&V Methods & Techniques • Methods of Certification • Industrial Needs • V&V of Safety Critical Systems • Conclusion V&V – Verification & Validation KBS- Knowledge-based Software

3. Software Certification for Industry Introduction • What is Verification and Validation? • V&V techniques used • What is Certification? • Components of Certification • Why Certification for KBS? • Issues with Certification

4. Software Certification for Industry Characteristics of KBS Applicable to V&V • New V&V techniques developed for KB component • Two major characteristics: • Type of task performed – What are specific task being carried out within a narrow domain? • Type of Internal architecture – The similarity of architecture across all KBS

5. Software Certification for Industry Characteristics of KBS Applicable to V&V • The vague nature of the requirement specification • Incremental nature of the knowledge elicitation process • Not amenable to formal analysis Task-Related : Issue: In KBS what serve as input is typically not well defined at the outset and the relation of possible inputs to the output may not be clear either compared to conventional system

6. Software Certification for Industry Characteristics of KBS Applicable to V&V • Uses declarative style programming languages (usually Rule-based) • Often cause different types of errors and anomalies (new V&V technique needed) Architecture-Related : Issue: While some of the errors are similar to conventional software, the effects and methods of detection are rather different

7. Software Certification for Industry Software Certification • Why software certification? • Gives confidence, quality and dependability • Determinant factor in software competitiveness • Assuring their conformance with requirements and specifications

8. Software Certification for Industry Software Certification ….cont’d • Composed of three types:- • Self certification- when a producer declares that their product conforms to specified standards • Buyer certification- a particular buyer may require that products be submitted for certification by a specified body • Third-party- a body independent of both the producer and the user carries out the certification process (the focus of this paper)

9. Software Certification for Industry Software Certification ….cont’d • Consist of three major activities:- • Software quality evaluation- involves V&V techniques, tests, static/dynamic analysis as a measurement to determine quality of the software • Software quality assessment- the process of comparing the actual tests and measurements of the functions of interest with the specifications • Software quality certification- an independent third party gives written assurance that a software product or process conforms to specified functions

10. Software Certification for Industry Software Certification ….cont’d Product vs. Process Certification • The idea that process-approach gave way to product-approach • Ultimate goal of certification is to assess the quality of the software product • Therefore, process certification is intended to analyse the software development process

11. Software Certification for Industry Software Certification ….cont’d • Process Oriented Certification steps include:- • Certifier and manufacturer agree on criticality of the software • A software methodology is then agreed upon • Software is developed, validated and tested according to the methodology • Documented results allow the certifier to approve the process and a certificate may be granted Product vs. Process Certification

12. Software Certification for Industry Software Certification ….cont’d • Product Oriented Certification steps include :- • Analysis of the actual requirement for the evaluation (This may refer to standards and regulations with which the product should comply) • Specification of the evaluation that is to comply with the requirements identified by the analysis • Design of the evaluation which shall identify evaluation techniques and plan • Performing the evaluation (consist of inspecting and testing product parts) • Reporting the result obtained from performing the above activities Product vs. Process Certification

13. Software Certification for Industry Components of Certification • The three major components are:- • Quality characteristics • V&V methods and techniques • Certification methods

14. Software Certification for Industry Components of Certification ….cont’d • This list consist of:- • Competency- this deals with the quality of the knowledge in the system relative to human skills • Completeness- this involves ensuring all the knowledge is referenced and there is no attempts to access non-existing knowledge • Correctness- the knowledge within the KB should be 100% • Reliability- determines how often a system fails to arrive at the correct solution to a problem • Testability- system should be designed so that a test plan can be carried out • Verifiability- possible to establish logical correctness of the KB through verification Quality characteristics

15. Software Certification for Industry Components of Certification ….cont’d • Clustered into two main groups:- • Non-method specific techniques- involves human analysis of a product • Method specific techniques- involves using a formal and structured approach V&V Methods and Techniques

16. Software Certification for Industry Components of Certification ….cont’d • Non-method specific techniques • Involves human analysis of product, relying on individuals to use their experience to find errors • Involves the general examination of programs by using reviews, walkthroughs and inspections to identify defects and discrepancies • Inspections are aim at detecting semantically incorrect knowledge V&V Methods and Techniques Issue: Use of this technique usually lead to bias and omissions

17. Software Certification for Industry Components of Certification ….cont’d • Method specific techniques • The properties of the product are well defined and can be expressed in a structured and formal way • This method may offer rigorous checks of the code functionality to mathematical proof of specification refinement • Can be further classified into:- • Empirical methods • Logical methods V&V Methods and Techniques

18. Software Certification for Industry Components of Certification ….cont’d Empirical Methods Involves running a prototype system on a selection of test cases and assessing the results. (Known as “black-box” validation ) Logical Methods Consist of identifying anomalies such as redundant, contradictory, circular or deficient knowledge. (Known as “white-box” validation ) V&V Methods and Techniques Issue: exhaustive testing is impossible due to the many paths through the KB

19. Software Certification for Industry Components of Certification ….cont’d • Common methods used are:- • Static Analysis • Dynamic Testing • Inspections • Modeling Methods of Certification

20. Software Certification for Industry Components of Certification ….cont’d Static Analysis • This involves subjecting the program texts (and its requirements and specifications documents) to scrutiny and review in order to detect inconsistencies and omissions Static Analysis • This is a generic form for techniques of software analysis which do not involve software execution • Other forms include, dataflow diagrams, symbolic execution, requirements tracing and program proving Methods of Certification Note: Static methods are in general judged cheaper, more practical to perform, and more amenable to automatic testing than dynamic methods

21. Software Certification for Industry Components of Certification ….cont’d Dynamic Testing • Encompasses any form of software analysis, which involves software execution. • In KBS, the input space may be so large that it is infeasible to examine the behavior for all inputs • Also for KBS, an expert is usually required to judge the errors detected, and many test cases need to be generated for systems with extensive chaining between rules • Other forms include, random testing, equivalence partitioning, boundary testing and cause-effect graphs Methods of Certification Note: Dynamic testing for evaluation in the view of certification would most likely involve re-running test cases prepared during the software process development and applied during the testing phase

22. Software Certification for Industry Industrial Needs • Software Certification has strategic importance to all industries • Mature software industry help to integrate other industries • Improved V&V techniques along with considerable experience and expertise has lead to a transfer to industry

23. Software Certification for Industry Industrial Needs….cont’d • As essential components- where the KB element performs an essential function within the system and without which the system will not perform • As advisory components- where they act in an advisory or supervisory capacity to enhance an independent system KBS in Safety Critical System Note: - When a KBS fails, it usually fails catastrophically rather than safely - KBS put a burden on the overall system hazard and safety analysis

24. Software Certification for Industry Conclusion • V&V of KBS are as much potential value to certification specialists as they are to software producers • Certification of KBS cannot be sufficiently accomplished by use of static testing means • V&V need objective points of reference on which to guide the certification activities

25. Software Certification for Industry Conclusion….cont’d • The “process culture” of traditional software engineering may at last lead to the “product culture” in knowledge engineering • Increasing use of software embedded in systems that need to be certified for safety reasons will demonstrate the necessity for software certification • Software certification has a strategic importance, as it is a way in which the software industry can become mature and integrate harmonically with other industries

26. Software Certification for Industry - Verification and Validation Issues in Expert Systems Questions?