A UNIFIED TEST MODEL FOR HARDWARE AND SOFTWARE SYSTEMS

1. A UNIFIED TEST MODEL FOR HARDWARE AND SOFTWARE SYSTEMS József SZIRAY Department of Informatics Széchenyi University Egyetem tér 1, H-9026 Győr, Hungary

2. I. INTRODUCTION • Reliable operation and application of hardware and software systems impose rigorous requirements for their development. The hardware and software have to undergo some specific verification and validation procedures. • The task ofverification: To check the consistence between the individual development phases. Validationis meant for the checking of the system whether it conforms to the user requirements.

3. Adequate performance of verification and validation implies the following advantages: • It helps in deciding whether the next development phase can be started. • It may reveal errors in an early period of the development. • It provides data and information for the quality and reliability in any of the phases. • It may point out at an early state that the hardware and software do not meet the requirements.

4. All this has to involve previously planned checking, as well as intensive testing of the hardware and software in each phase. Testing is an integral part of both verification and validation. • Here emphasis is placed on the so-called safety-criticalcomputer systems where the most important criterion is safe operation, without endangering life and great values. • This type of computing system is taken into account here, since it needs the most thoroughly planned and performed procedures, in comparison with other systems.

5. II. FAULT MODEL AND BASIC NOTIONS A. Hardware faults: • Specification faults (user faults, external faults): Faults which occur at the beginning of the development cycle, and manifest themselves in the malfunctioning of the hardware, by not meeting the real user requirements. • Design faults: These faults occur in the course of designing the hardware itself. It means the erroneous or incomplete design implementation of the specification. E. g., erroneous logic design. • Operational faults: Faults which originate from the manufacturing process, as well as occur in the course of normal functioning, when the hardware itself goes wrong. • Stuck-at-constant logic level faults. • Bridging (coupling) faults. • Behavioral (functional) faults. • Switch-level (transistor) faults. • Timing (delay) faults.

6. B. Software faults: The fundamental feature of software faults is that they are present all the time during operation. The question is how their effect manifests itself in various situations. There are two classes of software faults: • Specification faults (user faults, external faults, design faults): Faults which occur at the beginning of the development cycle, and manifest themselves in the malfunctioning of the software, by not meeting the real user requirements. • Programming faults (development faults, internal faults, implementation faults): It involves the wide group of faults that originate from the mistakes made by the programmers in the course of design and coding of the software.

7. Erroneous performance of a function. Missing functions. Data-management errors in accessing the data base. Starting and termination faults. Faults in the user interface. Underflow or overflow of boundary values. Coding error. Algorithmic error. Initialization error. Error in the control flow. Data transmission error. Input-Output errors Race among program blocks. Workload error. Programming faultsPossible types of faults:

8. III. THE VERIFICATION AND VALIDATION MODEL A. The mapping model for hardware: A hardware system can be considered as an entity which maps an input set to an output set. For the sake of simplicity, combinations and sequences of inputs will be considered as a single input. The response to these inputs is made by producing an output or a set of outputs. Let the set of all possible inputs to the hardware under consideration be denoted by HWINPD (input domain), and the set of all possible output responses by HWOUTD (output domain). Now the one-to-one mapping of HWINPD into HWOUTD by the hardware system will be defined in the following form:

9. HWM (HWINPD) = HWOUTD. (1) Relation (1) means that the properties of the system, i. e., the mapping HWM, determine the correspondence between the elements of HWINPD and the elements of HWOUTD.

10. Furthermore, we denote the set of inputs which cause malfunction by HWINER, while the set of erroneous outputs by HWOUTER, which is produced as a response to HWINER. For these sets the following mapping relation holds true: HWM (HWINER) = HWOUTER. (2) The scheme for the above two mappings is depicted in Figure 1.

11. Figure 1. Hardware mapping scheme with faults:

12. If we involve tests for fault detection in our mapping model, it will result in the modification of sets HWINER and HWOUTER. Suppose a test set HWINERT is capable of detecting a subset of yet undetected design faults. In this case HWINERT and HWINER necessarily have to possess common elements. As a usual consequence, this test must result in fault removal. After this, the new mapping relation for the corrected hardware or hardware design is

13. HWM (HWINER  HWINERT) = HWOUTER (3) where the minus sign stands for subtraction between sets. The corresponding scheme is seen in Figure 2.

14. Figure 2. Mapping scheme after testing:

15. At this point the following notations are introduced: ·The set of inputs which manifest themselves in design faults: HWDEFI. The output mapping of this set is HWDEFO. ·The set of inputs which manifest themselves in operational faults: HWOPFI. The output mapping of this set is HWOPFO. ·The set of test inputs which have been devised to detect design faults, i. e., the set of verification tests: HWDFT. ·The set of test inputs which have been devised to detect operational faults, i. e., the set of fault-detection tests: HWOPFT.

16. ·The set of inputs which manifest themselves in user faults: HWUSFI. The output mapping of this set is HWUSFO. ·The set of test inputs which have been devised to detect user faults, i. e., the set of validation tests: HWUSFT. Now that we have a way of mapping various input sets into output sets, we can give a general model to describe the mapping scheme of verification, testing, and validation related to a specific hardware system. The mapping relations are as follows:

17. HWM (HWINPD) = HWOUTD. (4) HWM (HWDEFI  HWDEFT) = HWDEFO. (5) HWM (HWOPFI  HWOPFT) = HWOPFO. (6) HWM (HWUSFI  HWUSFT) = HWUSFO. (7) Here relation (5) expresses the design-verification mapping, relation (6) expresses the mapping for operational testing, while relation (7) is for the validation mapping. From these relations it can be seen that (5) represents the undetected design faults, (6) represents the uncovered operational faults, whereas (7) represents the undetected user faults.

18. These test sets are advisable to be combined with each other, in order to increase their fault detection capability. Some useful combinations: HWM (HWDEFI  (HWDFT  HWOPFT)) = HWDEFO. (8) HWM (HWOPFI  (HWDFT  HWOPFT)) = HWOPFO.(9) HWM (HWUSFI  (HWDEFT  HWUSFT)) = HWUSFO.(10) HWM (HWUSFI  (HWDEFT  HWOPFT  HWUSFT)) = HWUSFO.(11)

19. B. The mapping model for software: • Software faults are derived from programming errors, as well as erroneous or imperfect specification. • Here we will refer to these categories as development faults and user faults, respectively. • The first category of fault is a result of inadequate development process, whereas the second one does not conform to the ultimate user requirements.

20. A software system can be considered as an entity which maps an input set to an output set. A system has many possible inputs. The response to these inputs is made by producing an output or a set of outputs. • The software faults manifest themselves in program malfunction when the faulty code segment is executed with a set of inputs which expose the actual fault. At the same time, the rest of the code may work properly for the other inputs.

21. Let the set of all possible inputs to the software under consideration be denoted by INPD (input domain), and the set of all possible output responses by OUTD (output domain). Now the mapping of INPD into OUTD by the software system will be defined in the following form: SWM (SWINPD) =SWOUTD. (12) • Relation (1) means that the properties of the system, i. e., the mapping SWM, determine the correspondence between the elements of INPD and the elements of OUTD.

22. We denote the set of inputs which cause malfunction by SWINER, while the set of erroneous outputs by SWOUTER, where these sets are associated with programming (development) faults. • For these sets the following mapping relation holds true: SWM (SWINER) = SWOUTER. (13)

23. If we involve tests for fault detection in our mapping model, it will result in the modification of sets INER and OUTER. Suppose a test set INERT is capable of detecting a subset of yet undetected faults. In this case INERT and INER necessarily have to possess common elements. As a usual consequence, this test must result in fault removal. It means that the corrected software will produce a modified output domain with a subset OUTER which does not represent the detected and so deleted faults any longer. After this, the new mapping relation for the corrected software is SWM (SWINER -SWINERT) = SWOUTER (14)

24. Notations: • The following notations are introduced: • The set of inputs which manifest themselves in development faults: SWDEFI. The output mapping of this set is SWDEFO. • The set of test inputs which have been devised to detect development faults, i. e., the set of verification tests: SWDEFT. • The set of inputs which manifest themselves in user faults: SWUSFI. The output mapping of this set is SWUSFO. • The set of test inputs which have been devised to detect user faults, i. e., the set of validation tests: SWUSFT.

25. The mapping relations for verification and validation: • SWM (SWINPD) = SWOUTD. (15) • SWM (SWDEFI – SWDEFT) = SWDEFO (16) • SWM (SWUSFI – SWUSFT) = SWUSFO (17) Here relation (16) expresses the verification mapping, while relation (17) expresses the validation mapping. From these relations it can be seen that (16) represents the undetected development faults, whereas (17) represents the undetected user faults.

26. For the purpose of decreasing the number of undetected faults it is worth combining the existent test sets as shown below: SWM (SWDEFI - (SWDEFT  SWUSFT)) = SWDEFO. (18) SWM (SWUSFI - (SWDEFT  SWUSFT)) = SWUSFO. (19) These relations are illustrated in Figure 3.

27. Figure 3. Mapping scheme for verification and validation of the software:

28. IV. THE USE OF FORMAL METHODS • The term formal methods describes the use of mathematical techniques in the specification, design, and analysis of computer hardware and software. A specification must be unambiguous, complete, consistent and correct. Documents written in natural languages are always susceptible to misunderstanding. It is also difficult to ensure that such documents represent full and correct description of the required system, or even to demonstrate that they are consistent.

29. Formal methods are based on the use of formal languages which have very precise and strict rules. This feature enables the specifications to be defined in a manner that can be interpreted unambiguously. It also makes possible the automatic checking of the specifications in order to find omissions and inconsistencies, i. e., to prove the completeness and consistency. Languages intended for this purpose are called system specification languages, or formal specification languages. Their use offers many potential advantages in all phases of the development cycle.

30. One of the greatest advantages of describing a system in a formal manner is that automated transformations among the necessary design phases may then be carried out on this description. As far as hardware is concerned, the most known specification language is VHDL (VLSI Hardware-Description Language). Here the final product is the layout design of the integrated circuit which realizes the operation of the specified and described hardware. This result justifies the term for this tool kit as a “silicon compiler”.

31. The final phase of a software development is the source code that can be executed on a computer. A typical standardized formal language for this purpose is UML (Unified Modeling Language), and its associated software tools.

32. The other advantage is that automated tests can be performed at the various design phases. This allows software tools to check for certain classes of error, but also allows different descriptions to be compared to decide if they are equivalent. This process is nothing else then the verification itself. Figure 4 shows that each step of transformation is followed by the task of confirming that the task has been carried out correctly. This involves demonstrating that the description which forms the input to a given phase is functionally equivalent to that produced at its output.

33. Figure 4. Application process of formal methods:

34. In an ideal case, the transformation procedures in the above outlined process are completely automated, having no human interaction, and are performed without any faults at any phase. If so, the external test input sequences for verification can be completely omitted. In our test model in relation (5) it means that HWDEFI = , where the symbol  represents the empty set.

35. By substitution into (5) we obtain HWM ( HWDEFT) = HWM () = HWDEFO = . (20) The same considerations apply to software verification, i.e., for the relation (16): SWDEFI = , and thus SWM ( SWDEFT) = SWM () = SWDEFO = . (21)

36. On the other hand, the initial formal specification of the software is always carried out by a manual way, and therefore design errors may never be excluded here, even if there is an automated consistency checking available. The reason for this is that a consistent design in itself does not guarantee the perfect fulfillment of the user requirements. Also, there is no guarantee for meeting the safety requirements either. As a consequence, the external validation testing is always necessary to apply. In the above ideal case: HWM (HWUSFI - HWUSFT) = HWUSFO. (22) SWM (SWUSFI - SWUSFT) = SWUSFO. (23)

37. Fully automatic performance is yet not applicable, that is, close cooperation and interaction of the skilled designers are further required. At present, the main advantage of formal methods is that they enable to perform the design and verification processes with greater reliability.

38. Conclusion: • In this presentation a general model for hardware/software testing has been proposed. The significance of the model is that it alleviates the clear differentiationbetween tests for verification, fault detection, and validation. This feature is important and useful in the process of test design and evaluation, especially in the case of safety-critical systems.