CTL Model-checking for Systems with Unspecified Components

1. CTL Model-checking for Systems with Unspecified Components Gaoyan Xie and Zhe Dang School of Electrical Engineering and Computer Science, Washington State University, Pullman, WA 99164, USA SAVCBS’04 Presented by: Gaoyan Xie

2. Motivation • Challenges to the quality assurance of component-based systems: • Externally obtained components may be new sources of system failures • Safely upgrading a deployed component is extremely difficult • Example: • The Ariane 5 disaster was caused by reusing certain software module from Ariane 4 without sufficiently testing it in the new environment SAVCBS’04

3. Two Dimensions of The Possible Solutions • Methodologies • Compositional approachesconventional, used most often • Bottom-up, from components to systems • Decompositional approachesnot many can be automated • Top-down, from systems to components • Techniques • Software Testing • A trial and error technique • Integration/system testing is often needed at the system level • Model-checking: • A formal and usually complete technique • The assume-guarantee paradigm is often used for system-level verification SAVCBS’04

4. Testing a Component in Isolation • Before it is released • By component developers • By third-party labscomponent certification • Difficulty: • A component may target a wide scope of deployment environments, which may not be tried out • When it is integrated into a system • By system developers • Expensive: • A component is usually built with multiple sets of functionalities • Infeasible: • The state space of a component’s interface may be too large to be thoroughly tested SAVCBS’04

5. Testing A Component-based System (CBS) • Integration/system testing • Difficulties: • Components within a CBS may run concurrently • Even inside a component may have multi-threads running concurrently, and • Concurrency is notoriously difficult to test • Locating the source of a bug is not easy! • Feasibility: • A component may be used for dynamic upgrading a running system, which may not be stopped for testing SAVCBS’04

6. Model-checking • Checks whether a finite-state system is a model of a given temporal formula • Requires a complete and formal specification of the finite-state system • Not always possible: design details or source codes of components in a CBS are generally unavailable • The assume-guarantee paradigm is often used for system-level verification • A compositional approach • Requires an assumption for the environment of a module SAVCBS’04

7. The Research Problem • A different perspective • How to ensure that acomponent functions correctly in a specific deployment environment (which is called a host system) • A model-checking problem • Given a host system M with an unspecified componentX and a CTL formula f, check whether f is a model of(M, X): (M, X)= f SAVCBS’04

8. Basic Ideas • Goal • Seek a definite answer (no approximation) to the model-checking problem, i.e., (M, X)= f • Solution • Automatically deduce a sufficient and necessary condition over the unspecified component • Check the condition through testing the unspecified component SAVCBS’04

9. The System Model • A system Sys= (M, X)consists of a host system M and an unspecified component X • The host system M is a well-specified finite state transition system • The unspecified component X can be viewed as a deterministic Mealy machine • Internal detail of X is unknown • Interface (input/output symbols) and an upper bound for the number of states in X are known • An implementation of X is available for testing • M and X run concurrently and communicate with each other by synchronizing over X’s input/output symbols SAVCBS’04

10. data yes Comm ack no An Example • Check: starting from state s0, is state s3 reachable? • (M, Comm), s0= E[true U s3] msg? msg? send/yes s0 s1 s2 send/no msg? ack/yes ack/yes s4 s3 s3 Host system M SAVCBS’04

11. An Examplecontd. • Check: starting from state s0, is state s3 reachable? • M, s0= E[true U s3] s0 s1 s2 s4 s3 Transition system M SAVCBS’04

12. An Examplecontd. • Check: starting from state s0, is state s3 reachable? • M, s0= E[true U s3] • Search the graph to see whether there is a path between s0 and s3 • s0s1s4s3 is such a witness path s0 s1 s2 s4 s3 Transition system M SAVCBS’04

13. send yes Comm ack no An Examplecontd. • Check: starting from state s0, is state s3 reachable? • (M, Comm), s0= E[true U s3] • Search the graph to see whether there is a path between s0 and s3 • Is s0s1s4s3 still a witness path? msg? Test Comm to see whether the communication trace along the path can be accepted by Comm: Will “send/no, ack/yes” be a successful test case for Comm? msg? send/yes s0 s1 s2 send/no msg? ack/yes ack/yes s4 s3 Host system M SAVCBS’04

14. send yes Comm ack no An Examplecontd. • Infinite number of paths could be witness path: • s0s1s2s1s4s3, s0s1s2s3, s0s1s2s1s2…s3,… • Infinite number of communication traces need to be tested: • “send/yes send/no ack/yes”, “send/yes ack/yes”, “send/yes send/yes…ack/yes”, … msg? msg? send/yes s0 s1 s2 send/no msg? ack/yes ack/yes s4 s3 Host system M SAVCBS’04

15. An Examplecontd. • Use an annotated directed graph Gh (called witness graph) to represent all those communication traces • The original model-checking problem is: • True, if one of the communication traces in Gh is a successful test case for Comm • False, if none of the communication traces in Gh is a successful test case for Comm msg? msg? send/yes send/yes s0 s1 s2 s1 s2 send/no send/no msg? ack/yes ack/yes ack/yes ack/yes s4 s3 s4 s3 Host system M Witness graph Gh SAVCBS’04

16. An Examplecontd. • How to handle communication traces with infinite length? • A result from traditional black-box testing: • An equivalent internal structure of a black-box can be recovered through testing if only an upper bound for its number of states is known • This result implies that • Only communication traces with bounded length need to be tested • The bound can be calculated from the number of states in M and the upper bound for the number of states in Comm SAVCBS’04

17. Our Algorithm • The original CTL model-checking algorithm for checking M= f: • Goes through a series of stages. • A subformula h of f is processed at each stage: • h is added to the labels of a state s if h is true at s • Returns true if s0 is labeled with f, or false otherwise • Our algorithm adapts the original algorithm to handle systems with unspecified components. • It follows a similar structure, except that during each stage for subformula h, the labeling of a state s is far more complicated SAVCBS’04

18. Our Algorithm—contd. • A state s could be labeled with • 1  if h is true at s and the truth has nothing to do with communications • A witness graph if h is an CTL operator and the truth of h at s depends on communication traces in the graph • A logical combination of witness graphs  if h is a logic combination of other formulas and the truth of h at s depends on communication traces in the graph • For each CTL operator in h, a witness graph is constructed and associated with an ID (which begins from 2) • A state is labeled with an ID expression (constructed from IDs and logic connectives , ) • A labeling function Lh is returned upon the completion of processing h SAVCBS’04

19. Our Algorithm—contd. • The algorithm outline: • ProcessCTL is the lableling process, it recursively handles the five cases of f:, , EX, EU,and EG by calling the procedures Negation, Union, ProcessEX, ProcessEU, and ProcessEG respectively Procedure CheckCTL(M, X, s0, f) Lf := ProcessCTL(M, f); If s0 is labeled by LfThen If Lf (s0) = 1 Then Return true; Else Return TestWG(X, reset, s0, Lf(s0)); Endif Else Return false; Endif End Procedure • The maximal length of communication traces to be tested is bounded by (k·n·m2), where k is the number of CTL operators in the formula f, m is the upper bound for the number of states in the unspecified component X, and n is the number of states in the host system M SAVCBS’04

20. send yes Comm ack no Another Example • Check: starting from the initial state s0, whenever the system reaches state s2, it would eventually reach s3; i.e., check (M, X), s0= AG(s2 AF s3) to be true • Taking a negation of the original problem, it’s equivalent to checking (M, X), s0= E[true U (s2  EG s3)] to be false msg? msg? send/yes s0 s1 s2 send/no msg? ack/yes ack/yes s4 s3 Host system M SAVCBS’04

21. send yes Comm ack no Check (M, X), s0=E[true U(s2EGs3)] • Step 1. Atomic subformula s2 is processed by ProcessCTL, which returns a labeling function L1 = {(s2, 1)} msg? msg? send/yes s0 s1 s2 send/no msg? ack/yes ack/yes s4 s3 Host system M SAVCBS’04

22. send yes Comm ack no Check (M, X), s0=E[true U(s2EGs3)] • Step 1. Atomic subformula s2 is processed by ProcessCTL, which returns a labeling function L1 = {(s2, 1)} • Step 2. Atomic subformula s3 is processed by ProcessCTL, which returns a labeling function L2 = {(s3, 1)} msg? msg? send/yes s0 s1 s2 send/no msg? ack/yes ack/yes s4 s3 Host system M SAVCBS’04

23. send yes Comm ack no Check (M, X), s0=E[true U(s2EGs3)] • Step 1. Atomic subformula s2 is processed by ProcessCTL, which returns a labeling function L1 = {(s2, 1)} • Step 2. Atomic subformula s3 is processed by ProcessCTL, which returns a labeling function L2 = {(s3, 1)} • Step 3. Subformula s3 is processed by Negation, which returns a labeling function L3 = {(s0, 1), (s1, 1), (s2, 1), (s4, 1)} msg? msg? send/yes s0 s1 s2 send/no msg? ack/yes ack/yes s4 s3 Host system M SAVCBS’04

24. Check (M, X), s0=E[true U(s2EGs3)] • Step 4. Subformula EG s3 is processed by ProcessEG, which constructs a witness graph G1 = <N, E, L3> with an ID 2 and returns a labeling function L4 = {(s0, 2), (s1, 2), (s2, 2)} msg? msg? send/yes s0 s1 s2 send/yes s0 s1 s2 send/no msg? ack/yes Witness graph G1 with ID 2 ack/yes s4 s3 The host system M SAVCBS’04

25. Check (M, X), s0=E[true U(s2EGs3)] • Step 4. Subformula EG s3 is processed by ProcessEG, which constructs a witness graph G1 = <N, E, L3> with an ID 2 and returns a labeling function L4 = {(s0, 2), (s1, 2), (s2, 2)} • Step 5. Subformula s2  EG s3 is processed by Negation and Union, which finally return a labeling function L5 = {(s2, 2)} msg? msg? send/yes s0 s1 s2 send/yes s0 s1 s2 send/no msg? ack/yes Witness graph G1 with ID 2 ack/yes s4 s3 The host system M SAVCBS’04

26. Check (M, X), s0=E[true U(s2EGs3)] • Step 6. The formula E[true U (s2  EG s3)]is processed by procedure ProcessEU, which constructs a witness graph G2 = <N, E, Ltrue, L5> with an ID 3 and returns a labeling function Lf = {(s0, 3), (s1, 3), (s2, 3), (s3, 3), (s4, 3)} • Lf (s0)  1, this indicates that the truth of the original problem depends on communications. Then we shall see whether Lf (s0) can be evaluated to be true msg? msg? send/yes send/yes s0 s1 s2 s0 s1 s2 send/no send/no msg? ack/yes ack/yes ack/yes ack/yes s4 s3 s4 s3 The host system M Witness graph G2 with ID 3 SAVCBS’04

27. send yes Comm ack no Evaluate ID Expression Lf (s0) • Evaluating Lf (s0) is a testing process with test-cases generated from the two witness graphs (details can be seen in the paper) • Essentially, we are testing whether some communication trace (of bounded length) with m consecutive symbol pairs “send yes” is a successful test-case for the unspecified component X. Errata: on page 6, in the paragraph above “Note.”, “… two consecutive symbol pairs …” should be “… m consecutive symbol pairs… ” msg? msg? send/yes s0 s1 s2 send/no msg? ack/yes ack/yes s4 s3 The host system M SAVCBS’04

28. Summary • A decompositional approach • Reduces a system verification problem to a testing problem over the unspecified component • A hybrid approach • Combines model-checking with software testing techniques • Advantages • Stronger confidence about the reliability of the system • Customization of testing over a component with respect to particular system requirement • Reuse of intermediate verification results • Automatic carrying out of the whole process • Complete and sound algorithm with an appropriate bound SAVCBS’04

29. Related Work • Specification-based testing [Heitmeyer et. al. 1999], generate test-cases from counter-examples produced by a model-checker • Black-box checking [Peled et. al. 1999], test a single black-box against a given LTL formula • Automatic assume-guarantee [Giannakopoulou et. al. 2004], automatic generation of assumptions of a component, uses a less expressive formalism (LTS) for properties • Automatic deduction of model-checking conditions [Fisler et. al. 2001], has false negative • [Xie & Dang 2004] • LTL algorithms for systems with only one, finite-state unspecified component (FATES’04) • An automata-theoretic and decompositional approach to testing a system of concurrent black-boxes (submitted) SAVCBS’04

30. Ongoing Work • Decompositional LTL model-checking • Less-restricted system model • Asynchronous communications • Multiple unspecified components • Unspecified components with infinite state space • Case studies • CORBA applications SAVCBS’04