Black Box Checking Book: Chapter 9
Model Checking • Finite state description of a system B. • LTL formula . Translate into an automaton P. • Check whether L(B) L(P)=. • If so, S satisfies . Otherwise, the intersection includes a counterexample. • Repeat for different properties.
Buchi automata (w-automata) S - finite set of states. (B has l n states) S0S - initial states. (P has m states) S - finite alphabet. (contains p letters) d S SS - transition relation. F S - accepting states. Accepting run: passes a state in F infinitely often. System automata: F=S, deterministic.
Example: check a a <>a a a, a
a a a a Example: check <>a <>a
Example: check <>a a, a <>~a a a Use automatic translation algorithms, e.g., [Gerth,Peled,Vardi,Wolper 95]
a c b System
Every element in the product is a counter example for the checked property. a a s1 s2 q1 a b c a a q2 s3 a s1,q1 s2,q1 Acceptance isdetermined byautomaton P. b a s1,q2 s3,q2 c
Testing • Unknown deterministic finite state system B. • Known: n states and alphabet . • An abstract model C of B. C satisfies all the properties we want from B. • Check conformance of B and C. • Another version: only a bound n on the number of states l is known.
Given Finite state system B. Transition relation of B known. Property represent by automaton P. Check if L(B) L(P)=. Graph theory or BDD techniques. Complexity: polynomial. Unknown Finite state system B. Alphabet and number of states of B or upper bound known. Specification given as an abstract system C. Check if B C. Complexity: polynomial if number states known. Exponential otherwise. Model Checking / Testing
Property represent by automaton P. Check if L(B) L(P)=. Graph theory techniques. Unknown Finite state system B. Alphabet and Upper bound on Number of states of B known. Complexity: exponential. Black box checking
Combination lock automaton Accepts only words with a specific suffix (cdab in the example). c d a b s1 s2 s3 s4 s5
b b a a b a b a a b Conformance testing Cannot distinguish if reduced or not.
a b a Conformance testing (cont.) When the black box is nondeterministic, we might never test some choices.
a Conformance testing (cont.) b b a a a a b b a a b Need: bound on number of states of B.
Need reliable RESET a b b s1 s2 a a s3
Vasilevskii algorithm • Known automaton A has l states. • Black box automaton has up to n states. • Check each transition. Check that there are no "combination lock" errors. • Complexity: O(l2 n p n-l+1). • When n=l: O(l3p).
reset a a b b c c try c try b a a b b c c a a b c b c fail Experiments
Simpler problem: deadlock? • Nondeterministic algorithm:guess a path of length n from the initial state to a deadlock state.Linear time, logarithmic space. • Deterministic algorithm:systematically try paths of length n, one after the other (and use reset), until deadlock is reached.Exponential time, linear space.
Deadlock complexity • Nondeterministic algorithm:Linear time, logarithmic space. • Deterministic algorithm:Exponential (p n-1) time, linear space. • Lower bound: Exponential time (usecombination lock automata). • How does this conform with what we know about complexity theory?
Modeling black box checking • Cannot model using Turing machines: not all the information about B is given. Only certain experiments are allowed. • We learn the model as we make the experiments. • Can use the model of games of incomplete information.
Games of incomplete information • Two players: $-player, -player (here, deterministic). • Finitely many configurations C. Including:Initial Ci , Winning : W+ and W- . • An equivalence relation @ on C (the $-player cannot distinguish between equivalent states). • Labels L on moves (try a, reset, success, fail). • The $-player has the moves labeled the same from configurations that are equivalent. • Strategy for the $-player: will lead to a configuration in W+ W-. Cannot distinguish equivalent conf. • Nondet. strategy: ends with W+. Can distinguish.
Modeling BBC as games • Each configuration contains an automaton and its current state (and more). • Moves of the $-player are labeled withtry a, reset... Moves of the -player withsuccess, fail. • c1@ c2 when the automata in c1and c2 would respond in the same way to the experiments so far.
A naive strategy for BBC • Learn first the structure of the black box. • Then apply the intersection. • Enumerate automata with n states (without repeating isomorphic automata). • For a current automata and newautomata, construct a distinguishing sequence. Only one of them survives. • Complexity: O((n+1)p (n+1)/n!)
On-the-fly strategy • Systematically (as in the deadlock case), find two sequences v1 and v2 of length <=m n. • Applying v1 to P brings us to a state t that is accepting. • Applying v2 to P brings us back to t. • Apply v1 (v2 )n+1 to B. If this succeeds,there is a cycle in the intersection labeled with v2, with t as the P (accepting) component. • Complexity: O(n2p2mnm).
Learning an automaton • Use Angluin’s algorithm for learning an automaton. • The learning algorithm queries whether some strings are in the automaton B. • It can also conjecture an automaton Miand asks for a counterexample. • It then generates an automaton with more states Mi+1and so forth.
A strategy based on learning • Start the learning algorithm. • Queries are just experiments to B. • For a conjectured automaton Mi , check if Mi P = • If so, we check conformance of Mi with B (Vasilevskii algorithm). • If nonempty, it contains some v1 (v2)w . We test B with v1 (v2)n+1. If this succeeds: error, otherwise, this is a counterexample for Mi .
Black Box Checking Strategy Incremental learning discrepancy false negative Model Path ModelChecking no counterexample counterexample black boxtesting Comparing counterexample System actual error conformance established Report error No error found
Complexity • l - real size of B. • n - an upper bound of size of B. • p - size of alphabet. • Lower bound: reachability is similar to deadlock. • O(l 3 p l + l 2mn) if there is an error. • O(l 3 p l + l 2 n p n-l+1+ l 2mn) if there is no error. If n is not known, check while time allows.
Some experiments • Basic system written in SML (by Alex Groce, CMU). • Experiment with black box using Unix I/O. • Allows model-free model checking of C code with inter-process communication. • Compiling tested code in SML with BBC program as one process.
Conclusions • Black box checking is a combination of testing and model checking. • If a tight bound on size of B is given: learn B first, then do model checking. • Tight lower bound on complexity, up to polynomial factor. • Use of games of incomplete information to model testing problems.