Black Box Checking

383 Views

Download Presentation
## Black Box Checking

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**Black Box Checking**Book: Chapter 9**Model Checking**• Finite state description of a system B. • LTL formula . Translate into an automaton P. • Check whether L(B) L(P)=. • If so, S satisfies . Otherwise, the intersection includes a counterexample. • Repeat for different properties. **Buchi automata (w-automata)**S - finite set of states. (B has l n states) S0S - initial states. (P has m states) S - finite alphabet. (contains p letters) d S SS - transition relation. F S - accepting states. Accepting run: passes a state in F infinitely often. System automata: F=S, deterministic.**Example: check a**a <>a a a, a**a**a a a Example: check <>a <>a**Example: check <>a**a, a <>~a a a Use automatic translation algorithms, e.g., [Gerth,Peled,Vardi,Wolper 95]**a**c b System**Every element in the product is a counter example for the**checked property. a a s1 s2 q1 a b c a a q2 s3 a s1,q1 s2,q1 Acceptance isdetermined byautomaton P. b a s1,q2 s3,q2 c****Testing • Unknown deterministic finite state system B. • Known: n states and alphabet . • An abstract model C of B. C satisfies all the properties we want from B. • Check conformance of B and C. • Another version: only a bound n on the number of states l is known.**Given Finite state system B.**Transition relation of B known. Property represent by automaton P. Check if L(B) L(P)=. Graph theory or BDD techniques. Complexity: polynomial. Unknown Finite state system B. Alphabet and number of states of B or upper bound known. Specification given as an abstract system C. Check if B C. Complexity: polynomial if number states known. Exponential otherwise. Model Checking / Testing**Property represent by automaton P.**Check if L(B) L(P)=. Graph theory techniques. Unknown Finite state system B. Alphabet and Upper bound on Number of states of B known. Complexity: exponential. Black box checking **Combination lock automaton**Accepts only words with a specific suffix (cdab in the example). c d a b s1 s2 s3 s4 s5****b b a a b a b a a b Conformance testing Cannot distinguish if reduced or not.**a**b a Conformance testing (cont.) When the black box is nondeterministic, we might never test some choices.**a**Conformance testing (cont.) b b a a a a b b a a b Need: bound on number of states of B.**Need reliable RESET**a b b s1 s2 a a s3**Vasilevskii algorithm**• Known automaton A has l states. • Black box automaton has up to n states. • Check each transition. Check that there are no "combination lock" errors. • Complexity: O(l2 n p n-l+1). • When n=l: O(l3p).**reset**a a b b c c try c try b a a b b c c a a b c b c fail Experiments**Simpler problem: deadlock?**• Nondeterministic algorithm:guess a path of length n from the initial state to a deadlock state.Linear time, logarithmic space. • Deterministic algorithm:systematically try paths of length n, one after the other (and use reset), until deadlock is reached.Exponential time, linear space.**Deadlock complexity**• Nondeterministic algorithm:Linear time, logarithmic space. • Deterministic algorithm:Exponential (p n-1) time, linear space. • Lower bound: Exponential time (usecombination lock automata). • How does this conform with what we know about complexity theory?**Modeling black box checking**• Cannot model using Turing machines: not all the information about B is given. Only certain experiments are allowed. • We learn the model as we make the experiments. • Can use the model of games of incomplete information.**Games of incomplete information**• Two players: $-player, -player (here, deterministic). • Finitely many configurations C. Including:Initial Ci , Winning : W+ and W- . • An equivalence relation @ on C (the $-player cannot distinguish between equivalent states). • Labels L on moves (try a, reset, success, fail). • The $-player has the moves labeled the same from configurations that are equivalent. • Strategy for the $-player: will lead to a configuration in W+ W-. Cannot distinguish equivalent conf. • Nondet. strategy: ends with W+. Can distinguish.**Modeling BBC as games**• Each configuration contains an automaton and its current state (and more). • Moves of the $-player are labeled withtry a, reset... Moves of the -player withsuccess, fail. • c1@ c2 when the automata in c1and c2 would respond in the same way to the experiments so far.**A naive strategy for BBC**• Learn first the structure of the black box. • Then apply the intersection. • Enumerate automata with n states (without repeating isomorphic automata). • For a current automata and newautomata, construct a distinguishing sequence. Only one of them survives. • Complexity: O((n+1)p (n+1)/n!)**On-the-fly strategy**• Systematically (as in the deadlock case), find two sequences v1 and v2 of length <=m n. • Applying v1 to P brings us to a state t that is accepting. • Applying v2 to P brings us back to t. • Apply v1 (v2 )n+1 to B. If this succeeds,there is a cycle in the intersection labeled with v2, with t as the P (accepting) component. • Complexity: O(n2p2mnm).**Learning an automaton**• Use Angluin’s algorithm for learning an automaton. • The learning algorithm queries whether some strings are in the automaton B. • It can also conjecture an automaton Miand asks for a counterexample. • It then generates an automaton with more states Mi+1and so forth.**A strategy based on learning**• Start the learning algorithm. • Queries are just experiments to B. • For a conjectured automaton Mi , check if Mi P = • If so, we check conformance of Mi with B (Vasilevskii algorithm). • If nonempty, it contains some v1 (v2)w . We test B with v1 (v2)n+1. If this succeeds: error, otherwise, this is a counterexample for Mi .**Black Box Checking Strategy**Incremental learning discrepancy false negative Model Path ModelChecking no counterexample counterexample black boxtesting Comparing counterexample System actual error conformance established Report error No error found**Complexity**• l - real size of B. • n - an upper bound of size of B. • p - size of alphabet. • Lower bound: reachability is similar to deadlock. • O(l 3 p l + l 2mn) if there is an error. • O(l 3 p l + l 2 n p n-l+1+ l 2mn) if there is no error. If n is not known, check while time allows.**Some experiments**• Basic system written in SML (by Alex Groce, CMU). • Experiment with black box using Unix I/O. • Allows model-free model checking of C code with inter-process communication. • Compiling tested code in SML with BBC program as one process.**Conclusions**• Black box checking is a combination of testing and model checking. • If a tight bound on size of B is given: learn B first, then do model checking. • Tight lower bound on complexity, up to polynomial factor. • Use of games of incomplete information to model testing problems.