1 / 31

Black Box Checking - PowerPoint PPT Presentation

Black Box Checking Book: Chapter 9 Model Checking Finite state description of a system B . LTL formula . Translate  into an automaton P . Check whether L( B )  L( P )=. If so, S satisfies . Otherwise, the intersection includes a counterexample. Repeat for different properties.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Black Box Checking

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Black Box Checking

Book: Chapter 9

Model Checking

• Finite state description of a system B.

• LTL formula . Translate  into an automaton P.

• Check whether L(B)  L(P)=.

• If so, S satisfies . Otherwise, the intersection includes a counterexample.

• Repeat for different properties.

Buchi automata (w-automata)

S - finite set of states. (B has l n states)

S0S - initial states. (P has m states)

S - finite alphabet. (contains p letters)

d S  SS - transition relation.

F S - accepting states.

Accepting run: passes a state in F infinitely often.

System automata: F=S, deterministic.

a

<>a

a

a, a

a

a

a

a

<>a

Example: check <>a

a, a

<>~a

a

a

Use automatic translation algorithms, e.g.,

[Gerth,Peled,Vardi,Wolper 95]

a

c

b

Every element in the product is a counter example for the checked property.

a

a

s1

s2

q1

a

b

c

a

a

q2

s3

a

s1,q1

s2,q1

Acceptance isdetermined byautomaton P.

b

a

s1,q2

s3,q2

c

Testing

• Unknown deterministic finite state system B.

• Known: n states and alphabet .

• An abstract model C of B. C satisfies all the properties we want from B.

• Check conformance of B and C.

• Another version: only a bound n on the number of states l is known.

Given Finite state system B.

Transition relation of B known.

Property represent by automaton P.

Check if L(B)  L(P)=.

Graph theory or BDD techniques.

Complexity: polynomial.

Unknown Finite state system B.

Alphabet and number of states of B or upper bound known.

Specification given as an abstract system C.

Check if B C.

Complexity: polynomial if number states known. Exponential otherwise.

Model Checking / Testing

Property represent by automaton P.

Check if L(B)  L(P)=.

Graph theory techniques.

Unknown Finite state system B.

Alphabet and Upper bound on Number of states of B known.

Complexity: exponential.

Combination lock automaton

Accepts only words with a specific suffix (cdab in the example).

c

d

a

b

s1

s2

s3

s4

s5

b

b

a

a

b

a

b

a

a

b

Conformance testing

Cannot distinguish if reduced or not.

a

b

a

Conformance testing (cont.)

When the black box is nondeterministic, we might never test some choices.

a

Conformance testing (cont.)

b

b

a

a

a

a

b

b

a

a

b

Need: bound on number of states of B.

a

b

b

s1

s2

a

a

s3

Vasilevskii algorithm

• Known automaton A has l states.

• Black box automaton has up to n states.

• Check each transition. Check that there are no "combination lock" errors.

• Complexity: O(l2 n p n-l+1).

• When n=l: O(l3p).

reset

a

a

b

b

c

c

try c

try b

a

a

b

b

c

c

a

a

b

c

b

c

fail

Experiments

• Nondeterministic algorithm:guess a path of length  n from the initial state to a deadlock state.Linear time, logarithmic space.

• Deterministic algorithm:systematically try paths of length n, one after the other (and use reset), until deadlock is reached.Exponential time, linear space.

• Nondeterministic algorithm:Linear time, logarithmic space.

• Deterministic algorithm:Exponential (p n-1) time, linear space.

• Lower bound: Exponential time (usecombination lock automata).

• How does this conform with what we know about complexity theory?

Modeling black box checking

• Cannot model using Turing machines: not all the information about B is given. Only certain experiments are allowed.

• We learn the model as we make the experiments.

• Can use the model of games of incomplete information.

Games of incomplete information

• Two players: \$-player, -player (here, deterministic).

• Finitely many configurations C. Including:Initial Ci , Winning : W+ and W- .

• An equivalence relation @ on C (the \$-player cannot distinguish between equivalent states).

• Labels L on moves (try a, reset, success, fail).

• The \$-player has the moves labeled the same from configurations that are equivalent.

• Strategy for the \$-player: will lead to a configuration in W+  W-. Cannot distinguish equivalent conf.

• Nondet. strategy: ends with W+. Can distinguish.

Modeling BBC as games

• Each configuration contains an automaton and its current state (and more).

• Moves of the \$-player are labeled withtry a, reset... Moves of the -player withsuccess, fail.

• c1@ c2 when the automata in c1and c2 would respond in the same way to the experiments so far.

A naive strategy for BBC

• Learn first the structure of the black box.

• Then apply the intersection.

• Enumerate automata with n states (without repeating isomorphic automata).

• For a current automata and newautomata, construct a distinguishing sequence. Only one of them survives.

• Complexity: O((n+1)p (n+1)/n!)

On-the-fly strategy

• Systematically (as in the deadlock case), find two sequences v1 and v2 of length <=m n.

• Applying v1 to P brings us to a state t that is accepting.

• Applying v2 to P brings us back to t.

• Apply v1 (v2 )n+1 to B. If this succeeds,there is a cycle in the intersection labeled with v2, with t as the P (accepting) component.

• Complexity: O(n2p2mnm).

Learning an automaton

• Use Angluin’s algorithm for learning an automaton.

• The learning algorithm queries whether some strings are in the automaton B.

• It can also conjecture an automaton Miand asks for a counterexample.

• It then generates an automaton with more states Mi+1and so forth.

A strategy based on learning

• Start the learning algorithm.

• Queries are just experiments to B.

• For a conjectured automaton Mi , check if Mi  P = 

• If so, we check conformance of Mi with B (Vasilevskii algorithm).

• If nonempty, it contains some v1 (v2)w . We test B with v1 (v2)n+1. If this succeeds: error, otherwise, this is a counterexample for Mi .

Black Box Checking Strategy

Incremental

learning

discrepancy

false negative

Model

Path

ModelChecking

no counterexample

counterexample

black boxtesting

Comparing

counterexample

System

actual error

conformance established

Report

error

No error

found

Complexity

• l - real size of B.

• n - an upper bound of size of B.

• p - size of alphabet.

• Lower bound: reachability is similar to deadlock.

• O(l 3 p l + l 2mn) if there is an error.

• O(l 3 p l + l 2 n p n-l+1+ l 2mn) if there is no error.

If n is not known, check while time allows.

Some experiments

• Basic system written in SML (by Alex Groce, CMU).

• Experiment with black box using Unix I/O.

• Allows model-free model checking of C code with inter-process communication.

• Compiling tested code in SML with BBC program as one process.

Conclusions

• Black box checking is a combination of testing and model checking.

• If a tight bound on size of B is given: learn B first, then do model checking.

• Tight lower bound on complexity, up to polynomial factor.

• Use of games of incomplete information to model testing problems.