Black Box Checking

1 / 31

Black Box Checking - PowerPoint PPT Presentation

Black Box Checking Book: Chapter 9 Model Checking Finite state description of a system B . LTL formula . Translate  into an automaton P . Check whether L( B )  L( P )=. If so, S satisfies . Otherwise, the intersection includes a counterexample. Repeat for different properties.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

PowerPoint Slideshow about 'Black Box Checking' - Ava

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Black Box Checking

Book: Chapter 9

Model Checking
• Finite state description of a system B.
• LTL formula . Translate  into an automaton P.
• Check whether L(B)  L(P)=.
• If so, S satisfies . Otherwise, the intersection includes a counterexample.
• Repeat for different properties.

Buchi automata (w-automata)

S - finite set of states. (B has l n states)

S0S - initial states. (P has m states)

S - finite alphabet. (contains p letters)

d S  SS - transition relation.

F S - accepting states.

Accepting run: passes a state in F infinitely often.

System automata: F=S, deterministic.

Example: check a

a

<>a

a

a, a

a

a

a

a

Example: check <>a

<>a

Example: check <>a

a, a

<>~a

a

a

Use automatic translation algorithms, e.g.,

[Gerth,Peled,Vardi,Wolper 95]

Every element in the product is a counter example for the checked property.

a

a

s1

s2

q1

a

b

c

a

a

q2

s3

a

s1,q1

s2,q1

Acceptance isdetermined byautomaton P.

b

a

s1,q2

s3,q2

c

Testing
• Unknown deterministic finite state system B.
• Known: n states and alphabet .
• An abstract model C of B. C satisfies all the properties we want from B.
• Check conformance of B and C.
• Another version: only a bound n on the number of states l is known.
Given Finite state system B.

Transition relation of B known.

Property represent by automaton P.

Check if L(B)  L(P)=.

Graph theory or BDD techniques.

Complexity: polynomial.

Unknown Finite state system B.

Alphabet and number of states of B or upper bound known.

Specification given as an abstract system C.

Check if B C.

Complexity: polynomial if number states known. Exponential otherwise.

Model Checking / Testing
Property represent by automaton P.

Check if L(B)  L(P)=.

Graph theory techniques.

Unknown Finite state system B.

Alphabet and Upper bound on Number of states of B known.

Complexity: exponential.

Black box checking

Combination lock automaton

Accepts only words with a specific suffix (cdab in the example).

c

d

a

b

s1

s2

s3

s4

s5

b

b

a

a

b

a

b

a

a

b

Conformance testing

Cannot distinguish if reduced or not.

a

b

a

Conformance testing (cont.)

When the black box is nondeterministic, we might never test some choices.

a

Conformance testing (cont.)

b

b

a

a

a

a

b

b

a

a

b

Need: bound on number of states of B.

Vasilevskii algorithm
• Known automaton A has l states.
• Black box automaton has up to n states.
• Check each transition. Check that there are no "combination lock" errors.
• Complexity: O(l2 n p n-l+1).
• When n=l: O(l3p).

reset

a

a

b

b

c

c

try c

try b

a

a

b

b

c

c

a

a

b

c

b

c

fail

Experiments
• Nondeterministic algorithm:guess a path of length  n from the initial state to a deadlock state.Linear time, logarithmic space.
• Deterministic algorithm:systematically try paths of length n, one after the other (and use reset), until deadlock is reached.Exponential time, linear space.
• Nondeterministic algorithm:Linear time, logarithmic space.
• Deterministic algorithm:Exponential (p n-1) time, linear space.
• Lower bound: Exponential time (usecombination lock automata).
• How does this conform with what we know about complexity theory?
Modeling black box checking
• Cannot model using Turing machines: not all the information about B is given. Only certain experiments are allowed.
• We learn the model as we make the experiments.
• Can use the model of games of incomplete information.
Games of incomplete information
• Two players: \$-player, -player (here, deterministic).
• Finitely many configurations C. Including:Initial Ci , Winning : W+ and W- .
• An equivalence relation @ on C (the \$-player cannot distinguish between equivalent states).
• Labels L on moves (try a, reset, success, fail).
• The \$-player has the moves labeled the same from configurations that are equivalent.
• Strategy for the \$-player: will lead to a configuration in W+  W-. Cannot distinguish equivalent conf.
• Nondet. strategy: ends with W+. Can distinguish.
Modeling BBC as games
• Each configuration contains an automaton and its current state (and more).
• Moves of the \$-player are labeled withtry a, reset... Moves of the -player withsuccess, fail.
• c1@ c2 when the automata in c1and c2 would respond in the same way to the experiments so far.
A naive strategy for BBC
• Learn first the structure of the black box.
• Then apply the intersection.
• Enumerate automata with n states (without repeating isomorphic automata).
• For a current automata and newautomata, construct a distinguishing sequence. Only one of them survives.
• Complexity: O((n+1)p (n+1)/n!)
On-the-fly strategy
• Systematically (as in the deadlock case), find two sequences v1 and v2 of length <=m n.
• Applying v1 to P brings us to a state t that is accepting.
• Applying v2 to P brings us back to t.
• Apply v1 (v2 )n+1 to B. If this succeeds,there is a cycle in the intersection labeled with v2, with t as the P (accepting) component.
• Complexity: O(n2p2mnm).
Learning an automaton
• Use Angluin’s algorithm for learning an automaton.
• The learning algorithm queries whether some strings are in the automaton B.
• It can also conjecture an automaton Miand asks for a counterexample.
• It then generates an automaton with more states Mi+1and so forth.
A strategy based on learning
• Start the learning algorithm.
• Queries are just experiments to B.
• For a conjectured automaton Mi , check if Mi  P = 
• If so, we check conformance of Mi with B (Vasilevskii algorithm).
• If nonempty, it contains some v1 (v2)w . We test B with v1 (v2)n+1. If this succeeds: error, otherwise, this is a counterexample for Mi .
Black Box Checking Strategy

Incremental

learning

discrepancy

false negative

Model

Path

ModelChecking

no counterexample

counterexample

black boxtesting

Comparing

counterexample

System

actual error

conformance established

Report

error

No error

found

Complexity
• l - real size of B.
• n - an upper bound of size of B.
• p - size of alphabet.
• Lower bound: reachability is similar to deadlock.
• O(l 3 p l + l 2mn) if there is an error.
• O(l 3 p l + l 2 n p n-l+1+ l 2mn) if there is no error.

If n is not known, check while time allows.

Some experiments
• Basic system written in SML (by Alex Groce, CMU).
• Experiment with black box using Unix I/O.
• Allows model-free model checking of C code with inter-process communication.
• Compiling tested code in SML with BBC program as one process.
Conclusions
• Black box checking is a combination of testing and model checking.
• If a tight bound on size of B is given: learn B first, then do model checking.
• Tight lower bound on complexity, up to polynomial factor.
• Use of games of incomplete information to model testing problems.