Black box checking
Download
1 / 31

Black Box Checking - PowerPoint PPT Presentation

Black Box Checking Book: Chapter 9 Model Checking Finite state description of a system B . LTL formula . Translate  into an automaton P . Check whether L( B )  L( P )=. If so, S satisfies . Otherwise, the intersection includes a counterexample. Repeat for different properties.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha

Download Presentation

Black Box Checking

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Black Box Checking

Book: Chapter 9


Model Checking

  • Finite state description of a system B.

  • LTL formula . Translate  into an automaton P.

  • Check whether L(B)  L(P)=.

  • If so, S satisfies . Otherwise, the intersection includes a counterexample.

  • Repeat for different properties.


Buchi automata (w-automata)

S - finite set of states. (B has l n states)

S0S - initial states. (P has m states)

S - finite alphabet. (contains p letters)

d S  SS - transition relation.

F S - accepting states.

Accepting run: passes a state in F infinitely often.

System automata: F=S, deterministic.


Example: check a

a

<>a

a

a, a


a

a

a

a

Example: check <>a

<>a


Example: check <>a

a, a

<>~a

a

a

Use automatic translation algorithms, e.g.,

[Gerth,Peled,Vardi,Wolper 95]


a

c

b

System


Every element in the product is a counter example for the checked property.

a

a

s1

s2

q1

a

b

c

a

a

q2

s3

a

s1,q1

s2,q1

Acceptance isdetermined byautomaton P.

b

a

s1,q2

s3,q2

c


Testing

  • Unknown deterministic finite state system B.

  • Known: n states and alphabet .

  • An abstract model C of B. C satisfies all the properties we want from B.

  • Check conformance of B and C.

  • Another version: only a bound n on the number of states l is known.


Given Finite state system B.

Transition relation of B known.

Property represent by automaton P.

Check if L(B)  L(P)=.

Graph theory or BDD techniques.

Complexity: polynomial.

Unknown Finite state system B.

Alphabet and number of states of B or upper bound known.

Specification given as an abstract system C.

Check if B C.

Complexity: polynomial if number states known. Exponential otherwise.

Model Checking / Testing


Property represent by automaton P.

Check if L(B)  L(P)=.

Graph theory techniques.

Unknown Finite state system B.

Alphabet and Upper bound on Number of states of B known.

Complexity: exponential.

Black box checking


Combination lock automaton

Accepts only words with a specific suffix (cdab in the example).

c

d

a

b

s1

s2

s3

s4

s5


b

b

a

a

b

a

b

a

a

b

Conformance testing

Cannot distinguish if reduced or not.


a

b

a

Conformance testing (cont.)

When the black box is nondeterministic, we might never test some choices.


a

Conformance testing (cont.)

b

b

a

a

a

a

b

b

a

a

b

Need: bound on number of states of B.


Need reliable RESET

a

b

b

s1

s2

a

a

s3


Vasilevskii algorithm

  • Known automaton A has l states.

  • Black box automaton has up to n states.

  • Check each transition. Check that there are no "combination lock" errors.

  • Complexity: O(l2 n p n-l+1).

  • When n=l: O(l3p).


reset

a

a

b

b

c

c

try c

try b

a

a

b

b

c

c

a

a

b

c

b

c

fail

Experiments


Simpler problem: deadlock?

  • Nondeterministic algorithm:guess a path of length  n from the initial state to a deadlock state.Linear time, logarithmic space.

  • Deterministic algorithm:systematically try paths of length n, one after the other (and use reset), until deadlock is reached.Exponential time, linear space.


Deadlock complexity

  • Nondeterministic algorithm:Linear time, logarithmic space.

  • Deterministic algorithm:Exponential (p n-1) time, linear space.

  • Lower bound: Exponential time (usecombination lock automata).

  • How does this conform with what we know about complexity theory?


Modeling black box checking

  • Cannot model using Turing machines: not all the information about B is given. Only certain experiments are allowed.

  • We learn the model as we make the experiments.

  • Can use the model of games of incomplete information.


Games of incomplete information

  • Two players: $-player, -player (here, deterministic).

  • Finitely many configurations C. Including:Initial Ci , Winning : W+ and W- .

  • An equivalence relation @ on C (the $-player cannot distinguish between equivalent states).

  • Labels L on moves (try a, reset, success, fail).

  • The $-player has the moves labeled the same from configurations that are equivalent.

  • Strategy for the $-player: will lead to a configuration in W+  W-. Cannot distinguish equivalent conf.

  • Nondet. strategy: ends with W+. Can distinguish.


Modeling BBC as games

  • Each configuration contains an automaton and its current state (and more).

  • Moves of the $-player are labeled withtry a, reset... Moves of the -player withsuccess, fail.

  • c1@ c2 when the automata in c1and c2 would respond in the same way to the experiments so far.


A naive strategy for BBC

  • Learn first the structure of the black box.

  • Then apply the intersection.

  • Enumerate automata with n states (without repeating isomorphic automata).

  • For a current automata and newautomata, construct a distinguishing sequence. Only one of them survives.

  • Complexity: O((n+1)p (n+1)/n!)


On-the-fly strategy

  • Systematically (as in the deadlock case), find two sequences v1 and v2 of length <=m n.

  • Applying v1 to P brings us to a state t that is accepting.

  • Applying v2 to P brings us back to t.

  • Apply v1 (v2 )n+1 to B. If this succeeds,there is a cycle in the intersection labeled with v2, with t as the P (accepting) component.

  • Complexity: O(n2p2mnm).


Learning an automaton

  • Use Angluin’s algorithm for learning an automaton.

  • The learning algorithm queries whether some strings are in the automaton B.

  • It can also conjecture an automaton Miand asks for a counterexample.

  • It then generates an automaton with more states Mi+1and so forth.


A strategy based on learning

  • Start the learning algorithm.

  • Queries are just experiments to B.

  • For a conjectured automaton Mi , check if Mi  P = 

  • If so, we check conformance of Mi with B (Vasilevskii algorithm).

  • If nonempty, it contains some v1 (v2)w . We test B with v1 (v2)n+1. If this succeeds: error, otherwise, this is a counterexample for Mi .


Black Box Checking Strategy

Incremental

learning

discrepancy

false negative

Model

Path

ModelChecking

no counterexample

counterexample

black boxtesting

Comparing

counterexample

System

actual error

conformance established

Report

error

No error

found


Complexity

  • l - real size of B.

  • n - an upper bound of size of B.

  • p - size of alphabet.

  • Lower bound: reachability is similar to deadlock.

  • O(l 3 p l + l 2mn) if there is an error.

  • O(l 3 p l + l 2 n p n-l+1+ l 2mn) if there is no error.

    If n is not known, check while time allows.


Some experiments

  • Basic system written in SML (by Alex Groce, CMU).

  • Experiment with black box using Unix I/O.

  • Allows model-free model checking of C code with inter-process communication.

  • Compiling tested code in SML with BBC program as one process.


Conclusions

  • Black box checking is a combination of testing and model checking.

  • If a tight bound on size of B is given: learn B first, then do model checking.

  • Tight lower bound on complexity, up to polynomial factor.

  • Use of games of incomplete information to model testing problems.


ad
  • Login