Black box checking
Download
1 / 31

Black Box Checking - PowerPoint PPT Presentation


  • 368 Views
  • Uploaded on

Black Box Checking Book: Chapter 9 Model Checking Finite state description of a system B . LTL formula . Translate  into an automaton P . Check whether L( B )  L( P )=. If so, S satisfies . Otherwise, the intersection includes a counterexample. Repeat for different properties.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Black Box Checking' - Ava


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Black box checking l.jpg

Black Box Checking

Book: Chapter 9


Model checking l.jpg
Model Checking

  • Finite state description of a system B.

  • LTL formula . Translate  into an automaton P.

  • Check whether L(B)  L(P)=.

  • If so, S satisfies . Otherwise, the intersection includes a counterexample.

  • Repeat for different properties.


Buchi automata w automata l.jpg
Buchi automata (w-automata)

S - finite set of states. (B has l n states)

S0S - initial states. (P has m states)

S - finite alphabet. (contains p letters)

d S  SS - transition relation.

F S - accepting states.

Accepting run: passes a state in F infinitely often.

System automata: F=S, deterministic.


Example check a l.jpg
Example: check a

a

<>a

a

a, a


Example check a5 l.jpg

a

a

a

a

Example: check <>a

<>a


Example check a6 l.jpg
Example: check <>a

a, a

<>~a

a

a

Use automatic translation algorithms, e.g.,

[Gerth,Peled,Vardi,Wolper 95]


System l.jpg

a

c

b

System


Every element in the product is a counter example for the checked property l.jpg
Every element in the product is a counter example for the checked property.

a

a

s1

s2

q1

a

b

c

a

a

q2

s3

a

s1,q1

s2,q1

Acceptance isdetermined byautomaton P.

b

a

s1,q2

s3,q2

c


Testing l.jpg

Testing

  • Unknown deterministic finite state system B.

  • Known: n states and alphabet .

  • An abstract model C of B. C satisfies all the properties we want from B.

  • Check conformance of B and C.

  • Another version: only a bound n on the number of states l is known.


Model checking testing l.jpg

Given Finite state system B.

Transition relation of B known.

Property represent by automaton P.

Check if L(B)  L(P)=.

Graph theory or BDD techniques.

Complexity: polynomial.

Unknown Finite state system B.

Alphabet and number of states of B or upper bound known.

Specification given as an abstract system C.

Check if B C.

Complexity: polynomial if number states known. Exponential otherwise.

Model Checking / Testing


Black box checking11 l.jpg

Property represent by automaton P.

Check if L(B)  L(P)=.

Graph theory techniques.

Unknown Finite state system B.

Alphabet and Upper bound on Number of states of B known.

Complexity: exponential.

Black box checking


Combination lock automaton l.jpg
Combination lock automaton

Accepts only words with a specific suffix (cdab in the example).

c

d

a

b

s1

s2

s3

s4

s5


Conformance testing l.jpg

b

b

a

a

b

a

b

a

a

b

Conformance testing

Cannot distinguish if reduced or not.


Conformance testing cont l.jpg

a

b

a

Conformance testing (cont.)

When the black box is nondeterministic, we might never test some choices.


Conformance testing cont15 l.jpg

a

Conformance testing (cont.)

b

b

a

a

a

a

b

b

a

a

b

Need: bound on number of states of B.


Need reliable reset l.jpg
Need reliable RESET

a

b

b

s1

s2

a

a

s3


Vasilevskii algorithm l.jpg
Vasilevskii algorithm

  • Known automaton A has l states.

  • Black box automaton has up to n states.

  • Check each transition. Check that there are no "combination lock" errors.

  • Complexity: O(l2 n p n-l+1).

  • When n=l: O(l3p).


Experiments l.jpg

reset

a

a

b

b

c

c

try c

try b

a

a

b

b

c

c

a

a

b

c

b

c

fail

Experiments


Simpler problem deadlock l.jpg
Simpler problem: deadlock?

  • Nondeterministic algorithm:guess a path of length  n from the initial state to a deadlock state.Linear time, logarithmic space.

  • Deterministic algorithm:systematically try paths of length n, one after the other (and use reset), until deadlock is reached.Exponential time, linear space.


Deadlock complexity l.jpg
Deadlock complexity

  • Nondeterministic algorithm:Linear time, logarithmic space.

  • Deterministic algorithm:Exponential (p n-1) time, linear space.

  • Lower bound: Exponential time (usecombination lock automata).

  • How does this conform with what we know about complexity theory?


Modeling black box checking l.jpg
Modeling black box checking

  • Cannot model using Turing machines: not all the information about B is given. Only certain experiments are allowed.

  • We learn the model as we make the experiments.

  • Can use the model of games of incomplete information.


Games of incomplete information l.jpg
Games of incomplete information

  • Two players: $-player, -player (here, deterministic).

  • Finitely many configurations C. Including:Initial Ci , Winning : W+ and W- .

  • An equivalence relation @ on C (the $-player cannot distinguish between equivalent states).

  • Labels L on moves (try a, reset, success, fail).

  • The $-player has the moves labeled the same from configurations that are equivalent.

  • Strategy for the $-player: will lead to a configuration in W+  W-. Cannot distinguish equivalent conf.

  • Nondet. strategy: ends with W+. Can distinguish.


Modeling bbc as games l.jpg
Modeling BBC as games

  • Each configuration contains an automaton and its current state (and more).

  • Moves of the $-player are labeled withtry a, reset... Moves of the -player withsuccess, fail.

  • c1@ c2 when the automata in c1and c2 would respond in the same way to the experiments so far.


A naive strategy for bbc l.jpg
A naive strategy for BBC

  • Learn first the structure of the black box.

  • Then apply the intersection.

  • Enumerate automata with n states (without repeating isomorphic automata).

  • For a current automata and newautomata, construct a distinguishing sequence. Only one of them survives.

  • Complexity: O((n+1)p (n+1)/n!)


On the fly strategy l.jpg
On-the-fly strategy

  • Systematically (as in the deadlock case), find two sequences v1 and v2 of length <=m n.

  • Applying v1 to P brings us to a state t that is accepting.

  • Applying v2 to P brings us back to t.

  • Apply v1 (v2 )n+1 to B. If this succeeds,there is a cycle in the intersection labeled with v2, with t as the P (accepting) component.

  • Complexity: O(n2p2mnm).


Learning an automaton l.jpg
Learning an automaton

  • Use Angluin’s algorithm for learning an automaton.

  • The learning algorithm queries whether some strings are in the automaton B.

  • It can also conjecture an automaton Miand asks for a counterexample.

  • It then generates an automaton with more states Mi+1and so forth.


A strategy based on learning l.jpg
A strategy based on learning

  • Start the learning algorithm.

  • Queries are just experiments to B.

  • For a conjectured automaton Mi , check if Mi  P = 

  • If so, we check conformance of Mi with B (Vasilevskii algorithm).

  • If nonempty, it contains some v1 (v2)w . We test B with v1 (v2)n+1. If this succeeds: error, otherwise, this is a counterexample for Mi .


Black box checking strategy l.jpg
Black Box Checking Strategy

Incremental

learning

discrepancy

false negative

Model

Path

ModelChecking

no counterexample

counterexample

black boxtesting

Comparing

counterexample

System

actual error

conformance established

Report

error

No error

found


Complexity l.jpg
Complexity

  • l - real size of B.

  • n - an upper bound of size of B.

  • p - size of alphabet.

  • Lower bound: reachability is similar to deadlock.

  • O(l 3 p l + l 2mn) if there is an error.

  • O(l 3 p l + l 2 n p n-l+1+ l 2mn) if there is no error.

    If n is not known, check while time allows.


Some experiments l.jpg
Some experiments

  • Basic system written in SML (by Alex Groce, CMU).

  • Experiment with black box using Unix I/O.

  • Allows model-free model checking of C code with inter-process communication.

  • Compiling tested code in SML with BBC program as one process.


Conclusions l.jpg
Conclusions

  • Black box checking is a combination of testing and model checking.

  • If a tight bound on size of B is given: learn B first, then do model checking.

  • Tight lower bound on complexity, up to polynomial factor.

  • Use of games of incomplete information to model testing problems.


ad