- 357 Views
- Uploaded on
- Presentation posted in: Sports / GamesEducation / CareerFashion / BeautyGraphics / DesignNews / Politics

Black Box Checking

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Black Box Checking

Book: Chapter 9

- Finite state description of a system B.
- LTL formula . Translate into an automaton P.
- Check whether L(B) L(P)=.
- If so, S satisfies . Otherwise, the intersection includes a counterexample.
- Repeat for different properties.

S - finite set of states. (B has l n states)

S0S - initial states. (P has m states)

S - finite alphabet. (contains p letters)

d S SS - transition relation.

F S - accepting states.

Accepting run: passes a state in F infinitely often.

System automata: F=S, deterministic.

a

<>a

a

a, a

a

a

a

a

<>a

a, a

<>~a

a

a

Use automatic translation algorithms, e.g.,

[Gerth,Peled,Vardi,Wolper 95]

a

c

b

a

a

s1

s2

q1

a

b

c

a

a

q2

s3

a

s1,q1

s2,q1

Acceptance isdetermined byautomaton P.

b

a

s1,q2

s3,q2

c

- Unknown deterministic finite state system B.
- Known: n states and alphabet .
- An abstract model C of B. C satisfies all the properties we want from B.
- Check conformance of B and C.
- Another version: only a bound n on the number of states l is known.

Given Finite state system B.

Transition relation of B known.

Property represent by automaton P.

Check if L(B) L(P)=.

Graph theory or BDD techniques.

Complexity: polynomial.

Unknown Finite state system B.

Alphabet and number of states of B or upper bound known.

Specification given as an abstract system C.

Check if B C.

Complexity: polynomial if number states known. Exponential otherwise.

Property represent by automaton P.

Check if L(B) L(P)=.

Graph theory techniques.

Unknown Finite state system B.

Alphabet and Upper bound on Number of states of B known.

Complexity: exponential.

Accepts only words with a specific suffix (cdab in the example).

c

d

a

b

s1

s2

s3

s4

s5

b

b

a

a

b

a

b

a

a

b

Cannot distinguish if reduced or not.

a

b

a

When the black box is nondeterministic, we might never test some choices.

a

b

b

a

a

a

a

b

b

a

a

b

Need: bound on number of states of B.

a

b

b

s1

s2

a

a

s3

- Known automaton A has l states.
- Black box automaton has up to n states.
- Check each transition. Check that there are no "combination lock" errors.
- Complexity: O(l2 n p n-l+1).
- When n=l: O(l3p).

reset

a

a

b

b

c

c

try c

try b

a

a

b

b

c

c

a

a

b

c

b

c

fail

- Nondeterministic algorithm:guess a path of length n from the initial state to a deadlock state.Linear time, logarithmic space.
- Deterministic algorithm:systematically try paths of length n, one after the other (and use reset), until deadlock is reached.Exponential time, linear space.

- Nondeterministic algorithm:Linear time, logarithmic space.
- Deterministic algorithm:Exponential (p n-1) time, linear space.
- Lower bound: Exponential time (usecombination lock automata).
- How does this conform with what we know about complexity theory?

- Cannot model using Turing machines: not all the information about B is given. Only certain experiments are allowed.
- We learn the model as we make the experiments.
- Can use the model of games of incomplete information.

- Two players: $-player, -player (here, deterministic).
- Finitely many configurations C. Including:Initial Ci , Winning : W+ and W- .
- An equivalence relation @ on C (the $-player cannot distinguish between equivalent states).
- Labels L on moves (try a, reset, success, fail).
- The $-player has the moves labeled the same from configurations that are equivalent.
- Strategy for the $-player: will lead to a configuration in W+ W-. Cannot distinguish equivalent conf.
- Nondet. strategy: ends with W+. Can distinguish.

- Each configuration contains an automaton and its current state (and more).
- Moves of the $-player are labeled withtry a, reset... Moves of the -player withsuccess, fail.
- c1@ c2 when the automata in c1and c2 would respond in the same way to the experiments so far.

- Learn first the structure of the black box.
- Then apply the intersection.
- Enumerate automata with n states (without repeating isomorphic automata).
- For a current automata and newautomata, construct a distinguishing sequence. Only one of them survives.
- Complexity: O((n+1)p (n+1)/n!)

- Systematically (as in the deadlock case), find two sequences v1 and v2 of length <=m n.
- Applying v1 to P brings us to a state t that is accepting.
- Applying v2 to P brings us back to t.
- Apply v1 (v2 )n+1 to B. If this succeeds,there is a cycle in the intersection labeled with v2, with t as the P (accepting) component.
- Complexity: O(n2p2mnm).

- Use Angluin’s algorithm for learning an automaton.
- The learning algorithm queries whether some strings are in the automaton B.
- It can also conjecture an automaton Miand asks for a counterexample.
- It then generates an automaton with more states Mi+1and so forth.

- Start the learning algorithm.
- Queries are just experiments to B.
- For a conjectured automaton Mi , check if Mi P =
- If so, we check conformance of Mi with B (Vasilevskii algorithm).
- If nonempty, it contains some v1 (v2)w . We test B with v1 (v2)n+1. If this succeeds: error, otherwise, this is a counterexample for Mi .

Incremental

learning

discrepancy

false negative

Model

Path

ModelChecking

no counterexample

counterexample

black boxtesting

Comparing

counterexample

System

actual error

conformance established

Report

error

No error

found

- l - real size of B.
- n - an upper bound of size of B.
- p - size of alphabet.
- Lower bound: reachability is similar to deadlock.
- O(l 3 p l + l 2mn) if there is an error.
- O(l 3 p l + l 2 n p n-l+1+ l 2mn) if there is no error.
If n is not known, check while time allows.

- Basic system written in SML (by Alex Groce, CMU).
- Experiment with black box using Unix I/O.
- Allows model-free model checking of C code with inter-process communication.
- Compiling tested code in SML with BBC program as one process.

- Black box checking is a combination of testing and model checking.
- If a tight bound on size of B is given: learn B first, then do model checking.
- Tight lower bound on complexity, up to polynomial factor.
- Use of games of incomplete information to model testing problems.