model checking and testing combined n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Model Checking and Testing combined PowerPoint Presentation
Download Presentation
Model Checking and Testing combined

Loading in 2 Seconds...

play fullscreen
1 / 71

Model Checking and Testing combined - PowerPoint PPT Presentation


  • 84 Views
  • Uploaded on

Model Checking and Testing combined. Doron Peled, University of Warwick. Want to verify hardware and code . Want to perform verification automatically . Manual methods are time consuming , difficult . Restricting to finite state systems. Willing to give up exhaustiveness .

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Model Checking and Testing combined' - marv


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
model checking and testing combined
Model Checking and Testingcombined

Doron Peled,

University of Warwick

why model checking
Want to verify hardware and code.

Want to perform verification automatically. Manual methods are time consuming, difficult.

Restricting to finite state systems.

Willing to give upexhaustiveness.

Checking a (mathematical) model of a system, not the system itself.

Want to obtain counterexamples.

Why “model checking”?
a transition system
A transition system
  • A (finite) set of variables V.
  • A set of states .
  • A (finite) set of transitions T, each transition et has
    • an enabling condition e and
    • a transformation t.
  • An initial condition p.
  • Denote that s’ is a successor of s by R(s,s’).
the interleaving model
The interleaving model
  • An execution is a finite or infinite sequence of states s0, s1, s2, …
  • The initial state satisfies the initial condition, i.e., p(s0).
  • Moving from one state si to si+1 is by executing a transition et:
    • e(si), i.e., si satisfies e.
    • si+1 is obtained by applying t to si.
slide5
L0:While True do

NC0:wait(Turn=0);

CR0:Turn=1

endwhile ||

L1:While True do

NC1:wait(Turn=1);

CR1:Turn=0

endwhile

T0:PC0=L0PC0=NC0

T1:PC0=NC0/\Turn=0

PC0:=CR0

T2:PC0=CR0

(PC0,Turn):=(L0,1)

T3:PC1==L1PC1=NC1

T4:PC1=NC1/\Turn=1

PC1:=CR1

T5:PC1=CR1

(PC1,Turn):=(L1,0)

Initially: PC0=L0/\PC1=L1

the state space

Turn=0

L0,L1

Turn=1

L0,L1

Turn=0

L0,NC1

Turn=0

NC0,L1

Turn=1

L0,NC1

Turn=1

NC0,L1

Turn=0

NC0,NC1

Turn=0

CR0,L1

Turn=1

L0,CR1

Turn=1

NC0,NC1

Turn=0

CR0,NC1

Turn=1

NC0,CR1

The state space
invariant pc0 cr0 pc1 cr1

Turn=0

L0,L1

Turn=1

L0,L1

Turn=0

L0,NC1

Turn=0

NC0,L1

Turn=1

L0,NC1

Turn=1

NC0,L1

Turn=0

NC0,NC1

Turn=0

CR0,L1

Turn=1

L0,CR1

Turn=1

NC0,NC1

Turn=0

CR0,NC1

Turn=1

NC0,CR1

Invariant: (PC0=CR0/\PC1=CR1)
how can we check the model
How can we check the model?
  • The model is a graph.
  • The specification should refer the the graph representation.
  • Apply graph theory algorithms.
what properties can we check without using temporal s pecification
What properties can we check without using temporal specification?
  • Invariants: a property that needs to hold in each state.
  • Deadlock detection: can we reach a state where the program is blocked?
  • Dead code: does the program have parts that are never executed.
how to perform the check
How to perform the check?
  • Apply a search strategy (Depth first search, Breadth first search).
  • Check states/transitions during the search.
  • If property does not hold, report counterexample!
  • DFS – on-the-fly verification.
  • BFS – finds the shortest counterexample.
if it is so good why learn deductive verification methods
If it is so good, why learn deductive verification methods?
  • Model checking works for finite state systems. Would not work with
    • Unconstrained integers.
    • Unbounded message queues.
    • General data structures:
      • queues
      • trees
      • stacks
    • parametric algorithms and systems.
  • Some new algorithms for infinite systems.
the state space explosion
The state space explosion
  • Need to represent the state space of a program in the computer memory.
    • Each state can be as big as the entire memory!
    • Many states:
      • Each integer variable has 2^32 possibilities. Two such variables have 2^64 possibilities.
      • In concurrent protocols, the number of states usually grows exponentially with the number of processes.
if it is so constrained is it of any use
If it is so constrained, is it of any use?
  • Many protocols are finite state.
  • Many programs or procedures are finite state in nature. Can use abstraction techniques.
  • Sometimes it is possible to decompose a program, and prove part of it by model checking and part by theorem proving.
  • Many techniques to reduce the state space explosion (BDDs, Partial Order Reduction).
depth first search
Program DFS

For each s such that q(s)

dfs(s)

end DFS

Procedure dfs(s)

for each s’ such that R(s,s’) do

If new(s’) then dfs(s’)

end dfs.

Depth First Search
start from an initial state
Start from an initial state

Hash table:

q1

q1

q2

q3

Stack:

q1

q4

q5

continue with a successor
Continue with a successor

Hash table:

q1

q1 q2

q2

q3

Stack:

q1

q2

q4

q5

one successor of q2
One successor of q2.

Hash table:

q1

q1 q2 q4

q2

q3

Stack:

q1

q2

q4

q4

q5

backtrack to q2 no new successors for q4
Backtrack to q2(no new successors for q4).

Hash table:

q1

q1 q2 q4

q2

q3

Stack:

q1

q2

q4

q5

backtracked to q1
Backtracked to q1

Hash table:

q1

q1 q2 q4

q2

q3

Stack:

q1

q4

q5

second successor to q1 q4 has been already visited
Second successor to q1q4 has been already visited.

Hash table:

q1

q1 q2 q4 q3

q2

q3

Stack:

q1

q3

q4

q5

backtrack again to q1
Backtrack again to q1.

Hash table:

q1

q1 q2 q4 q3

q2

q3

Stack:

q1

q4

q5

how can we check properties with dfs
How can we check properties with DFS?
  • Invariants: check that all reachable statessatisfy the invariant property. If not, showa path from an initial state to a bad state.
  • Deadlocks: check whether a state where noprocess can continue is reached.
  • Dead code: as we progress with the DFS, mark all the transitions that are executed at least once.
want to do more
Want to do more!
  • Want to check more properties.
  • Want to have a unique algorithm to deal with all kinds of properties.
  • This is done by writing specification is temporal logics.
  • Temporal logic specification can be translated into graphs (finite automata).
temporal logic
Temporal Logic
  • First order logic or propositional assertions describe a state.
  • Modalities:
    • <>p means p will happen eventually.
    • []p means p will happen always.

p

p

p

p

p

p

p

p

more temporal logic
More temporal logic
  • p U q– p has to hold until q holds.

p

p

p

p

q

  • []<>p --- its always the case that there is a later p, i.e., p happens infinitely often.
  • <>[]p --- At some point p will hold forever, i.e., p is stable.
  • <>p/\<>q both p and q would happen eventually.Note, this is not the same as <>(p/\q).
turn 0 turn 1

Turn=0

L0,L1

Turn=1

L0,L1

Turn=0

L0,NC1

Turn=0

NC0,L1

Turn=1

L0,NC1

Turn=1

NC0,L1

Turn=0

NC0,NC1

Turn=0

CR0,L1

Turn=1

L0,CR1

Turn=1

NC0,NC1

Turn=0

CR0,NC1

Turn=1

NC0,CR1

[](Turn=0<>Turn=1)
correctness condition
Correctness condition
  • We want to find a correctness condition for a model to satisfy a specification.
  • Language of a model: L(Model)
  • Language of a specification: L(Spec).
  • We need: L(Model)  L(Spec).
correctness
Correctness

Sequencessatisfying Spec

Program executions

All sequences

incorrectness

Counterexamples

Sequencessatisfying Spec

Program executions

All sequences

Incorrectness
  • Counterexamples are sometimes more interesting and useful than finding that the program is “correct” due to:
  • Underspecification
  • Modeling errors
  • Algorithm and tool limitation
how to check correctness
How to check correctness?
  • Show that L(Model)  L(Spec).
  • Equivalently: ______Show that L(Model)  L(Spec) = Ø.
  • Also: can obtain L(Spec) by translating from LTL!
what do we need to know
What do we need to know?
  • How to intersect two automata?
  • How to complement an automaton?Well, not really, if the specification is given in LTL, we can negate the specification and then translate.
  • How to translate from LTL to an automaton?
b chi automata w automata
Büchi automata (w-automata)

S - finite set of states.

S0S - initial states.

S - finite alphabet.

d S  SS - transition relation.

F S - accepting states.

Accepting run: passes a state in F infinitely often.

System automata: F=S.

example check a

a

a

a, a

Example: check a

<>a

example check a1
Example: check <>a

a

<>a

a

a

a

example check a2
Example: check <>a

a, a

<>a

a

a

Use automatic translation algorithms, e.g.,

[Gerth,Peled,Vardi,Wolper 95]

slide36

init

Turn=0

L0,L1

Turn=1

L0,L1

Turn=0

L0,L1

Turn=1

L0,L1

  • Add an initial node.
  • Propositions are attached to incoming nodes.
  • All nodes are accepting.

Technicality…

system

a

s1

s2

c

b

s3

System

All states are accepting! = no fairness conditions

every element in the product is a counter example for the checked property

a

s1

s2

a

s1,q1

s2,q1

c

b

b

a

s3

s1,q2

s3,q2

c

Every element in the product is a counter example for the checked property.

a

q1

a

a

a

q2

Acceptance isdetermined byautomaton P.

how to check for non emptiness

a

s1,q1

s2,q1

b

a

s1,q2

s3,q2

c

How to check for (non)emptiness?
nonemptiness
Nonemptiness...

Need to check if there exists an accepting run, i.e., infinite sequence that passes through an accepting state infinitely often.

finding accepting runs
Finding accepting runs

If there is an accepting run, then at least one accepting state repeats on it forever. This state must appear on a cycle. So, find a reachable accepting state on a cycle.

equivalently
Equivalently...

A strongly connected component: a maximal set of nodes where each node is reachable by a path from each other node. Find a reachable strongly connected component with an accepting node.

how to complement
How to complement?
  • Complementation is hard!
  • Can ask for the negated property (the sequences that should never occur).
  • Can translate from LTL formula  to automaton A, and complement A. But: can translate ¬ into an automaton directly!
model checking under fairness
Model Checking under Fairness

Express the fairness as a property φ.To prove a property ψ under fairness,model check φψ.

Counter

example

Fair (φ)

Bad (¬ψ)

Program

model checking under fairness1
Model Checking under Fairness

Specialize model checking. For weak process fairness: search for a reachable strongly connected component, where for each process P either

  • it contains on occurrence of a transition from P, or
  • it contains a state where P is disabled.
conformance testing
Conformance Testing
  • Unknown deterministic finite state system B.
  • Known: n states and alphabet .
  • An abstract model C of B. C satisfies all the properties we want from B. C has m states.
  • Check conformance of B and C.
  • Another version: only a bound n on the number of states l is known.

model checking testing
Given Finite state system B.

Transition relation of B known.

Property represent by automaton P.

Check if L(B)  L(P)=.

Graph theory or BDD techniques.

Complexity: polynomial.

Unknown Finite state system B.

Alphabet and number of states of B or upper bound known.

Specification given as an abstract system C.

Check if B C.

Complexity: polynomial if number states known. Exponential otherwise.

Model Checking / Testing
black box checking
Property represent by automaton P.

Check if L(B)  L(P)=.

Graph theory techniques.

Unknown Finite state system B.

Alphabet and upper bound on number of states of B known.

Complexity: exponential.

Black box checking



combination lock automaton
Combination lock automaton

Accepts only words with a specific suffix (cdab in the example).

c

d

a

b

s1

s2

s3

s4

s5

Any other input

conformance testing1

b

b

a

a

b

a

b

a

a

b

Cannot distinguish if reduced or not.

Conformance testing
conformance testing cont

a

b

a

Conformance testing (cont.)

When the black box is nondeterministic, we might never test some choices.

conformance testing cont1

a

Conformance testing (cont.)

b

b

a

a

a

a

b

b

a

a

b

Need: bound on number of states of B.

need reliable reset

a

b

b

s1

s2

a

a

s3

Need reliable RESET

Start with a:in case of being in s1or s3we’ll move to s1and cannot distinguish.Start with b:In case of being in s1or s2we’ll move to s2 and cannot distinguish.

The kind of experiment we do affects what we can distinguish. Much like the Heisenberg principle in Physics.

vc algorithm

Reset or homing

Reset or homing

s1

a/1

b/1

s3

s2

Words of length n-m+1

[VC] algorithm
  • Known automaton A has l states.
  • Black box automaton has up to n states.
  • Check each transition. Check that there are no "combination lock" errors.
  • Complexity: O(l2 n p n-l+1).
  • When n=l: O(l3p).

Distinguishing sequences

experiments

reset

a

a

b

b

c

c

try c

try b

a

a

b

b

c

c

a

a

b

c

b

c

fail

Experiments
simpler problem deadlock
Simpler problem: deadlock?
  • Nondeterministic algorithm:guess a path of length  n from the initial state to a deadlock state.Linear time, logarithmic space.
  • Deterministic algorithm:systematically try paths of length n, one after the other (and use reset), until deadlock is reached.Exponential time, linear space.
deadlock complexity
Deadlock complexity
  • Nondeterministic algorithm:Linear time, logarithmic space.
  • Deterministic algorithm:Exponential (pn-1) time, linear space.
  • Lower bound: Exponential time (usecombination lock automata).
  • How does this conform with what we know about complexity theory?
modeling black box checking
Modeling black box checking
  • Cannot model using Turing machines: not all the information about B is given. Only certain experiments are allowed.
  • We learn the model as we make the experiments.
  • Can use the model of games of incomplete information.
games of incomplete information
Games of incomplete information
  • Two players: $-player, -player (here, deterministic).
  • Finitely many configurations C. Including:Initial Ci , Winning : W+ and W- .
  • An equivalence relation @ on C (the $-player cannot distinguish between equivalent states).
  • Labels L on moves (try a, reset, success, fail).
  • The $-player has the moves labeled the same from configurations that are equivalent.
  • Deterministic strategy for the $-player: will lead to a configuration in W+  W-. Cannot distinguish equivalent conference.
  • Nondeterministic strategy: Can distinguish.
modeling bbc as games
Modeling BBC as games
  • Each configuration contains an automaton and its current state (and more).
  • Moves of the $-player are labeled withtry a, reset... Moves of the -player withsuccess, fail.
  • c1@ c2 when the automata in c1and c2 would respond in the same way to the experiments so far.
a naive strategy for bbc
A naive strategy for BBC
  • Learn first the structure of the black box.
  • Then apply the intersection.
  • Enumerate automata with n states (without repeating isomorphic automata).
  • For a current automata and newautomata, construct a distinguishing sequence. Only one of them survives.
  • Complexity: O((n+1)p (n+1)/n!)
on the fly strategy
On-the-fly strategy
  • Systematically (as in the deadlock case), find two sequences v1 and v2 of length <=m n.
  • Applying v1 to P brings us to a state t that is accepting.
  • Applying v2 to P brings us back to t.
  • Apply v1 v2 n to B. If this succeeds,there is a cycle in the intersection labeled with v2, with t as the P (accepting) component.
  • Complexity: O(n2p2mnm).

v1

v2

learning an automaton
Learning an automaton
  • Use Angluin’s algorithm for learning an automaton.
  • The learning algorithm queries whether some strings are in the automaton B.
  • It can also conjecture an automaton Miand asks for a counterexample.
  • It then generates an automaton with more states Mi+1and so forth.
a strategy based on learning pvy
A strategy based on learning [PVY]
  • Start the learning algorithm.
  • Queries are just experiments to B.
  • For a conjectured automaton Mi , check if Mi  P = 
  • If so, we check conformance of Mi with B ([VC] algorithm).
  • If nonempty, it contains some v1 (v2)w . We test B with v1 v2n+1. If this succeeds: error, otherwise, this is a counterexample for Mi .
complexity
Complexity
  • l - real size of B.
  • n - an upper bound of size of B.
  • p - size of alphabet.
  • Lower bound: reachability is similar to deadlock.
  • O(l 3 p l + l 2mn) if there is an error.
  • O(l 3 p l + l 2 n p n-l+1+ l 2mn) if there is no error.

If n is not known, check while time allows.

Average complexity: polynomial.

some experiments
Some experiments
  • Basic system written in SML (by Alex Groce, CMU).
  • Experiment with black box using Unix I/O.
  • Allows model-free model checking of C code with inter-process communication.
  • Compiling tested code in SML with BBC program as one process.
  • Another application: Adaptive Model Checking when the model may not be accurate [GPY].
unit checking gp93
Unit checking [GP93]
  • Check a unit of code, e.g., a bunch of interacting procedures, a-la unit testing.
  • No initial states are given, not finite state, parametric, compositional.
  • Use temporal properties to describe suspicious paths in the execution.
  • Guide the path search with property.
  • Use flow chart instead of “state space”.
  • Cannot check whether a state occurred, use DFS and iterative deepening.
unit testing of code calculating path condition
Unit testing of code:Calculating path condition

(A≠2 /\ X/A>1) /\ (A>1 & B=0)

A>1 & B=0

Need to find a satisfying assignment:

A=3, X=6, B=0

A≠2 /\ X/A>1

no

yes

X=X/A

A≠2 /\ X>1

A=2 | X>1

If deterministic code, starting with such initial values will enforce executing this path

true

no

yes

X=X+1

true

spec at l 2 u at l 2 x y at l 2 at l 2 u at l 2 x 2 y

l2:x:=x+z

l3:x<t

l1:…

Spec: ¬at l2U (at l2/\ xy /\ (¬at l2/\(¬at l2U at l2 /\ x2y )))

xy

l2:x:=x+z

¬at l2

X

=

at l2/\

xy

l3:x<t

x2y

¬at l2

l2:x:=x+z

at l2/\

x2y

Now simplify condition using theorem proving.

test case generation based on ltl specification

LTLAut

Model

Checker

Path

Path condition

calculation

Flowchart

Compiler

Transitions

First order

instantiator

Test

monitoring

Test case generation based on LTL specification
conclusions
Conclusions
  • Model checking is useful for automatically finding errors in hardware/software design.
  • Testing is nonexhaustive yet practical.
  • Combining model checking and testing methods enhances capabilities and alleviates limitations.
  • Black Box Checking allows model checking a system directly, without first modeling it.
  • Unit Checking allows systematic testing of temporal properties of systems.