On Bridging Simulation and Formal Verification

1. On Bridging Simulation and Formal Verification Eugene Goldberg Cadence Research Labs (USA) VMCAI-2008, San Francisco, USA

2. Summary • Introduction • motivation • main idea • Stable sets of points (SSPs) and proofs • checking if a set of points contains an SSP • simulation based proof system • extraction of sufficient test sets from proofs • Generation of practical tests • test generation for design change/manufacturing faults • theoretical and experimental justification of our approach • Conclusion

3. Motivation Given a design,formal verification is to check if property  holds at all points of the search space. Simulation is to “informally” check if property  holds by testing it for a (relatively small) set of points. Simulation is still the main workhorse of verification. It works surprisingly well taking into account that only negligible part of the search space is sampled!! In this paper, we study this phenomenon by the example of the satisfiability problem. As an application, we consider testing combinational circuits.

4. Simulation in Terms of SAT Let F be a CNF formula (i.e. conjunction of disjunctions also called clauses). Satisfiability problem (SAT): Given F, find an assignment psuch that F(p)=1 or prove that F(all_space)=0. Simulation: Given a CNF formula F, find a set of points T s.t. if F(T) = 0,then F is (most likely) unsatisfiable. In other words, find T such that F(T)=0 implies F(all_space) = 0

5. Interpretation of Points as Tests Let N be a circuit. Let X,Y describe the input and internal variables of N. Let z describe the output variable of N. Let CNF formula FN(X,Y,z) specify N.Checking satisfiability of N reduces to checking the satisfiability of FN  z. z N A point p is a complete assignment to the variables X  Y  {z}. The corresponding test t is the projection of p to X. Y …. So p consists of a test t and a computation vector (y,z). Since p falsifies clauses of FN it is computation with a fault. X

6. Main Idea We formulate simulation as a proof system. Instead of proving one “big” theorem (F implies an empty clause), we deduce a sequence of simpler lemmas (F implies a clause L). Each derived clause is added to F. In contrast to “regular” simulation, we use points not only for sampling search space but for derivation. In this derivation we employ the machinery of Stable Sets of Points.

7. Summary • Introduction • motivation • main idea • Stable sets of points (SSPs) and proofs • checking if a set of points contains an SSP • simulation based proof system • extraction of sufficient test sets from proofs • Generation of practical tests • test generation for design change/manufacturing faults • theoretical and experimental justification of our approach • Conclusion

8. A Stable Set of Points (SSP) A complete assignment  a point. Example of 1-neighborhood of point p w.r.t a clause C (written Nbhd(p,C)) such that C(p)=0. C = x  z, p=(x=0,y=0,z=0). Nbhd(p,C) = {p1,p2}, where p1=(x=1,y=0,z=0) and p2=(x=0,y=0,z=1). Let F be a CNF formula and T be a set of points. T is called an SSP if  p  T,  clause C of F such that Nbhd(p,C)  T. A CNF formula F is unsatisfiable iff there is a set of points that is an SSP.

9. Testing if a Set of Points Contains SSP Let F be a CNF formula and T be a set of points. To check if T is an SSP : For every point p of T, check if there is a clause C of F s.t. Nbhd(p,C)  T. This procedure is linear both in T and F. Even if T is not an SSP, a subset T* of T may be an SSP. To check if T contains an SSP : For every point p of T, check if there is a clause C of F s.t. Nbhd(p,C)  T. If C does not exist, remove p from T and restart this procedure. This procedure is quadratic in T and linear in F.

10. Simulation as a Proof System We formulate simulation as a proof system with two derivation rules. Rule 1: Generate a point piand derive the value of F(pi). Rule 2: Let T={p1,…,pk} be the current set of points. Derive a clause C such that F implies C and add it to F. Derivation of C is valid only if T contains an SSP of the CNF formula F ~C (and so F  C holds).

11. A Sufficient Test Set Given a CNF formula F, a set of points T is called sufficient if there is a set of lemma clauses L1,…,Lk (Lk=) such that all k derivations F  L1, F  L1  L2 , …. F  L1 …  Lk-1  Lk are valid (i.e. they can be proved by “simulation” using T) In other words, T is a sufficient test set if the set of valid derivations is sufficient to prove that F is unsatisfiable by applyingonly the second rule (no new points are needed).

12. Extracting a Sufficient Test Set from a Resolution Proof Let L={L1,..,Lk} be the set of resolvents of a resolution proof R that F is unsatisfiable. We assume that resolvents Li are numbered in their derivation order (so Lk = ). Then there is always a sufficient test set whose size is  2k (proving that FL1 …  Li-1  Li needs only two points of T). Sufficient test sets can be very small even for very large formulas!!!!

13. “Proving” Resolvent by Building a Stable Set of Two Points Let C be the resolvent of C and C. One can prove that C C C by building an SSP of only two points. Example: Let C = ~x  y  z, C = x  w. The resolvent C is y  z w. To prove C  C C it suffices to show the CNF formula C  C  ~C isunsatisfiable. (Here ~C= ~y  ~z  ~w). After making assignments y=0,z=0,w=0, the formula C  C  ~C turns into ~x  x.Anytwo pointsp and pwithy=0,z=0,w=0, that are different only in the value of variable x is an SSP for C  C  ~C.

14. Summary • Introduction • motivation • main idea • Stable sets of points (SSPs) and proofs • checking if a set of points contains an SSP • simulation based proof system • extraction of sufficient test sets from proofs • Generation of practical tests • test generation for design change/manufacturing faults • theoretical and experimental justification of our approach • Conclusion

15. Applying Our Theory to (design change/manufacturing testing) We cannot apply our theory directly to checking if F is unsatisfiable. (To prove that a set of points is sufficient we need a set of lemma clauses i.e. another proof) However, there are numerous indirect ways to apply our theory. One application is to detect incorrectdesign changes/ manufacturing faults. Let F be a CNF formula. Let R be a proof that F is unsatisfiable. Find a set of points such that most likely detect satisfiable variations of F. (These variations describe tech. faults/design changes.) Such a test set can be extracted from a sufficient set of points specified by R.

16. Tight Sufficient Tests Let T be a sufficient test set for F. Informally, T is tight if every point p Tfalsifies as few clauses as possible. Given a resolution proof L1,..,Lk, a sufficient test set T is built as T1 … Tk. Here Ti is a two point SSP that proves that C  C  Li where C  C are the parents of resolvent Li. When building Ti one can arbitrarily assign variables that are not in C or C. To build a tight sufficient test, free variables are assigned so as to minimize the number of falsified clauses.

17. Big Picture Let F be an unsatisfiable formula. To detect satisfiable variations (“faults” in F) we generate a tight sufficient test set T. Such a set can be extracted, for example, from a resolution proof that F is unsatisfiable. The sufficiency of T guarantees its “completeness”. The tightness of T increases the probability of detec-ting small satisfiable variations. Suppose, for example, that a point p of T falsifies only one clause C of F. Then if F* is a variation of F consisting of disappearanceof C (along with some other changes) it is likely that F*(p) = 1. Hence p detects that F* is satisfiable.

18. “Theoretical” Justification of Our Approach Let N1 and N2 be identical copies of circuit N.Let Rnat be a “natural” resolution proof that their miter is unsatisfiable. Let T={p1,..,pm} be a tight sufficient test set specified by Rnat. We show that the set inp(T)= {inp(p1),…,inp(pm)} detects all (testable) stuck-at faults of N. So, in a sense, our theory “predicts” the high quality of the stuck-at fault model for circuit testing.

19. Experimental Justification We experimentally compared quality of random test sets and test sets extracted from resolution proofs that two copies of a circuit were identical. Resolution proofs were generated by a state-of-the-art SAT-solver. We used MCNC benchmark circuits. We applied these two kinds of test sets (of the same size) to detecting literal appearance faults. Such faults are more “subtle” than stuck-at faults and so are harder to detect. Experiments show that tests extracted from resolution proofs significantly outperform random tests.

20. Conclusion We use SAT to show that there is a close relation between high quality test sets and formal proofs. We show how high quality tests can be extracted from proofs of unsatisfiability represented as a sequence of lemma clauses. • Our approach answers (at least in principle) two important questions. • When does one stop simulation? (When the set of generated points “encrypts” a formal proof). • What is an “ideal” metric in functional verification? (A formal proof.)