A Tutorial on Automated Verification Tevfik Bultan

1. A Tutorial on Automated VerificationTevfik Bultan

2. 2007 Clarke, Edmund M 2007 Emerson, E Allen 2007 Sifakis, Joseph 1996 Pnueli, Amir 1991 Milner, Robin 1980 Hoare, C. Antony R. 1978 Floyd, Robert W 1972 Dijkstra, E. W. Who are these people and what do they have in common?

3. Outline • Software’s Chronic Crisis • Temporal Logics and Model Checking Problem • Symbolic Model Checking • Automata Theoretic Model Checking • Software Verification Using Explicit State Model Checking with Java Path Finder • Bounded Model Checking • Symbolic Software Model Checking with Predicate Abstraction and Counter-Example Guided Abstraction Refinement

4. Software’s Chronic Crisis Large software systems often: • Do not provide the desired functionality • Take too long to build • Cost too much to build • Require too much resources (time, space) to run • Cannot evolve to meet changing needs • For every 6 large software projects that become operational, 2 of them are canceled • On the average software development projects overshoot their schedule by half • 3 quarters of the large systems do not provide required functionality

5. Software Failures • There is a long list of failed software projects and software failures • You can find a list of famous software bugs at: http://www5.in.tum.de/~huckle/bugse.html • I will talk about two famous and interesting software bugs

6. A software bug caused European Space Agency’s Ariane 5 rocket to crash 40 seconds into its first flight in 1996 (cost: half billion dollars) The bug was caused because of a software component that was being reused from Ariane 4 A software exception occurred during execution of a data conversion from 64-bit floating point to 16-bit signed integer value The value was larger than 32,767, the largest integer storable in a 16 bit signed integer, and thus the conversion failed and an exception was raised by the program When the primary computer system failed due to this problem, the secondary system started running. The secondary system was running the same software, so it failed too! Ariane 5 Failure

7. Ariane 5 Failure • The programmers for Ariane 4 had decided that this particular velocity figure would never be large enough to raise this exception. • Ariane 5 was a faster rocket than Ariane 4! • The calculation containing the bug actually served no purpose once the rocket was in the air. • Engineers chose long ago, in an earlier version of the Ariane rocket, to leave this function running for the first 40 seconds of flight to make it easy to restart the system in the event of a brief hold in the countdown. • You can read the report of Ariane 5 failure at: http://www.ima.umn.edu/~arnold/disasters/ariane5rep.html

8. Priority inversion occurs when a thread that has higher priority is waiting for a resource held by thread with a lower priority Pathfinder contained a data bus shared among multiple threads and protected by a mutex lock Two threads that accessed the data bus were: a high-priority bus management thread and a low-priority meteorological data gathering thread Yet another thread with medium-priority was a long running communications thread (which did not access the data bus) A few days into its mission, NASA’s Mars Pathfinder computer system started rebooting itself Cause: Priority inversion during preemptive priority scheduling of threads Mars Pathfinder

9. Mars Pathfinder • The scenario that caused the reboot was: • The meteorological data gathering thread accesses the bus and obtains the mutex lock • While the meteorological data gathering thread is accessing the bus, an interrupt causes the high-priority bus management thread to be scheduled • Bus management thread tries to access the bus and blocks on the mutex lock • Scheduler starts running the meteorological thread again • Before the meteorological thread finishes its task yet another interrupt occurs and the medium-priority (and long running) communications thread gets scheduled • At this point high-priority bus management thread is waiting for the low-priority meteorological data gathering thread, and the low-priority meteorological data gathering thread is waiting for the medium-priority communications thread • Since communications thread had long-running tasks, after a while a watchdog timer would go off and notice that the high-priority bus management thread has not been executed for some time and conclude that something was wrong and reboot the system

10. Software’s Chronic Crisis • Software product size is increasing exponentially • faster, smaller, cheaper hardware • Software is everywhere: from TV sets to cell-phones • Software is in safety-critical systems • cars, airplanes, nuclear-power plants • We are seeing more of • distributed systems • embedded systems • real-time systems • These kinds of systems are harder to build • Software requirements change • software evolves rather than being built

11. Summary • Software’s chronic crisis: Development of large software systems is a challenging task • Large software systems often: Do not provide the desired functionality; Take too long to build; Cost too much to build Require too much resources (time, space) to run; Cannot evolve to meet changing needs

12. What is this?

13. First Computer Bug • In 1947, Grace Murray Hopper was working on the Harvard University Mark II Aiken Relay Calculator (a primitive computer). • On the 9th of September, 1947, when the machine was experiencing problems, an investigation showed that there was a moth trapped between the points of Relay #70, in Panel F. • The operators removed the moth and affixed it to the log. The entry reads: "First actual case of bug being found." • The word went out that they had "debugged" the machine and the term "debugging a computer program" was born.

14. Can Model Checking Help • The question is: Can the automated verification techniques we have been discussing be used in finding bugs in software systems? • Today I will discuss some automated verification techniques that have been successful in identifying bugs

15. State of the art in automated verification: Model Checking • What is model checking? • Automated verification technique • Focuses on bug finding rather than proving correctness • The basic idea is to exhaustively search for bugs in software • Has many flavors • Explicit-state model checking • Symbolic model checking • Bounded model checking

16. Model Checking Evolution • Earlier model checkers had their own input specification languages • For example Spin, SMV • This requires translation of the system to be verified to the input langauge of the model checker • Most of the time these translations are not automated and use ad-hoc simplifications and abstractions • More recently several researchers developed tools for model checking programs • These model checkers work directly on programs, i.e., their input language is a programming language • These model checkers use well-defined techniques for restricting the state space or use automated abstraction techniques

17. Explicit-State Model Checking Programs • Verisoft from Bell Labs • C programs, handles concurrency, bounded search, bounded recursion. • Uses stateless search and partial order reduction. • Java Path Finder (JPF) at NASA Ames • Explicit state model checking for Java programs, bounded search, bounded recursion, handles concurrency. • Uses techniques similar to the techniques used in Spin. • CMC from Stanford for checking systems code written in C

18. Symbolic Model Checking of Programs • CBMC • This is the bounded model checker we discussed earlier, bounds the loop iterations and recursion depth. • Uses a SAT solver. • SLAM project at Microsoft Research • Symbolic model checking for C programs. Can handle unbounded recursion but does not handle concurrency. • Uses predicate abstraction and BDDs.

19. Beyond Model Checking • Promising results obtained in the model checking area created a new interest in automated verification • Nowadays, there is a wide spectrum of verification/analysis/testing techniques with varying levels of power and scalability • Bounded verification using SAT solvers • Symbolic execution using combinations of decision procedures • Dynamic symbolic execution (aka concolic execution) • Various types of symbolic analysis: shape analysis, string analysis, size analysis, etc.

20. What to Verify? • Before we start talking about automated verification techniques, we need to identify what we want to verify • It turns out that this is not a very simple question • First we will discuss issues related to this question

21. Temporal Logics and Model Checking Problem

22. A Mutual Exclusion Protocol Two concurrently executing processes are trying to enter a critical section without violating mutual exclusion Process 1: while (true) { out: a := true; turn := true; wait: await (!b or !turn); cs: a := false; } || Process 2: while (true) { out: b := true; turn := false; wait: await (!a or turn); cs: b := false; }

23. Reactive Systems: A Very Simple Model • We will use a very simple model for reactive systems • A reactive system generates a set of execution paths • An execution path is a concatenation of the states (configurations) of the system, starting from some initial state • There is a transition relation which specifies the next-state relation, i.e., given a state what are the states that can follow that state

24. State Space • The state space of a program can be captured by the valuations of the variables and the program counters • For our example, we have • two program counters: pc1, pc2 domains of the program counters: {out, wait, cs} • three boolean variables: turn, a, b boolean domain: {True, False} • Each state of the program is a valuation of all the variables

25. State Space • Each state can be written as a tuple (pc1,pc2,turn,a,b) • Initial states: {(o,o,F,F,F), (o,o,F,F,T), (o,o,F,T,F), (o,o,F,T,T), (o,o,T,F,F), (o,o,T,F,T), (o,o,T,T,F), (o,o,T,T,T)} • initially: pc1=o and pc2=o • How many states total? 3 * 3 * 2 * 2 * 2 = 72 exponential in the number of variables and the number of concurrent components

26. Transition Relation • Transition Relation specifies the next-state relation, i.e., given a state what are the states that can come immediately after that state • For example, given the initial state (o,o,F,F,F) Process 1 can execute: out: a := true; turn := true; or Process 2 can execute: out: b := true; turn := false; • If process 1 executes, the next state is (w,o,T,T,F) • If process 2 executes, the next state is (o,w,F,F,T) • So the state pairs ((o,o,F,F,F),(w,o,T,T,F)) and ((o,o,F,F,F),(o,w,F,F,T)) are included in the transition relation

27. Transition Relation The transition relation is like a graph, edges represent the next-state relation (o,o,F,F,F) (w,o,T,T,F) (o,w,F,F,T) (w,w,T,T,T) (o,c,F,F,T)

28. Transition System • A transition system T = (S, I, R) consists of • a set of states S • a set of initial states I  S • and a transition relation R  S  S • A common assumption in model checking • R is total, i.e., for all s  S, there exists s’ such that (s,s’)  R

29. Execution Paths • An execution path is an infinite sequence of states x = s0, s1, s2, ... such that s0  I and for all i  0, (si,si+1)  R Notation: For any path x xi denotes the i’th state on the path (i.e., si) xi denotes the i’th suffix of the path (i.e., si, si+1, si+2, ... )

30. Execution Paths A possible execution path: ((o,o,F,F,F), (o,w,F,F,T), (o,c,F,F,T)) ( means repeat the above three states infinitely many times) (o,o,F,F,F) (w,o,T,T,F) (o,w,F,F,T) (w,w,T,T,T) (o,c,F,F,T)

31. Temporal Logics • Pnueli proposed using temporal logics for reasoning about the properties of reactive systems • Temporal logics are a type of modal logics • Modal logics were developed to express modalities such as “necessity” or “possibility” • Temporal logics focus on the modality of temporal progression • Temporal logics can be used to express, for example, that: • an assertion is an invariant (i.e., it is true all the time) • an assertion eventually becomes true (i.e., it will become true sometime in the future)

32. Temporal Logics • We will assume that there is a set of basic (atomic) properties called AP • These are used to write the basic (non-temporal) assertions about the program • Examples: a=true, pc0=c, x=y+1 • We will use the usual boolean connectives:  ,  ,  • We will also use four temporal operators: Invariantp : G p (aka p) (Globally) Eventuallyp : F p (aka p) (Future) Next p : X p (aka p) (neXt) pUntilq : p U q

33. LTL Properties . . . X p p . . . G p p p p p p p . . . F p p . . . p U q p p p p q

34. Example Properties mutual exclusion: G (  (pc1=c  pc2=c)) starvation freedom: G(pc1=w  F(pc1=c))  G(pc2=w  F(pc2=c)) Given the execution path: x =((o,o,F,F,F), (o,w,F,F,T), (o,c,F,F,T)) x |= pc1=o x |= X (pc2=w) x |= F (pc2=c) x |= (turn) U (pc2=c  b) x |= G (  (pc1=c  pc2=c)) x |= G(pc1=w  F(pc1=c))  G(pc2=w  F(pc2=c))

35. LTL Model Checking • Given a transition system T and an LTL property p T |= p iff for all execution paths x in T, x |= p For example: T |=? G (  (pc1=c  pc2=c)) T |=? G(pc1=w  F(pc1=c))  G(pc2=w  F(pc2=c)) Model checking problem: Given a transition system T and an LTL property p, determine if T is a model for p (i.e., if T |=p) Complexity: (|S|+|R|)  2O(|f|)

36. Linear Time vs. Branching Time • In linear time logics we look at the execution paths individually • In branching time logics we view the computation as a tree • computation tree: unroll the transition relation Transition System Execution Paths Computation Tree s3 s3 s3 s1 s2 s3 s4 s1 s4 s4 s1 s2 s3 s3 s2 . . . s3 s4 s1 s3 . . . . . . . . . . . . s4 s1 . . . . . . . . .

37. Computation Tree Logic (CTL) • In CTL we quantify over the paths in the computation tree • We use the same four temporal operators: X, G, F, U • However we attach path quantifiers to these temporal operators: • A : for all paths • E : there exists a path • We end up with eight temporal operators: • AX, EX, AG, EG, AF, EF, AU, EU

38. CTL Properties Transition System Computation Tree p s3 p p s1 s2 s3 s4 p s1 s4 p s2 s3 s3 |= p s4 |= p s1 |=  p s2 |=  p s3 |= EX p s3 |= EX  p s3 |=  AX p s3 |=  AX  p s3 |= EG p s3 |=  EG  p s3 |= AF p s3 |= EF  p s3 |=  AF  p p p s3 s4 s1 . . . . . . . . . p s4 s1 . . . . . . . . .

39. CTL Model Checking • Given a transition system T= (S, I, R) and a CTL property p T |= p iff for all initial state s  I, s |= p Model checking problem: Given a transition system T and a CTL property p, determine if T is a model for p (i.e., if T |=p) Complexity: O(|f|  (|S|+|R|)) For example: T |=? AG (  (pc1=c  pc2=c)) T |=? AG(pc1=w  AF(pc1=c))  AG(pc2=w  AF(pc2=c)) • Question: Are CTL and LTL equivalent?

40. CTL vs. LTL • CTL and LTL are not equivalent • There are properties that can be expressed in LTL but cannot be expressed in CTL • For example: FG p • There are properties that can be expressed in CTL but cannot be expressed in LTL • For example: AG(EF p) • Hence, expressive power of CTL and LTL are not comparable

41. Symbolic Model Checking

42. Temporal Properties  Fixpoints [Emerson and Clarke 80] Here are some interesting CTL equivalences: AG p = p  AX AG p EG p = p  EX EG p AF p = p  AX AF p EF p = p  EX EF p p AU q = q  (p  AX (p AU q)) p EU q = q  (p  EX (p EU q)) Note that we wrote the CTL temporal operators in terms of themselves and EX and AX operators

43. Fixpoint Characterization Equivalences AG p =  y . p  AX y AG p = p  AX AG p EG p =  y . p  EX y EG p = p  EX EG p AF p =  y . p  AX y AF p = p  AX AF p EF p =  y . p  EX y EF p = p  EX EF p p AU q =  y . q  (p  AX (y)) p AU q=q  (p  AX (p AU q)) p EU q =  y . q  (p  EX (y)) p EU q = q  (p  EX (p EU q)) Fixpoint Characterizations

44. Least Fixpoint Given a monotonic function F, the least fixpoint  y . F y is the limit of the following sequence (assuming F is -continuous): , F , F2 , F3 , ... If S is finite, then we can compute the least fixpoint using the above sequence

45. EF Fixpoint Computation EF p =  y . p  EX y is the limit of the sequence: , pEX , pEX(pEX ) , pEX(pEX(p EX )) , ... which is equivalent to , p, p  EX p , p  EX (p  EX (p) ) , ...

46. EF Fixpoint Computation p s1 s2 s3 s4 p Start  1st iteration pEX  = {s1,s4} EX()= {s1,s4} ={s1,s4} 2nd iteration pEX(pEX )= {s1,s4} EX({s1,s4})= {s1,s4} {s3}={s1,s3,s4} 3rd iteration pEX(pEX(p EX ))= {s1,s4} EX({s1,s3,s4})= {s1,s4} {s2,s3,s4}={s1,s2,s3,s4} 4th iteration pEX(pEX(pEX(p EX )))= {s1,s4} EX({s1,s2,s3,s4})= {s1,s4}  {s1,s2,s3,s4} = {s1,s2,s3,s4}

47. EF Fixpoint Computation EF(p)states that can reach p  p EX(p)  EX(EX(p))  ... p • • • EF(p)

48. Greatest Fixpoint Given a monotonic function F, the greatest fixpoint  y . F y is the limit of the following sequence (assuming F is -continuous): S, F S, F2 S, F3 S, ... If S is finite, then we can compute the greatest fixpoint using the above sequence

49. EG Fixpoint Computation Similarly, EG p =  y . p  EX y is the limit of the sequence: S, pEX S, pEX(p  EX S) , pEX(p  EX (p  EX S)) , ... which is equivalent to S, p, p  EX p , p  EX (p  EX (p) ) , ...

50. EG Fixpoint Computation p p s1 s2 s3 s4 p Start S = {s1,s2,s3,s4} 1st iteration pEX S = {s1,s3,s4}EX({s1,s2,s3,s4})= {s1,s3,s4}{s1,s2,s3,s4}={s1,s3,s4} 2nd iteration pEX(pEX S) = {s1,s3,s4}EX({s1,s3,s4})= {s1,s3,s4}{s2,s3,s4}={s3,s4} 3rd iteration pEX(pEX(pEX S)) = {s1,s3,s4}EX({s3,s4})= {s1,s3,s4}{s2,s3,s4}={s3,s4}