Using Parallelism to Improve Theorem Prover Interactivity

1. Using Parallelism to Improve Theorem Prover Interactivity David L. Rager May 17, 2010 Committee Members: Warren A Hunt Jr. (Chair), Matt Kaufmann, J Strother Moore, James C Browne, Emmett Witchel

2. Project Goal • Reduce the latency between when a user submits a conjecture and when the user receives useful feedback concerning that conjecture’s provability

3. Outline • Introduction • The Automated Proof Process – the Waterfall • The ways ACL2 currently uses parallelism • Research Contributions • Key Steps • Removing sequential dependencies • Introducing Parallelism into the Proof Process • Adding futures to the underlying implementation language • Adding parallelism abstractions to the logic • Using parallelism in the waterfall • Managing output • Managing user interrupts • Evaluating the Soundness and Performance of our Approach

4. The Proof Process • Named “the waterfall” • We hope to parallelize the application of all but the induction heuristic

5. The Proof Process • Each proof obligation that is not the original goal that needs to go through the waterfall is called a subgoal • We parallelize at the subgoal level because: • Highest level of granularity available without considering parallelizing the proofs of theorems themselves • Parallelizing the proofs of goals is thought to be much less useful • Already attempted parallelizing the rewriter without much practical gains • The waterfall is mostly functional in nature, whereas the code above the waterfall (which includes the induction code) tends to have more side-effects

6. Current Use of Parallelism in ACL2 • Process-level parallelism • GNU’s “make –j#” and cert.pl • Certifies the regression suite in parallel • Plet/pargs/pand/por • User level parallelism • Useful for improving performance of proofs by simulation

7. Research Contributions • Maintain Interactivity • Continue to support ACL2 users’ ability to use the prover despite parallel execution in its proof process • Mechanisms for Early Feedback • Provide feedback to the theorem prover user asap • Could result in super-linear speedup

8. Research Contributions • Improve Support for Lisp-Level Programming • Interface that unifies multi-threading libraries • Provide Parallelism Abstractions • Enrich the built-in theory with primitives that allow parallel execution • spec-mv-let, a better plet, etc. • Evaluate our Approach • Determine the usefulness of parallelizing a modern semi-automatic theorem prover at the subgoal level • Speedup • How we present non-deterministic output

9. Key Steps • Create version of ACL2 without sequential dependencies (e.g., the modification of state and pspv) in the main proof process (the waterfall) • Introduce raw Lisp primitives and ACL2 abstractions necessary to evaluate the waterfall in parallel • Reincorporate the interactive portion of the waterfall (output and interrupts) • Evaluate the soundness and performance of our approach

10. Removing Sequential Dependencies from the Waterfall • State-based Challenges • State is a special type of variable in ACL2 • Used when performing I/O, when performing system calls, etc. • Contains the “logical story” for these side-effects • ACL2 restricts the use of state in the following two ways • ACL2 restricts the name “state” from being used a variable anywhere that it doesn’t represent this one particular instance • ACL2 requires that if state is modified, that the modified state be returned as part of the return value • So, if we remove the modification of state from the waterfall, we will know that the waterfall is (mostly) side-effect free. • Allows us to more easily find the big “gotchas” (e.g., I/O)

11. Removing Sequential Dependencies from the Waterfall • State-based Solutions • Preliminary step: remove I/O from the waterfall • Disable proof techniques that require modifying state (e.g., clause processors and computed hints) • Skip the proofs of libraries that require those techniques • Results in skipping about 7% of the regression suite • Our current thoughts are that we can reinstate most of these techniques as demand occurs

12. Removing Sequential Dependencies from the Waterfall • PSPV-based Challenges • The Prover SPecial Variables (PSPV) data container acts as an accumulator for changes to variables that would be global, if ACL2 were written in a non-functional manner. • Need to find a way to combine these changes that does not negatively affect the soundness or performance of ACL2

13. Removing Sequential Dependencies from the Waterfall • PSPV-based Solutions • If we know how to combine the changes between the two proof steps, do so. • Otherwise, terminate the latter proof step and restart it with the intermediate PSPV value • Results in a computation exactly the same as the serial computation, without a need to combine PSPVs

14. Component 2:Introducing Parallelism • Adding futures to the underlying implementation language (Lisp) • Adding parallelism abstractions to the logic (ACL2’s logic) • Managing output • Managing user interrupts

15. Adding Futures to the Underlying Implementation Language • Create functions for spawning, reading, and terminating the evaluation of futures • (future x) :: X -> Future-structure • (future-read x) :: Future-structure -> X • (future-abort x) :: Future-structure -> C • Relies upon our multi-threading interface that unifies CCL and SBCL features

16. Adding Parallelism Abstractions to the Logic • Modify plet to support multiple values and speculative evaluation • Give the ACL2 user a means to specify that a variable is unused in a particular expression • Therefore, once evaluation enters a branch with such a specified branch, plet can terminate the evaluation of the unnecessary variable values • Already defined plet differently in raw Lisp and the ACL2 logic • This is an enhancement of what we already created [Rager 2008, Rager and Hunt 2009] • Example usage: (plet (((x y) (mv 3 4)) ((q r) (mv 8 9))) (if (equal q 8) (check-vars-unused-and-kill (x y) (+ q r)) (+ x y q r)))

17. Adding Parallelism Abstractions to the Logic • By using this enhanced version of plet, we can create new abstractions, e.g. spec-mv-let • Spec-mv-let automatically performs the check that certain variables are unused and automatically terminates unnecessary computations • Example annotated usage: (spec-mv-let (x y) ;; speculatively evaluate (mv 3 4) (mv 3 4) (mv-let (q r) (mv 8 9) (if (equal q 8) ;; the speculative evaluation is irrelevant, return ;; a value that doesn’t use those results (+ q r) ;; the speculative evaluation is useful, return a ;; value that uses those results (+ x y q r))))

18. Component 3:Managing Interactivity • Original Goal: maintain output consistent between serial and parallel proofs of subgoals • Store the output in a data structure and signal when the next piece of output is ready for printing • I believe this to be feasible, but we would like to do more

19. Managing Output • New Goal: print meaningful output as it becomes available • Print the proof checkpoints as soon as they are computed • There is usually a very minimal amount of checkpoint output compared to the amount of output that occurs with a full proof attempt’s narrative • Users suggest that in the relatively rare case that a user wants to see the full narrative, that they would likely be satisfied with replaying the proof in a serial manner.

20. Managing Output • It is unclear what type of interface changes will be necessary to accommodate non-deterministic output • An investigation into such accommodations is a planned part of this work • An example of one potential interface follows

22. Managing User Interrupts • When a user aborts a proof, all subgoal computations will be gracefully terminated and the parallelism state of ACL2 will be reset to its initial state • When a user attempts to debug a proof: • Automatically interrupt and pause other threads • Automatically order those threads to resume evaluation • Users also have access to these functions that pause and resume subgoal computations

23. Component 4:Evaluating our Approach • Soundness • Performance

24. Evaluating Soundness • A large subset of the regression suite (93%) will be able to pass with parallelism enabled • We rely on our strategic development and use of macros as our main source of credibility • Insert assertions that check for unexpected changes in the program’s state. • This branch of the build will be tuned for interactivity. Those desiring the assurance level of non-parallel ACL2 can run the serial version.

25. Evaluating Performance • Goal: Reduce the latency between when a user submits a conjecture and when the user receives useful feedback concerning that conjecture’s provability • We do not try to improve the performance of proof attempts that take < 1 second

26. Evaluating Performance • We are not concerned with proof attempts that take less than a second to compute, because anything between 0.10 seconds and 1.0 seconds feels the same to a user. • Example performance result for certifying one of the ACL2 libraries: Summary for making books/ordinals/ordinal-addition.cert: Average sequential time was: 6.22s Average parallel time was: 5.09s Sequential minimum was: 6.12s Parallel minimum was: 4.87s Of 10 iterations, the parallel version was faster than the sequential version 10 times.

27. What We’ve Done so Far • Removed the modification of state and output from the waterfall • Modified Lisp library and ACL2 to permit parallel evaluation of the waterfall • Run some preliminary performance benchmarks

28. What’s Left • Reintroducing output in a helpful way • Improving performance • Evaluating our solution

29. Outline • Introduction • The Automated Proof Process – the Waterfall • The ways ACL2 currently uses parallelism • Research Contributions • Key Steps • Removing sequential dependencies • Introducing Parallelism into the Proof Process • Adding futures to the underlying implementation language • Adding parallelism abstractions to the logic • Using parallelism in the waterfall • Managing output • Managing user interrupts • Evaluating the Soundness and Performance of our Approach