Uncovering Deep Bugs: Dynamic Formal FPV Strategies

Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods Erik Seligman CS 510, Lecture 13, February 2009

Agenda • Motivation: FPV and Deep Bounds • 0in’s ‘Dynamic Formal’ FPV • Synopsys ‘Hybrid Formal’ FPV • When FPV Is Not FPV • Some Meta-Comments

Bounded FPV • Majority of FPV work done at low bound • Around 50-100 cycles • Many modules testable this way • Limits of engines: exponential blowup • Good in many (most?) cases • Verification of isolated modules • Tricky bugs often don’t need much depth • with right input & constraints • Cases where deep queues/counters not critical • Double-check with coverage points!

Problem: FPV Bounds • Bounded FPV is ignoring possibilities • Known industry cases of “deep” errors • Deep bugs suspected in some cases • Example: 2 independent 13-bit ctrs • How many combinations possible? • How many cycles needed? • Bugs may be lurking beyond FPV bounds • Maybe too much logic to solve with pruning  Strategies desired for hi-bound FPV

Compromise: Abstraction • Counters, state machines: free outputs • As we saw in memory controller example • All values possible • Useful in many cases • But at significant cost • Realism of cases: values jump around • No coverage for abstracted logic

Compromise: Initial State • Set initial state to something ‘interesting’ • Fill queues, set high counters, etc. • FPV no longer exhaustive • Errors from different initial state not covered • Focus is bug hunting in suspicious area • Can we do better?

Limits of Pruning • Pruning often helps complexity issues • In best case, bounded  full proof • Or can deepen bounds • But it can’t help in worst cases • Maybe long, complex interactions of parts • Think about 2-counter example again • Lots of logic may be involved

Key Insight: Leverage Simulation • Simulation env has design knowledge • Usually set up to create interesting states • Wide variety of states possible in sim • Can we leverage simulation for FPV? • Simple example: Snapshot a simulation state, use as FPV input • Are there bigger opportunities?

0in Dynamic Formal FPV • Monitor simulation runs • Tool runs as plugin to simulator • Identify “interesting” states • Hit cover point • Transition counter, state machine, etc. • Launch many low-bound FPV runs

Ordinary FPV & State Space rst • Small proof radius covered after reset

Dynamic Formal Approach rst • Many mini-FPV runs launched from sim • Smaller proof radius for each FPV run

Advantages/Disadvantages • Advantages • Based on real tests  likely good examples • Produce realistic counterexamples since most cycles are from simulation • Finds deep bugs “almost” tested in sim • Disadvantages?

Advantages/Disadvantages • Advantages • Based on real tests  likely good examples • Produce realistic counterexamples since most cycles are from simulation • Finds deep bugs “almost” tested in sim • Disadvantages? • Depends on setting up simulation env • Not usable really early or arbitrary hierarchy • Can’t find counterexamples distant from tests • Misses major selling point of FPV: corner cases not conceived by designer • Tool pain: FPV & sim tools interact? • Harder than it sounds!

Synopsys Magellan Approach • Similar motivations to 0in • Recognize problem: simulation env • Few real designers use ‘raw’ sim tool • Many layers of scripts & wrappers • Really painful to enable 0in-like method • Solution: Random Simulation • Still uses simulation engine • Randomly tries to simulate to get cover pts • No use of real tests like 0in

Hybrid Formal Approach rst • Lots of random simulation paths • Many mini-FPV runs launched from sim • Smaller proof radius for each FPV run

Advantages/Disadvantages • Advantages • Not limited by tests from validation team • Can theoretically find very deep bugs • Independent of simulation environment • Disadvantages?

Advantages/Disadvantages • Advantages • Not limited by tests from validation team • Can theoretically find very deep bugs • Independent of simulation environment • Disadvantages? • Very random: need to get lucky for bug • Loses both comprehensiveness & test-guidance • Good cover points may mitigate • Still have sim/fpv tool integration issues • Many “simulation-synthesis mismatches”: time delays, system functions, non-det behavior, X/Z vals, etc.

Can We Use A Non-Simulation Approach? • FPV user suspects deep error • But proof bounds not enough • Standard complexity techniques not working • What does FPV user know? • Probably has likely scenarios in mind • But FPV tool isn’t getting there • So why not use user-guided FPV? • Try to manually get FPV tool close to error • Then launch bounded, comprehensive run

User-Guided FPV • Create cover points for reset state • Suspected intermediate state on way to err • Ask FPV tool to reach cover point • CEX at cover == reset state for next • Save state manually • Nicer if tool help, but not seen in current tools • After cover pt, use real bounded FPV • Run isn’t full formal proof of properties • But is comprehensive bug hunt from there

User Guided Approach rst • Use FPV engine to visit cover point • Launch bounded FPV from targeted pt

Advantages/Disadvantages • Advantages • Completely eliminates sim engine issues • Can theoretically find very deep bugs • Takes advantage of user knowledge • Disadvantages?

Advantages/Disadvantages • Advantages • Completely eliminates sim engine issues • Can theoretically find very deep bugs • Takes advantage of user knowledge • Disadvantages? • Highly dependent on user intuition • Loses ‘automatic search’ aspect of 0in/Synopsys • Problem areas must be specifically targeted

Deep Bounds: Why ‘Sexy’? • Much effort in industry/academia • Recognition of limits of bounded FPV • Potential for fame and fortune • Really cool to claim find of 10000-cycle-deep error nobody noticed! • Common view: One such error justifies cost of full deployment + BMW for the validator • But is this really the key FPV problem?

Where Do FPV users Spend Their Time? • Intuitive explanation, take with grain of salt • 70% “Wiggling” • Early attempts with quick counterexamples • Root-causing CEXs, adding assumptions, & finding occasional shallow bugs • Many FPV efforts never get beyond this stage • 20% “Solidifying” • Have bounded proofs on most assertions • Incrementally deepening bounds, checking covers • 10% “Heroics” • Extra, expert efforts to get full proofs or deep proofs in suspected problem areas • Trying exotic FPV technologies

Where Should FPV Research Be Concentrated? • Vast majority of FPV effort is “wiggling” • Assuming previous slide roughly accurate! • Then next largest chunk is “solidifying” • Understaning, improving low bounds • Basic complexity issues • What are FPVers actually doing? • Running low-bound proof attempts • Root-causing failures to add assumptions • Attacking basic complexity issues  (IMHO) The real problem is Usability

What Would An FPV Breakthrough Look Like? • Need quantum leaps in productivity of “wiggling” stage • Secondary priority: addressing complexity • Key functionality I want to see • Viewing and understanding counterexamples • More & better live what-if experiments • Intuitive user interaction • Really fast incremental responsiveness • Support for interactive complexity analysis

My Advice To Academia • Better engines & deeper proofs are nice • But only leveraged by small group of engineers who progress to “heroic” FPV stage • Best impact: where engineers spend time • Wiggling stage: quickly understanding cex • Solidifying stage: root-causing & fixing complexity  Focus on usability, interactivity, and incremental resposiveness!

References / Further Reading • http://www.edacafe.com/Vision/200208/design3.html • http://drona.csa.iisc.ernet.in/~deepakd/talks/formal-iisc-0306.pdf • http://www.te.rl.ac.uk/europractice/vendors/magellan_ds.pdf

Uncovering Deep Bugs: Dynamic Formal FPV Strategies