Verification Classic Mistakes Verification Leadership Seminar

1. Verification Classic Mistakes Verification Leadership Seminar Racheli Ganot FlexLight Networks

2. Introduction It’s easy to make mistakes when verifying a DUT or planning a testing effort. Some mistakes are made so often, so repeatedly, by so many different people, that they deserve the label Classic Mistakes.

3. The presentation contents is divided to three main topics: • Verification team / personnel issues • Planning the testing efforts • The verificator at work

4. Declarations: Mistake Suggesting solution.

5. Verification team Personnel Issues Who should test?

6. Hire one verificator per two or more designers.   Ideal verification team includes at least equal amount of engineers as a design team, In particular if the designers don’t write tests at all.

7. Using new and inexperienced engineers as verificators.  A bozo in verification is dangerous like a bozo in design ( and even more ).

8. Recruiting verificators from the ranks of failed designers.   A bad designer likely has some work habits that will make him a bad verificator too. For example, someone who makes lots of bugs because he is inattentive to details will miss lots of bugs from the same reason.

9. Relying on designer explanations about his updates.   Suspicion is a good quality for the verificator.

10. Designers can’t test their own code. Designers test their own code all the time and they do find bugs, just not enough of them, which is why we need independent verification. Designers can do a perfectly fine job, finding basic functional bugs, but the verificators are the ones to perform the most thorough tests ( different & complex scenarios, special cases, error cases, interaction with other blocks etc.).

11. Planning the testing effort How should the whole team’s work be organized?

12. Starting verification too late. Starting verification design when starting HDL design. It doesn’t only improve the environment and give time to develop friendly user interface, but also can prevent designer’s bugs (because he knows the cases that will be checked).

13. Working with no methodologies. Defining set of verification rules for the team: • Enable each one to understand and support the others’ code. • Enable smooth interaction between blocks in the full chip level. • Very helpful for new employees.

14. Design block level environment with no preparation / matching for full-chip. These plans are never perfect, but they still decrease the full-chip conversions time.

15. Test plans are biased toward too specific functional testing. Test plans should include real scenarios, i.e. a set of functions in reasonable order.

16. Verification plan review is done by the verificator and the block design owner Including more verificators and designers in these reviews can save a lot of time. It is done by learning from others’ experience, sharing similar features and parts of code.

17. Complete verification of a single module before starting the next one. It is preferable to verify more features for basic functions, than checking deeply less features.

18. Implementing stress tests on the last minute. For example: DUT that works with several channels and has been checked with only a single one, is almost the same as if this DUT hasn’t been checked at all.

19. The Verificator at work Designing, writing, debugging, and maintaining tests.

20. Writing / updating specs at the end of the project (“when we will have time”). Keep your docs open on your desktop during coding and update changes immediately.

21. Writing tests without enough knowledge about the DUT. It’s better to waste two more days of learning and understanding the DUT, instead of ten days of confusion and uncertainty debugging.

22. Effort to find bugs for rare scenarios. Important bug is the one that important to customers.

23. Tests that are understandable only by their owners. Printing clear information and detailed error messages during the test.

24. Paying more attention to running tests than to designing them. • Test review meetings • Adding coverage items • Increase / decrease randomization in tests according to the coverage results

25. Relying on the generation in the checker. Less is better mainly for the modularity in full-chip.

26. Checking that the product does what it’s supposed to do, but not that it doesn’t do what it isn’t suppose to do. Examples: • The output signals need to be checked all the time, not only when a change is expected. • All registers need to be checked after updating a single register (particular test).

27. Repeat same mistakes at the same team. Sharing knowledge and insights can save a lot of the team’s spent time.

28. Failing to take notes for the next project After each project is ended, it is mandatory to take notes and come up with conclusions, and then follow them in the next projects.

29. The End! "If I had to live my life again, I'd make the same mistakes, only sooner". -- Tallulah Bankhead