1 / 32

Simulation based verification : coverage

Verification coverage estimates the quality of simulation. Simulation based verification : coverage. Coverage metrics. System can’t be simulated with all possible stimuli Question : how many stimuli are needed ? Coverage metrics used

gbillingsley
Download Presentation

Simulation based verification : coverage

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Verification coverage estimates the quality of simulation Simulation based verification: coverage Digitaalsüsteemide verifitseerimise kursus

  2. Coverage metrics • System can’t be simulated with all possible stimuli • Question: how many stimuli are needed? • Coverage metrics used • Coverage shows our confidence in how thoroughly we verified Digitaalsüsteemide verifitseerimise kursus

  3. Coverage metrics • Three types: • Code coverage • Parameter coverage • Functional coverage • Also will talk about • Domain coverage, cross coverage, stuck-at fault coverage Digitaalsüsteemide verifitseerimise kursus

  4. Coverage metrics: code coverage • Code coverage shows how thoroughly is the code examined during simulation • Based on implementation only • Easy to compute Digitaalsüsteemide verifitseerimise kursus

  5. Coverage metrics: parameter coverage • How well parameters and dimensions covered in functional blocks • Based on implementation, spec or both Digitaalsüsteemide verifitseerimise kursus

  6. Coverage metrics: functional coverage • Based on behaviors derived from the spec • Shows the number of covered behaviors • Problem: how to formally generate the behaviors and guarantee that the full functionality is covered… Digitaalsüsteemide verifitseerimise kursus

  7. Comparison: coverage metrics • None of the three types is universally best • Advantages of code coverage are simplicity and formalisation • Advantage of functional coverage is high verification power • Parameter coverage is a trade-off between the two • Coverage metrics may be used for guiding test generation • To reach 100 % coverage is very much harder than to reach 90 %... Digitaalsüsteemide verifitseerimise kursus

  8. Code coverage: statement coverage • All lines exceptbegin, end, else • Example: a > x, x = y • Statement coverage 8 out of 10 = 80% Digitaalsüsteemide verifitseerimise kursus

  9. Code coverage: block coverage Digitaalsüsteemide verifitseerimise kursus

  10. Instrumented code • Instrumented code applied in code coverage analysis. Thus, simulation slowed down considerably. • Turn off the code already covered by simulation • In that case almost no extra time penalty towards the end of simulation Digitaalsüsteemide verifitseerimise kursus

  11. Instrumented code Digitaalsüsteemide verifitseerimise kursus

  12. Code coverage: branch and path • More stringent than statement coverage • Path coverage does not scale Digitaalsüsteemide verifitseerimise kursus

  13. Code coverage: expression coverage • Again, more stringent than statement coverage • We analyze which expressions were evaluated • Expressions divided into layers: E = (x1x2 + x3x4) layer 1: E = y1 + y2 layer 2: y1 = x1x2, y2 = x3x4 • With multi-layer expressions, either the top layer or all layers analyzed Digitaalsüsteemide verifitseerimise kursus

  14. f = x1 & x2 f = (x > y) f = (y ? x1 : x2) x > y x1 x2 - y meaning x1 x2 0 0 1 0 0 - 1 x <= y 1 1 0 0 1 0 1 x > y 0 1 1 1 1 0 1 0 0 0 Code coverage: expression coverage • Minimum input tables applied: • Expression x1 & x2 and two input stimuli: (x1=1,x2=0) and (x1=1,x2=1) • In that case the expression coverage for & is 66.67% Digitaalsüsteemide verifitseerimise kursus

  15. Code coverage: state coverage • Sequence b,a,b,a,b,a,b,a,... gives 66.67% of state coverage, as S1 and S2 traversed(2 out of 3). • S4 is illegal state, thus max state coverage is 75%! Digitaalsüsteemide verifitseerimise kursus

  16. Code coverage: transition coverage • Sequence b,a,b,a,b,a,b,a,... gives 40% of transition coverage as 2 out of 5 transitions traversed. • Transition S2/b not defined! • Calculated by transitions specified (5) or all possible transitions (6) Digitaalsüsteemide verifitseerimise kursus

  17. Code coverage: sequence coverage • User can specify state sequences at the basis of the coverage calculation • Normally essential functionality or corner cases specified • Possible to keep track on which desired (or undesired) sequences were simulated Digitaalsüsteemide verifitseerimise kursus

  18. Code coverage: toggle coverage • Measures how many bits in the signals changed their state between 0 and 1 • E.g. a signal for a 4-bit bus and values 0000 and 1101 would give a toggle coverage of 75% Digitaalsüsteemide verifitseerimise kursus

  19. Parameter Range Server configuration {(S1,S2): S1, S2 passive or active} Task processing time {3,...,6} Length of the queue {(m1,m2,m3): 1  mi8, 1  i 3 } Parameter coverage Digitaalsüsteemide verifitseerimise kursus

  20. Parameter Range {(S1,S2): S1, S2 passive or active} {3,...,6} {(m1,m2,m3): 1  mi8, 1  i 3 } Parameter coverage: example Server configuration Task processing time Length of the queue • If simulation provides 2 server configurations: (S1active,S2 passive) ja (S1active, S2 active); processing times 3, 4 and 5; maximum que lengths (6,4,8), then ... • ... Parameter coverages: server conf. 50 %, processing times 75 %, queue lengths 75%,50%,100% Digitaalsüsteemide verifitseerimise kursus

  21. Functional coverage • Functional coverage based on behaviors derived from the specification • Functional coverage reports the ratio of covered behaviors • Problem: how to formally generate behaviors and guarantee that the entire functionality is covered Digitaalsüsteemide verifitseerimise kursus

  22. Functional coverage • User defined mapping of each functional feature to a ‘cover point’ • Cover points have certain conditions (ranges, defined transitions or cross etc.) • Conditions for a cover point are defined in form of ‘bins’. Digitaalsüsteemide verifitseerimise kursus

  23. Functional coverage • During simulation, when conditions of a coverpointhit, those bins are covered. • A number of cover points can be captured under one covergroup’ • Collection of cover groups is usually called a ‘functional coverage model’.  Digitaalsüsteemide verifitseerimise kursus

  24. Cross coverage • Cartesian product of 2 or more coverages • More stringent than using just a single coverage metric • Cartesian product of several coverages may represent a huge space Digitaalsüsteemide verifitseerimise kursus

  25. out=1 in2 out=0 in1 Domain coverage (boundary coverage) • Every condition defines a partition of the input domain • Small design error may slightly shift the partitioning foo (in1, in2) if (in1 < in2) out = 1; else out = 0; • What if in1 < in2 should be rather in1 <= in2? • What if the value of in1 or in2 is slightly wrong? Digitaalsüsteemide verifitseerimise kursus

  26. in2 out=1 out=0 2 1 1 in1 Domain coverage: Setting up a test • Testing with a value at the boundary and with a value immediately off the boundary foo (in1, in2) if (in1 < in2) out = 1; else out = 0; • Test 1: in1, in2 = 1,1 – false, at the boundary • Test 2: in1, in2 = 1,2 – true, immediately off the boundary Digitaalsüsteemide verifitseerimise kursus

  27. Domain coverage: example • Per each condition (predicate) two tests: • True and immediately off the boundary • False and immediately off the boundary Test 1: in1, in2 = 0,0 • c = 0, and a = 0 • falseand at the boundary forc < a • falseand at the boundary forc < in2 a = in1 + in2; b = 0; c = 0; while (c < a) c = c + in1; if (c < in2) out = a + b; else out = a + c; Test 2: in1, in2 = 1,0 • c = 0, and a = 1 • true and at the boundary forc < a • c set to 1 • falseand at the boundary forc < in2 • 3 conditions out of 4 satisfied, 75% domain coverage Digitaalsüsteemide verifitseerimise kursus

  28. in2 2 in1 in2 in1 Issues with domain coverage • Non-linear boundaries Partly linear boundaries: test each part separately (in2 > in1) and (in1 > 2) Higher order boundaries: no standard solutions (in2 > in12) Digitaalsüsteemide verifitseerimise kursus

  29. Observation coverage • Code coverage ignores the observability aspect • It may happen that the covered code items have no influence to any observable point • It has been showed that 100% code coverage may result in 60-80%observation coverage in the worst case [Fallah, Devadas, Keutzer DAC’98]. • Thus, we should consider propagating the bug effect to an observable output Digitaalsüsteemide verifitseerimise kursus

  30. Other metrics: stuck-at coverage • Assume that a bug manifests itself as a signal line being constantly stuck to 0 (or 1) • Assune single faults, i.e. only one signal line is faulty at a time • Stuck-at coverage indicates how well we have excercised the structure Digitaalsüsteemide verifitseerimise kursus

  31. Issues with coverage • None of the coverage metrics provides 100% of confidence in correct functionality • E.g. 100 % stuck-at coverage test for OR gatedoes not discover the design error OR XOR = Digitaalsüsteemide verifitseerimise kursus

  32. Issues with coverage • Similar problem with expression coverage • 100 % expression coveragefor OR gatedoes not discover the error OR XOR = Thus, we need to consider a set of coverages! Digitaalsüsteemide verifitseerimise kursus

More Related