1 / 59

A Theory of Predicate-complete Test Coverage and Generation

venedict
Download Presentation

A Theory of Predicate-complete Test Coverage and Generation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

    1. A Theory of Predicate-complete Test Coverage and Generation Thomas Ball Testing, Verification and Measurement Microsoft Research May 2004 HP mp 3130 projector – tiny!HP mp 3130 projector – tiny!

    2. Unit Test Generation

    3. Unit Test Generation

    4. Unit Test Generation

    5. Unit Test Generation

    6. MSIL Unit Test Tool a hybrid helper Goal capture developer knowledge ASAP via a strong set of unit tests to form a specification of the code’s behavior How generate tests based on analysis of MSIL symbolic execution + constraint satisfaction runtime analysis to check complicated invariants Facets complements specification-based test generation positive feedback cycle with programmer

    7. What criteria should guide unit test generation?

    8. Control-flow Coverage Criteria Statement/branch coverage widely used 100% coverage ? a bug-free program!! More stringent criteria modified-condition-decision, predicate, data-flow, mutation, path, …

    9. Predicate-complete Testing Predicates relational expression such as (x<0) the expression (x<0) || (y>0) has two predicates predicates come from program and safe runtime semantics Consider a program with m statements and n predicates predicates partition input domain m x 2n possible observable states S Goal of Predicate-complete Testing: cover all reachable observable states R ? S

    10. PCT Coverage L2: if (A || B) S else T L3: if (C || D) U else V PCT requires covering all logical combinations over {A,B,C,D} at L2 and L3 S, T, U and V Some combinations may not be reachable

    11. PCT Coverage does not imply Path Coverage

    12. PCT Coverage does not imply Path Coverage

    13. PCT Coverage does not imply Path Coverage

    14. PCT Coverage does not imply Path Coverage

    15. Path Coverage does not imply PCT Coverage We assume that output matters.We assume that output matters.

    16. Path Coverage does not imply PCT Coverage

    17. Denominator Problem Coverage metrics require a denominator e.g. statements executed / total statements Easy to define for observable states executed observable states / (m x 2n) But (m x 2n) is not a very good denominator! most observable states will not be reachable R <<< S

    18. Upper and Lower Bounds

    19. Overview Upper and lower bounds Example Test case generation Refinement Discussion Conclusions

    21. Predicate Abstraction if Q ? SP(P,s) then (P,Q) ? onto Note that P => WP(s,Q) is equivalent to SP(s,P) => Q However, Q=>SP(s,P) is not equivalent to WP(s,Q) => PNote that P => WP(s,Q) is equivalent to SP(s,P) => Q However, Q=>SP(s,P) is not equivalent to WP(s,Q) => P

    23. Upper Bound: May-Reachability

    24. Upper Bound: May-Reachability

    28. Postdominance pd(a) = { a } ? ? pd(b) : (a,b) ? may mention LFP vs GFPmention LFP vs GFP

    29. Postdominance pd(a) = { a } ? ? pd(b) : (a,b) ? may mention LFP vs GFPmention LFP vs GFP

    30. Postdominance pd(a) = { a } ? ? pd(b) : (a,b) ? may mention LFP vs GFPmention LFP vs GFP

    31. Postdominance pd(a) = { a } ? ? pd(b) : (a,b) ? may mention LFP vs GFPmention LFP vs GFP

    32. Postdominance pd(a) = { a } ? ? pd(b) : (a,b) ? may mention LFP vs GFPmention LFP vs GFP

    33. Optimistic Lower Bound

    34. Overview Upper and lower bounds Example Test case generation Refinement Discussion Conclusions

    35. Example Note that to uncover error requires multiple executions of lo++Note that to uncover error requires multiple executions of lo++

    36. Observation Vector [ lo<hi, lo<=hi, a[lo]<=pivot, a[hi]>pivot ] lo<hi ? lo<=hi ?lo<hi ? lo<=hi ? (a[lo]<=pivot ? ?a[hi]>pivot) ? (?a[lo]<=pivot ? a[hi]>pivot) Only 10/16 observations possible

    38. Boolean Program

    39. State Space of Boolean Program

    40. Make sure to describe graph in full detail labeling of nodes coloring = lower boundMake sure to describe graph in full detail labeling of nodes coloring = lower bound

    41. Overview Upper and lower bounds Example Test case generation Refinement Discussion Conclusions

    42. Test Generation DFS of Lp generates covering set of paths Symbolically execute paths to generate tests Run program on tests to find errors and compute coverage of observable states

    45. Generated Inputs (L0:TTTT,L4:FTFT) { 0,-8,1 } (L0:TTTT,L4:TTFT) { 0,-8,2,1 } (L0:TTTT,L4:TTTT) { 0,-8,-8,1 } (L0:TTTF,L4:TTFF) { 1,-7,3,0 } (L0:TTTF,L4:FTTF) { 0,-7,-8 } (L0:TTTF,L4:TTTF) { 1,-7,-7,0 } (L0:TTFT,L7:TTFF) { 0,2,-8,1 } (L0:TTFT,L7:FTFT) { 0,1,2 } (L0:TTFT,L7:TTFT) { 0,3,1,2 } (L0:TTFF,L0:TTTT) { 1,2,-1,0 }

    46. Results Buggy partition function U=49, L=43, Tested=42 Fixed partition function U=56, L=37, Tested=43 What about the remaining 13 states?

    47. Overview Upper and lower bounds Example Test case generation Refinement Discussion Conclusions

    48. Unreachable State

    49. Refinement

    50. New Observation Vector [ lo<hi, lo<=hi, lo=hi+1, a[lo]<=pivot, a[hi]>pivot, a[lo-1]<=pivot, a[hi+1]>pivot ] Only 48/128 observations possible For this set of predicates, Lp = U

    51. Overview Upper and lower bounds Example Test case generation Refinement Discussion Conclusions

    52. Discussion Comparison to bisimulation Completeness of abstractions Related work

    53. Bisimulation

    54. Bisimulation

    55. Abstraction Completeness

    56. Abstraction Completeness

    57. Related Work Test coverage criteria Symbolic execution/constraint satisfaction Abstraction-guided test generation Three-valued model checking Modal transition systems

    58. Conclusions PCT coverage new form of state-based coverage similar to path coverage but finite Upper and lower bounds computed using predicate abstraction and modal transitions use lower bound to guide test generation refine bounds

    59. Directions How does PCT work in practice? MUTT implementation underway Other problems the oracle problem specification inference compositional test generation object construction

    60. For More Information http://research.microsoft.com/tvm/ A theory of predicate-complete test coverage and generation MSR-TR-2004-28, April 2004 to appear Third International Symposium on Formal Methods for Components and Objects (FMCO 2004), November 2004

More Related