1 / 28

Applications of Optimization to Logic Testing

Applications of Optimization to Logic Testing. Gary Kaminski and Paul Ammann ICST 2010 CSTVA Workshop. Outline. Motivation Test Sets and Coverage Criteria Should We Apply Well Known Optimization Techniques? Testing DNF Predicates For Logic Faults Minimal DNF Summary

chiku
Download Presentation

Applications of Optimization to Logic Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Applications of Optimization to Logic Testing Gary Kaminski and Paul Ammann ICST 2010 CSTVA Workshop

  2. Outline Motivation Test Sets and Coverage Criteria Should We Apply Well Known Optimization Techniques? Testing DNF Predicates For Logic Faults Minimal DNF Summary DNF Faults and Fault Hierarchy Coverage Criteria Optimizing DNF Criteria Results Optimized Test Set Sizes Benefit, Computational Cost Optimizing Individual Tests Implications For Regression Testing Thoughts On Where to Go from Here

  3. Motivation Test Sets and Coverage Criteria Problem 1: Find Test Set that Satisfies Criterion Very Common Approach Often Tests Are Redundant Problem 2: Find Minimal Test Set that Satisfies Criterion Given A Test Set, Throw Out Redundant Tests Now Every Test Is There For a Reason! Problem 3: Find Minimum Test Set that Satisfies Criterion Generally Much Harder Can’t Be Found With “Local” Information Need To Make Globally Optimal Choices Question: Is It Useful To Recast Coverage As Standard Optimization Problem? Gain? Cost? Here We Address This Question In Context Of DNF Logic Testing

  4. Minimal DNF Summary Terms separated by OR, literals by AND ab + a!c vs. a(b + !c) • Make each term true and other terms false ab + ac vs. ab + abc • Impossible to remove a literal without changing the predicate semantics ab vs. abc + ab!c Green – in minimal DNF Red – not in minimal DNF

  5. Nine Minimal DNF Logic Faults Original: ab + bc • Expression Negation Fault (ENF) !(ab + bc) • Operator Reference Fault (ORF) abbc • Operator Reference Fault (ORF) a + b + bc • Term Negation Fault (TNF) !(ab) + bc • Term Omission Fault (TOF) bc • Literal Negation Fault (LNF) !ab + bc • Literal Insertion Fault (LIF) abc + bc • Literal Reference Fault (LRF) ac + bc • Literal Omission Fault (LOF) a + bc

  6. Fault Detection Hierarchy Literal Insertion Fault LIF Term Omission Fault TOF Literal Reference Fault LRF Literal Negation Fault LNF Operator Reference Fault ORF+ Literal Omission Fault LOF Term Negation Fault TNF Operator Reference Fault ORF* Expression Negation Fault ENF Key Fact: Detection Relations Depend on Whether Fault is Feasible

  7. Coverage Criteria Preamble : Unique True Points and Near False Points • UTP: An assignment of values such that only one term evaluates to true. ab + bc: TTF is a UTP for term ab • NFP: An assignment of values such that the predicate evaluates to false but when a literal is omitted, it evaluates to true. ab + bc: TFF is an NFP for b in term ab Closely Related to MCDC Notion

  8. For each implicant find unique true points (UTPs) so that Literals not in term take on values T and F Consider the DNF predicate: f = ab + cd For term ab Choose TTFT, TTTF For term cd Choose FTTT, TFTT MUTP test set {TTFT, TTTF, FTTT, TFTT} MUTP Criterion (Chen, Lau, Yu) ab cd 00 01 11 10 t 00 01 t 11 t t t t 10 t

  9. CUTPNFP Criterion (Chen, Lau, Yu) Consider f = ab + cd For term ab For a, choose UTP, NFP pair TTFF, FTFF For b, choose UTP, NFP pair TTFT, TFFT For term cd For c, choose UTP, NFP pair FFTT, FFFT For d, choose UTP, NFP pair FFTT, FFTF Possible CUTPNFP test set {TTFF, TTFT, FFTT //UTPs FTFF, TFFT, FFFT, FFTF} //NFPs ab cd 00 01 11 10 t 00 01 t 11 t t t t 10 t

  10. Find NFP tests for each literal such that all literals not in the term attain F and T Consider the DNF predicate: f = ab + cd For implicantab Choose FTFT, FTTF for a Choose TFFT, TFTF for b For implicantcd Choose FTFT, TFFT for c Choose FTTF, TFTF for d MNFP test set {TFTF, TFFT, FTTF, TFTF} Example is small, but generally MNFP is large MNFP Criterion (Chen, Lau, Yu) ab cd 00 01 11 10 t 00 01 t 11 t t t t 10 t

  11. MNFP CUTPNFP feasible? For Each Literal In Term MUTP feasible? Test Set = MUTP + MNFP For Each Term Test Set = MUTP + NFP Test Set = MUTP + CUTPNFP MUMCUT/Minimal-MUMCUT Criteria • MUMCUT is simply MUTP + CUTPNFP + MNFP • Minimal-MUMCUT omits tests via low level criterion feasibility analysis • Adds CUTPNFP and MNFP tests only when necessary • Minimal-MUMCUT guarantees detecting LIF, LRF, LOF • And thus all 9 faults in the hierarchy, even if some faults infeasible Key Point: Minimal != Minimum Local vs. Global Perspective

  12. MUTP Optimization • Minimize test set size subject to detecting LRFs with UTPs • LIFs Guaranteed Be Detected When LRFs are Feasible • Each UTP is a 0-1 variable (integer programming) • Each constraint represents the UTPs that detect a LRF • Can reduce number of variables and constraints by focusing on a fault called the TRF-LIF instead of the LRF (see paper for details) ab + cd Minimize x1100 + x1101 + x1110 + x0011 + x0111 + x1011 Subject to x1100 + x1101 >= 1 (cb + cd) x1110 >= 1 (!cb + cd) x1100 + x1110 >= 1 (db + cd) x1101 >= 1 (!db + cd) …….

  13. CUTPNFP Optimization • Minimize test set size subject to finding UTP – NFP corresponding pairs • Each point is a 0-1 variable • Constraints represent an implication that the UTP and NFP be corresponding • Additional constraint to reflect that an UTP must be selected ab + cd Minimize x1100 + x0100 + x1000 + x1101 + … Subject To • x1100 <= M1; -- if 1100 is selected as an UTP then M1 must = 1 • x0100 >= M1; -- if M1 = 1, 0100 must be chosen as a NFP • x1000 >= M1; -- if M1 = 1, 1000 must be chosen as a NFP • x1101 <= M2; -- if 1101 is selected as an UTP then M2 must = 1 • x0101 >= M2; -- if M2 = 1, 0101 must be chosen as a NFP • x1001 >= M2; -- if M2 = 1,1001 must be chosen as a NFP • x1110 <= M3; -- if 1110 is selected as an UTP then M3 must = 1 • x0110 >= M3; -- if M3 = 1, 0110 must be chosen as a NFP • x1010 >= M3; -- if M3 = 1, 1010 must be chosen as a NFP • x1100 + x1101 + x1110 >= 1; -- an UTP must be selected

  14. MNFP Optimization • Minimize test set size subject to detecting LRFs with NFPs • Each NFP is a 0-1 variable • Each constraint represents the NFPs that detect a LRF ab + cd Minimize x0100 + x0101 + x0110 + x1000 + x1001 + x1010 ... Subject to x0110 >= 1 (cb + cd) x0100 + x0101 >= 1 (!cb + cd) x1001 >= 1 (db + cd) x1000 + x1010 >= 1 (!db + cd) …….

  15. Minimal-MUMCUT Optimization • Applies optimization techniques for MUTP, CUTPNFP, and MNFP above based on criterion feasibility • When MUTP is feasible, can select any NFP • When MUTP is not feasible, but CUTPNFP is feasible, apply CUTPNFP optimization • Otherwise, apply MNFP optimization ab + cd Minimize x0100 + x0101 + x0110 + x1000 + x1001 + x1010 + … Subject to x0100 + x0101 + x0110 >= 1 (any NFP for a works as MUTP is feasible) x1000 + x1001 + x1010 >= 1 (any NFP for b works as MUTP is feasible) …….

  16. Case Study • Analyzed 19 Boolean predicates in an avionics software system (Weyuker, Chen, Lau, and Yu) • Number of unique literals range: 5 to 13 • Built a tool in Java to automatically: • generate MUTP, CUTPNFP, MNFP, Minimal-MUMCUT tests based on heuristics • generate an optimization model that when solved by CPLEX yields minimized test sets • Examined test set size and computation time difference between heuristics and optimization

  17. Average Test Set Size Results

  18. Average Computation Time Results (sec) 18

  19. Average Optimization Model Size 19

  20. Interpretation of Results For a small difference in computation time, optimization produces smaller test sets Running tests is time-consuming and expensive, so reducing test set size by one can be significant Choose optimization over heuristics 20

  21. Fault Detection Maximization Testers have limitations on the number of tests they can run and thus it may be impossible to satisfy a criterion. In this case, it would help to know what tests maximize fault detection subject to a given test set size. Each point (test) is a 0-1 variable - x1100 represents the point (test) where a is T, b is T, c is F, d is F - x1100 will be 0 if it is not in the test set and 1 if it is in the test set Each fault is a 0-1 variable - LIF_1_c represents a fault where literal c is inserted into term 1 - LIF_1_c will be 0 if it is not detected and 1 if it is detected - objective function is to maximize the sum of all the fault variables - a fault variable exists for each fault based on the 9 DNF fault types 21

  22. Fault Detection Maximization Consider ab + cd Maximize …. + LIF_1_c + ... Subject to • x1101 + x1100 >= M_LIF_1_c; (1) • LIF_1_c <= M_LIF_1_c; (2) …… LIF_1_c represents a fault where c is inserted into term 1 (ab). If neither 1101 nor 1100 is selected, the x1101 and x1100 variables will be 0. For constraint (1) to hold, the variable M_LIF_1_c must then be 0. Thus, variable LIF_1_c must be 0 as well and the fault goes undetected. If 1101 or 1100 is selected, either x1101 and x1100 will be 1. This allows the variable M_LIF_1_c to be 1, which in turn allows LIF_1_c to be 1 and thus the fault is detected.

  23. Case Study • Used same predicates as in the first study • Built a tool in Java to automatically: • generate an optimization model that when solved yields a test set that maximizes fault detection subject to a given test set size • generate an optimization model that when solved yields a test set that minimizes fault detection subject to a given test set size (by flipping constraints in the model) • Used the tool to compare test set size between a best, random, and worst case scenario

  24. Case Study Results Amongst all 19 predicates: • As many as 288 times more faults are detected in the best case than the worst case for a single test. • As many as 7 times more faults are detected in the best case than the random case for a single test. Implication for regression testing

  25. Case Study Results 25

  26. Fraser and Gargantini Study Found fault detecting tests for the same 19 predicates by formalizing the problem as a logic satisfiability problem which can be solved by a SAT algorithm (not optimization) 3 approaches Minimal-MUMCUT heuristic  every test needed for fault detection Satisfiability Algorithm  improves on heuristic Optimization  guaranteed smallest test set size 26

  27. Conclusion • Applied optimization techniques in the Minimal DNF logic testing domain to: • Minimize test set size subject to guaranteeing fault detection (by satisfying a given criterion) • Maximize fault detection subject to a certain test set size • Case study showed for a small sacrifice in computation time, optimization produced smaller test sets • Case study showed large difference in fault detection between a best, random, and worst case scenario • Applications in testing of programs with large predicates

  28. Thoughts On Where To Go Next • Reformulating Model To Handle Infeasible Tests • Crucial for Handling Predicates Embedded In Code • Requires Removal Of Some Constraints • Some Faults Become May Become Equivalent • Hence Constraint To Trigger Fault is Infeasible • Also Requires Adding Additional Constraints • Fault Hierarchy Only Applies For Feasible Faults • Applying Optimization To Other Coverage Criteria • Eg: Branch Coverage • Building Optimal “Test Databases” • Given a Query With A Particular “Where” Clause • What Minimal Database Guarantees Detecting All 9 Faults in Query? • Tool Available at Ammann/Offutt Book Web Site • http://cs.gmu.edu/~offutt/softwaretest

More Related