Software Logic Mutation Testing

1. Software Logic Mutation Testing Presented by Gary Kaminski

2. Overview • Introduction and Motivation • Logic Faults and Detection • Case Study Comparison • Context • Conclusion

3. Logic Mutation Testing Process • Systematically create program variations (mutants) each containing a single typical logic fault if (a && b)  if (a || b) • For each mutant, find an input that distinguishes the original program from the mutant program (killing) - a && b and a || b evaluate to different truth values • Focus is on step 1)

4. Logic Mutation Testing • Logic mutants are created by applying mutation operators to Boolean predicates • Boolean predicates may come from: • UML Diagrams 2) Security Requirements 3) Looping constructs (while / do-while / for) 4) Preconditions for methods (if - then)

5. Motivation Current mutation systems: • generate excess logic mutants • do not detect certain classes of logic faults Goal: - generate fewer logic mutants - detect more logic faults Assumption: - formally restrict attention to minimal Disjunctive Normal Form (DNF) Boolean predicates

6. Minimal DNF • Terms separated by OR, literals by AND ab + a!c vs. a(b + !c) • Make each term true and other terms false ab + ac vs. ab + abc • Impossible to remove a literal or term without changing the predicate ab vs. abc + ab!c

7. Excess Logic Mutants - Example Original: ab + b!c + !bc 1) mutant generated multiple times: ab + _ + !bc 2) equivalent mutants: ac + b!c + !bc 3) mutants guaranteed to be killed by an input that kills some other mutant: !ab + b!c + !bc and cb + b!c + !bc 4) highly effective mutation operators not used: abc + b!c + !bc

8. Minimal DNF Logic Faults Original: ab + b!c + !bc • Expression Negation Fault: !(ab + b!c + !bc) • Expression Stuck At Fault 0: FALSE • Expression Stuck At Fault 1: TRUE • Term Negation Fault: !(ab) + b!c + !bc • Term Omission Fault: ab + _ + !bc • Scalar Variable Replacement TOF: ab + c!c + !bc • TOF generates same mutant twice: ab + _ + !bc

9. Minimal DNF Logic Faults Original: ab + b!c + !bc • Operator Reference Fault+: abb!c + !bc • Operator Reference Fault .: a + b + b!c + !bc • Operator Reference Fault+0: FALSE + !bc • Operator Reference Fault+1: TRUE + !bc • Operator Reference Fault.0: FALSE + b!c + !bc • Operator Reference Fault.1: TRUE + b!c + !bc

10. Minimal DNF Logic Faults Original: ab + b!c + !bc • Literal Negation Fault: !ab + b!c + !bc • Literal Stuck At Fault 0: FALSEb + b!c + !bc • Literal Stuck At Fault 1: TRUEb + b!c + !bc • Literal Omission Fault: _b + b!c + !bc • SVR LOF: bb + b!c + !bc • LOF generates same mutant twice original: abc  _bc, a_c, a_c, ab_

11. Minimal DNF Logic Faults Original: ab + b!c + !bc • Literal Reference Fault: ac + b!c + !bc • Literal Reference Fault: a!c + b!c + !bc • Literal Insertion Fault: abc + b!c + !bc • Literal Insertion Fault: ab!c + b!c + !bc • LRF and LIF generate equivalent mutants

12. LRF LOF LIF SVRLOF TOF LSTF1 SVRTOF ORF. LSTF0 ORF.1 ORF.0 LNF ORF+1 ORF+ TNF ESTF1 ORF+0 ESTF0 ENF Logic Fault Minimal DNF Detection Red Box – Mutation Operator does not exist Blue Box – Mutation Operator partially exists BlackBox – Mutation Operator exists Green Arrow – Detection may not hold Black Arrow – Detection always holds

13. Logic Mutation Comparison Original: ab + b!c + !bc • 61 mutants vs. 4 LIFs, 3 LOFs, 3 LRFs • Equivalent LIF: abc + b!c + !bc • LRF: cb + b!c + !bc & ac + b!c + !bc • LOF: a_ + b!c + !bc • Equivalent LIF: ab + b!c!a + !bc • LRFs: ab + !a!c + !bc & ab + b!a + !bc • No Equivalent LIFs for term !bc • LOFs: ab + b!c + _c & ab + b!c + !b_

14. Mutation Efficiency Measures Mutation Efficiency Score = Faults Detected - Mutants Generated Mutation Efficiency Ratio = Faults Detected / Mutants Generated • FD assumes an input is found to kill the mutant • FD includes equivalent mutants detected automatically ab + b!c + !bc Current Mutation Systems: • MES: 60 FD - 61 MG = -1 • MER: 60 FD / 61 MG = .98 LIF-LRF-LOF • MES: 73 FD – 10 MG = 63 • MER: 73 FD / 10 MG = 7.30

15. Case Study • Analyzed 19 Boolean predicates in an avionics software system • Number of unique literals range: 5 to 13 • Compared MES and MER for current mutation systems vs. LIF-LRF-LOF • Examined how MES and MER vary based on number of unique literals

16. Case Study Results • LIF-LRF-LOF generated 18% of the mutants current mutation systems would (150 vs. 819) • Current mutation systems guarantee detecting 75% of faults LIF-LRF-LOF does (819 vs. 1082) • Number of literals matters for MES but not for MER • MES = FD – MG and MER = FD / MG

17. Mutation Efficiency Score (FD – MG) vs. Number of Unique Literals

18. Mutation Efficiency Ratio (FD/MG) vs. Number of Unique Literals

19. Context: Internal Variable Problem • What input values kill a mutant? • Mutation may be buried deep in the code • Must reach predicate and have mutant predicate and original predicate evaluate to different truth values • Partial solutions using constraints exist

20. Context: Minimal DNF in Practice • 95% of 20,256 Boolean predicates in avionics software were in minimal DNF* 2) Detected > 99% of corresponding faults in non-minimal DNF Boolean predicates* *Source: Y.T Yu and M.F. Lau. Comparing Several Coverage Criteria for Detecting Faults in Logical Decisions. In Proceedings QSIC 2004: 4th International Conference on Quality Software, Pages 14-21.

21. Context: Industry Audience • LIF-LRF-LOF for software with Boolean predicates having >= 5 unique literals • Exhaustive coverage for < 5 unique literals • ab + b!c + bc (8 tests vs. 10 mutants) • Avionics software* *Source: J.J Chilenski and S.P. Miller. Applicability of modified condition/decision coverage to software testing. IEE/BCS Software Engineering Journal, 9(5): 193-200, September 1994.

22. Related Work • J.J. Chilenski. An Investigation of Three Forms of the Modified Condition Decision Coverage (MCDC) Criterion. Final Technical Report, DOT/FAA/AR-01/18, April 2001. • Non-Minimal DNF based logic coverage criteria 2) T.Y. Chen, M.F. Lau, and Y.T. Yu. MUMCUT: A Fault-Based Strategy for Testing Boolean Specifications. Software Engineering Conference, 1999. (APSEC ’99) Proceedings. Sixth Asia Pacific. Pages 606-613. • Minimal DNF based logic coverage criteria and Double Faults 3) D. Richard Kuhn. Fault Classes and Error Detection Capability of Specification Based Testing. ACM Transactions on Software Engineering and Methodology, 8(4): 411-424, October 1999. • Seminal work in logic fault hierarchy

23. Conclusion • Current logic mutation testing is inefficient • Excess mutants generated and faults are missed • Fault hierarchy approach based on minimal DNF: detect more faults with less mutants • Applications in avionics software

24. Other Research and Publications Logic Coverage Criteria: • Reduce test set size and maintain fault detection • Use feasibility of smaller, less expensive component criteria Internal variable problem • CoJava and constraint programming Publications: G. Kaminski, G. Williams, and P. Ammann. Reconciling Perspectives of Logic Testing for Software. Submitted to Journal of Software Testing, Verification, and Reliability, January 2008. Online.