Comparing and Combining Test-Suite Reduction and Regression Test Selection

1. Comparing and Combining Test-Suite Reduction and Regression Test Selection August Shi, Tifany Yung, Alex Gyori, and Darko Marinov FSE 2015 Bergamo, Italy 09/02/2015 NSF Grant Nos. CCF-1012759, CCF-1421503, CCF-1434590, CCF-1439957

2. Testing is Important but Slow test0 test1 test2 test3 … testN-1 testN Code Under Test V0

3. Regression Testing is Slow(er) test0 test1 test2 test3 … testN-1 testN Code Under Test V0 test0 test1 test2 test3 … testN-1 testN Code Under Test V1 test0 test1 test2 test3 … testN-1 testN Code Under Test V2

4. Speeding up Regression Testing • Test-Suite Reduction • Regression Test Selection • Test-Suite Parallelization • Refactoring Tests • Many More

5. Speeding up Regression Testing • Test-Suite Reduction • Regression Test Selection • Test-Suite Parallelization • Refactoring Tests • Many More

6. Test-Suite Reduction (TSR) test0 test1 test2 test3 … testN-1 testN Code Under Test V0 test0 test1 test2 test3 … testN-1 testN Code Under Test V1 test0 test1 test2 test3 … testN-1 testN Code Under Test V2

7. Regression Test Selection (RTS) Δ test0 test1 test2 test3 … testN-1 testN Code Under Test V0 test0 test1 test2 test3 … testN-1 testN Code Under Test V1

8. Regression Test Selection (RTS) test0 test1 test2 test3 … testN-1 testN Code Under Test V0 test0 test1 test2 test3 … testN-1 testN Code Under Test V1 test0 test1 test2 test3 … testN-1 testN Code Under Test V2

9. TSR versus RTS(Known Qualitative Comparison) Test-Suite Reduction Regression Test Selection How are tests chosen to run? Redundancy (one revision) Changes (two revisions) How often is analysis performed? Infrequently Every revision Can it miss failing tests from the original test suite? Yes No (if safe)

10. How do TSR and RTS compare quantitatively?How can TSR and RTS be combined?

11. How do TSR and RTS compare quantitatively?How can TSR and RTS be combined?

12. TSR Background T = Tests S = Statements

13. TSR Background T = Tests S = Statements Reduced Test Suite R = {T3,T5}

14. TSR Background T = Tests S = Statements R = {T3,T5} Size

15. TSR Background T = Tests S = Statements R = {T3,T5} Size

16. TSR Background T = Tests S = Statements R = {T3,T5} Size Fault-Detection Capability

17. TSR Background T = Tests S = Statements M = Mutants R = {T3,T5} Size Fault-Detection Capability

18. RTS Background T = Tests S = Statements Δ S2 changed S6 added Vi-1 Vi

19. RTS Background T = Tests S = Statements Δ S2 changed S6 added Vi-1 Vi Selected Tests Si,Δ = {T1,T2,T3}

20. RTS Background T = Tests S = Statements Δ S2 changed S6 added Vi-1 Vi Si,Δ= {T1,T2,T3} Size

21. RTS Background T = Tests S = Statements Δ S2 changed S6 added S2 changed S6 added Vi-1 Vi Si,Δ= {T1,T2,T3} Size Fault-Detection Capability Safe RTS does not fail to detect change-related faults

22. How can TSR and RTS be combined?

23. Applying RTS after TSR T = Tests S = Statements Vi-1

24. Applying RTS after TSR T = Tests S = Statements Vi-1 R = {T3,T5}

25. Applying RTS after TSR T = Tests S = Statements Δ S2 changed S6 added S2 changed S6 added Vi Vi-1 Ri = {T3,T5}

26. Applying RTS after TSR T = Tests S = Statements Δ S2 changed S6 added S2 changed S6 added Vi-1 Vi Ri = {T3,T5}

27. Applying RTS after TSR T = Tests S = Statements Δ S2 changed S6 added S2 changed S6 added {T3} Vi-1 Vi Fault-Detection Capability Size If RTS is safe, then as good as reduced test-suite Selection of Reduction (SeRe)

28. Metrics to compare between approaches • Size Decrease: • TSR: • RTS: • SeRe: • Fault-Detection Capability Decrease • Currently, NO metric for fault-detection capability between approaches • We need a metric that takes CHANGE into account

29. Map Tests to Faults T = Tests F= Faults Vi-1 Vi Need criteria that includes these change-related faults

30. Detect all Faults? T = Tests F= Faults Vi-1 Vi

31. Which is Better? T = Tests F= Faults Vi-1 Vi Detects 5 faults Detects 1 change-related fault

32. Which is Better? T = Tests F= Faults Vi-1 Vi Detects 5 faults Detects 1 change-related fault Detects 4 faults Detects 2 change-related faults If criteria is to detect all faults, can get misleading comparisons with respect to these change-related faults!

33. Finding Change-Related Faults T = Tests F= Faults Vi-1 Vi Safe RTS will not fail to select tests whose behavior differs after the change

34. Finding Change-Related Faults T = Tests F= Faults Vi-1 Vi Si,Δ = {T1,T2,T3}

35. Finding Change-Related Faults T = Tests F= Faults Vi-1 Vi Si,Δ = {T1,T2,T3} Faults(Si,Δ) = {F1,F2,F3,F4,F6}

36. Finding Change-Related Faults Faults detected by non-selected tests cannot be change-related! ChangeRelatedFaultsi,Δ = Faults(Si,Δ) \ Faults(Oi\ Si,Δ) Faults(Si,Δ) \ Faults(Oi \ Si,Δ) = Faults({T1,T2,T3}) \ Faults({T4,T5}) Faults(S1,Δ) \ Faults(O1 \ S1,Δ= {F1,F2,F3,F4,F6} \ {F1,F3,F5} = {F2,F4,F6} Vi Vi-1

37. Change-Related Requirements (CRR) Use testing requirements (statements covered or mutants killed) to approximate fault-detection capability of test suite T chosen from Oi Evaluate loss in change-related fault-detection capability of reduced test suite

38. Evaluation Setup

39. Projects

40. Experimental Setup • Use Greedy heuristic to perform TSR • Remove redundant tests with respect to statement coverage • Statement coverage/mutants killed collected using PIT • http://pitest.org • Use Ekstazi to perform (safe) RTS • Select tests based on file-level dependencies • Tests selected at test class level • http://www.ekstazi.org • Simulate evolving reduced test suite and selection of reduction

41. Evolving Reduced Test Suite test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 Code Under Test V0

42. Evolving Reduced Test Suite test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 Code Under Test V0

43. Evolving Reduced Test Suite test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 Code Under Test V0

44. Evolving Reduced Test Suite test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1

45. Evolving Reduced Test Suite test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1

46. Evolving Reduced Test Suite test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 test4 Code Under Test V2 test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1

47. Evolving Reduced Test Suite test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 test4 Code Under Test V2 test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 test4 Code Under Test V2

48. Evolving Reduced Test Suite test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 test4 Code Under Test V2 test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 test4 Code Under Test V2

49. Evolving Reduced Test Suite test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 test4 Code Under Test V2 test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 test4 Code Under Test V2

50. Selection of Reduction • Can see result of selection of reduction by looking at tests chosen by TSR and RTS • Given and , can intersect the two to see what tests from are selected due to changes