Mainul Islam Supervisor: Dr. Christoph Csallner

1. PhD Research Proposal Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. ChristophCsallner The University of Texas at Arlington December 12th, 2012

2. Outline • Problem Description • Motivation • Limitation of Current Approaches (with example) • Thesis Statement • Background • Solution Approach • Experimental Results • Plan of Action

3. Problem Description • Current state of the art tools are not very good at generating test cases when the code under test: • requires additional pieces of code that are not yet part of the program. • uses multiple interfaces, annotations or reflection. • imposes complex (type) constraints. • uses native code.

4. Motivation • Automatic test case generation is important • At the initial stage of any development the implementation of some code may not be available • Existing techniques (such as Pex and Moles*) are not very good at generating test cases when the code under test uses interfaces/annotations, multiple inheritance and native code. • Significant number of Java programs use native code * http://research.microsoft.com/en-us/projects/pex/

5. Limitation of Current Techniques: Motivating Example 1 public @interface A { /* … */ } public interface I { public intm1(); /* … */ } public interface J { public intm2(); /* … */ } public class C { publicstatic void foo(Ii) { intx = i.m1(); if( iinstanceofJ ) { J j = (J) i; inty = j.m2(); } if( i.getClass().isAnnotationPresent(A.class) ) { // .. } } } To reach this block of code ‘i’ must be an instance of I, as well as an instance of J To reach this block of code ‘i’ must be an instance of I, as well as annotated with A

6. Motivating Example 2 (Dependency on Native Code) Java code C++ code public class C { public nativeboolean isDivisible(intx, inty); publicstatic void NativeTest(int a, int b) { booleandivisible = new C().isDivisible(a, b); if(divisible) { // … } } } boolisDivisible(intx, int y) { if( y*(x/y) == x ) return true; return false; } To reach this block of code ‘a’ must be divisible by ‘b’

7. Motivation: Patterns in Real World Program

8. Pattern 1 R m(…, Tt, …) { // … if( … (t instanceofX) ) // … } • ‘T’ an ‘X’ are non-compatible • ‘m’ is user-defined • At least one of {T, X} is an interface • None of {T, X} is final • At least one of {T, X} is a user-type

9. Thesis Statement • We can generate automatic test inputs and systematically increase code coverage compared to existing techniques specially when the code under test: • requires additional pieces of code that are not yet part of the program. • uses multiple interfaces, annotations or reflection. • imposes complex (type) constraints. • uses native code.

10. Background: Symbolic Execution • Systematically explore all feasible execution paths • Initialize the input with symbolic values and execute the program over symbolic values • At conditional statements check if either of the branches can be taken • For each path get an accumulated path condition If (S) then … else … C S true false C` = C ⋀ S C` = C ⋀⌝S

11. Dynamic Symbolic Execution by Exampletaken with permission from Nikolai Tillmann (Microsoft Research) publicstatic void TestMe(int a[]) { if(a ==null) return; if(a.length> 0) if(a[0] == 123) throw new Exception(“Error”); } Choose Next Path Solve Execute Constraints to Solve Input (a) Observed Constraints null a ==null a ==null a !=null {} a !=null && !(a.length> 0) F T a !=null && a.length> 0 {0} a !=null && a.length> 0 && a[0] !=123 a.length> 0 F T a[0]==123 a !=null && a.length>0&& a[0] ==123 {123} a !=null && a.length> 0 && a[0] == 123 F T Done: No Path Left

12. Sub-/Supertype relation in Java • Java defines a binary sub-type relation • If type B implements/extends type A then, • A is a direct super-type of B • B is a direct sub-type of A Reflexive:A is also a subtype of itself Transitive: if B is a subtype of A and C is a subtype of B then C is also a subtype of A

13. Sub-/Supertype relation in Java • A class has one direct classsuper type and arbitrarily many interfacesuper types. • Exceptions: type object – has no direct super type type null – has no direct sub type

14. Our Solution Approach • Introduce Dynamic Symbolic Mock Classes Technique implemented on: • Dsc1 – Dynamic Symbolic Execution Engine for Java • Z32 – SMT solver (from Microsoft Research) 1 - http://ranger.uta.edu/~csallner/dsc/index.html 2 - http://z3.codeplex.com/

15. Solution Workflow Default Input values (0, null, …) Invoke DSE on given input values and collect path constraints New Test Cases (and corresponding mock classes ) More Paths? Stop No Yes Add mock classes and map each of them to a constraint literal Invert one of the collected path constraints (Map each reference type to a constraint literal and encode their properties to build a constraint system) Encode properties (e.g., subtype relation) of mock classes in the constraint system Constraint System Satisfiable? Constraint System Satisfiable? No No Yes Map the constraint solver model to new test cases (and mock classes) Yes

16. Subtype Constraints Object public @interface A { /* … */ } public interface I { public intm1(); /* … */ } public interface J { public intm2(); /* … */ } public class C { publicstatic void foo(Ii) { intx = i.m1(); if( iinstanceofJ ) { J j = (J) i; inty = j.m2(); } // .. } } Annotation I J C A M null Initial Types in the system A desired solution with new Type: M, to reach the code block Constraints: type(i) subtypeofI type(i) !=null type type(i) subtypeofJ

17. Subtype Constraints Object public @interface A { /* … */ } public interface I { public intm1(); /* … */ } public interface J { public intm2(); /* … */ } public class C { publicstatic void foo(Ii) { intx = i.m1(); // .. if( i.getClass(). isAnnotationPresent(A.class) ) { // .. } } Annotation I J C A M1 null Initial Types in the system A desired solution with new Type: M1, to reach the code block Constraints: type(i) subtypeofI type(i) !=null type type(i) subtypeofA

18. Subtype Relation Matrix public @interface A { /* … */ } public interface I { public intm1(); /* … */ } public interface J { public intm2(); /* … */ } public class C { publicstatic void foo(Ii) { intx = i.m1(); if( iinstanceofJ ) { J j = (J) i; inty = j.m2(); } // .. } } Solution: mI = true mJ= true mC= false mA= mAn= false

19. Experimental Results (1) • Experiments are done on simplified version of real world code • C# codes are translated manually

20. Experimental Results (2) • Experiments are done on original code • Randoop has several side effects

21. Workflow: to handle Native Code Default Input values (0, null, …) Invoke DSE on given input values and collect path constraints New Test Cases Native Code Invoked? More Paths? Stop No No Yes Yes Invert one of the collected path constraints and build constraint system Collect the constraints from the native code for current input values Constraint System Satisfiable? No Convert the native code constraints to Java constraints Yes Map the constraint solver model to new test cases

22. Technique to handle Native Code In each iteration of the dynamic symbolic execution of a Java program: • Check if a native code call is invoked. If yes, start executing the native code on current input values • Use any existing tool (such as Klee* for C++ code) to collect the path constraints of the native code • Convert the constraints collected from the native code and merge them with the constraints previously collected from the Java program • Solve the whole constraint system using a Constraint solver and generate test case. • http://klee.llvm.org/

23. Initial Experiments In JDK 1.6: • Total # types: 23799 • Total # of types (have at least one native method): 381 • Total # of native methods: 2019 • http://klee.llvm.org/

24. Plan of Actions

25. List of publications • Mainul Islam and Christoph Csallner. Generating Test Cases for Programs that are Coded Against Interfaces and Annotations (submitted) • Mainul Islam and ChristophCsallner. Dsc+Mock: A test case + mock class generator in support of coding against interfaces (In Proc. 8th International Workshop on Dynamic Analysis (WODA), co-located with International Symposium on Software Testing and Analysis (ISSTA), 2010)

26. Thank You!

27. Backup Slides

28. Pattern 2 class P {Tt, …} class Q { M m(…, Pp, …){ // … if( … (p.t instanceofX) ) // … }

29. Pattern 3 M m(…) { // … if( … (..).foo() instanceofX ) // … }

30. Related work Mock classes inferred from Programmer-Written specification: • EasyMock, jMock, Mockito • NMock • Google Mock • SynthiaMock

31. Related work: example with EasyMock public interface K extends I, J { /* … */ } public class Test { public void testFooIsInstanceOfJ(){ Kmock=createMock(K.class); expect(mock.m1()).andReturn(0); expect(mock.m2()).andReturn(0); replay(mock); C.foo(mock); verify(mock); } }

32. u1 u3 Visual similarity 0.5 0.6 0.4 0.2 0.9 0.5 0.8 0.3 0.3 0.5 0.9 u5 u2 0.2 0.4 0.2 0.4 Geo similarity u4