1 / 57

Automation of Test Case Generation

Automation of Test Case Generation. Testing is expensive. We have talked about 3 different kinds of coverage However, the three coverage are mainly used As measure for progress Not as target to be achieved Why? Money coverage Time coverage The most used targets…. 2.

aelkins
Download Presentation

Automation of Test Case Generation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Automation of Test Case Generation

  2. Testing is expensive We have talked about 3 different kinds of coverage However, the three coverage are mainly used As measure for progress Not as target to be achieved Why? Money coverage Time coverage The most used targets… 2

  3. Testing is expensive Numbers How many lines of test code About 40% in Microsoft code base How much is spent each year on software testing It is not easy to even estimate the time/effort spent on testing inside a IT company (Recall words of Bill Gates) The software testing market (out source testing) is about $17B in year 2012 How many new testers More than 30k each year in US recently Well more than 150K each year globally 3

  4. Automation is the way to reduce cost Good or bad news, no way to fully automate testing in predictable future Three phase Execution automation Mostly done for unit testing: xUnit Somewhat done for system testing: scripts Test case generation automation Quite hot topic in academia Great progress made recently Test oracle check automation Some good trials 4

  5. Automating different steps Automating test execution Unit Testing Framework Record and Replay for GUI testing Scripts for Command-line tools and configuration testing Automating test case generation (this lecture) 5

  6. Automatic test case generation Random testing Pure Random Rule-based Combinatory Feedback-guided Exploration testing Systematical testing Symbolic execution Chaining Search-based testing 6

  7. Random Testing Randomly generate inputs to feed in a software Easiest way to do automatic test case generation Do not require any preparation and easy to implement Problems Semantically redundant inputs E.g., for a simple program 10/x, providing any input except 0 means the same 7

  8. Rule-based Random Testing Try to reduce redundancy Use rules Try boundary cases 0, 1, -1, and other known boundaries Use distributions Generate random values following certain distribution Very effective if the distribution of input is known e.g, scores of a final exam 8

  9. Combinatory Testing Recall input combination coverage Try to achieve better coverage until reach 100% Add a new test case that covers uncovered value combinations Give priority to the test case that covers multiple new possible values For multiple features of a given input, use constraint solvers to generate a value that satisfies feature combinations (e.g., length = 2 && starts-with A) 9

  10. Adaptive random testing Intuition Input patterns for bug revealing Strips: bug is revealed when x + y = 4 Blocks: bug is revealed when 3 < x, y < 4 Trying two very similar test cases does not make much sense 10

  11. Adaptive random testing Approach When choosing the next test case, choose the test case having the largest minimum distance from existing test cases {t1,t2,…tn} are existing test cases, choose ch so that Example: one input, int16 x T3: -32768 T2:32767 T5: -16374 T1: -20 T6:... T4: 16394 0

  12. Feedback-Guided Random Testing Question: It is easy to generate random values for primitive types, how to generate a random object? You may generate a random object id = -1, content = “”, child = null May not make sense… For a class, id may be always positive, content may not be empty, and child cannot be null… 12

  13. Feedback-Guided Random Testing Consequence: Randomly generated objects may cause a lot of bugs But they are not real bugs… id = -1; content = ""; child = null; class BugClass{ int id; String content; Object child; public BugClass(String str){ this.id = Global.id ++; this.content = "bug:" + str; this.child = Bug.createDefaultChild(); } public do(){ String type = Global.type[this.id]; String content = this.content.substring(4); this.child.do(); } } Index array < 0!!! Index out of string length!!! Null pointer reference!!! 13

  14. Feedback-Guided Random Testing Solution: Try from the methods taking primitive types first When an object of a class is generated, use the object in following test cases Do not use duplicate and null objects Do not use objects generated with exception 14

  15. Tool Introduction-Randoop Feedback-based random testing Joint work by MIT and Microsoft Research The most robust and effective random testing tool for Java Have both stand-alone tool and eclipse plug-in Tool website: https://randoop.github.io/randoop/ 15

  16. Tool Introduction - Randoop Process Install with from website site: https://randoop.github.io/randoop/ Run in command line Configuration How long should randoop run Size of test methods Timeout of certain thread #Test methods per file 16

  17. Tool Introduction - Randoop Results Add support Jar file Use Junit to re-execute the test suite Remove some invalid test cases Use EclEmma to check results 17

  18. Exploration Testing Very commonly used in testing of GUI & Web software Explore the user interface Click all clickable Feed in random value to input boxes May often achieve good coverage One problem is the login part Or any inputs that requires some semantics 18

  19. Exploration Testing Example Menu Window About Play Records Setting Audio Setting Select Level Login Game Setting Play Confirm 19

  20. Exploration Testing Return to the last window Return button, go back, … Not always available Record the state of the software at the previous window Sometimes high overhead Usually Require instrumentation of the code To check outputs (sent to the screen) with oracles To tell the explorer that the window is ready (the drawing of the window is done) 20

  21. Exploration Testing Signal based exploration testing Software under test Driver Click event Event handler finishes Analyze new GUI Click event Event handler finishes 21

  22. Search-based Testing Deem test case generation as a optimization problem Somewhat between random testing and systematical testing Based on random testing, and focus on the input domains Use code coverage as guidance 22

  23. Optimization Try to find the maximal or minimal value of a certain function Numerous practical problems can be viewed as optimization problem Least cost to travel to a number of cities Least camera to cover an area Distribution of stores to attract most customers Design of pipe systems with least material Put items into a backpack (with limited volume) with highest value 23

  24. Illustration of Optimization global peak local peak 24

  25. Solution of Optimization Hill climbing Start from a random point Try all neighboring points, and go to the point with highest value, until all neighboring points has a value lower than the current point Easy to find a local peak Random-restart Hill climbing Restart hill climbing for many times To avoid local peaks 25

  26. Solution of Optimization Annealing simulation Adaptation of hill climbing Has a probability to move after reached local peak The probability drops as time goes by Genetic algorithm Simulate the process of evolution Start with random points Select a number of best points Combine and mutate these points Until no more improvements can be made 26

  27. Transform Testing to Optimization Using code coverage as criterion: Try to generate a test case that covers certain code element (method, statement, branch, …) Measure how well we have solve the problem A simple fitness function How far is the already covered elements from the target code elements Try to make the distance 0 27

  28. Go from random testing A list of random test cases as the start point Each test case is a point in the input domain Using various optimization algorithms to find the best test case 28

  29. An example with hill climbing Target is st2 Start from 0, 0 f(0, 0) = 2 steps Try (0,1) (1,0), (0,-1), (-1,0) No improvement Go all directions equally Until reach (5,5) f(5,5) = 1 step Increase x and y Until reach (7,5) Done! read x, y if x+y >= 10 N y st1 st3 N if x >= 7 read x, y; if(x+y>=10){ st1; if(x>=7){ st2; // target } } st3; y y st2 29

  30. Problem The fitness function is too simple Requires a lot of random walk before getting to the correct place (5,5) Better algorithms may help, but not much, because all algorithms require to compare the value of fitness functions A better fitness function: Consider the value gap at all branches 30

  31. Rerun example with hill climbing Target is st2 Start from 0, 0 f(0, 0) = 10 value gap Try (0,1) (1,0), (0,-1), (-1,0) Go to (0,1) or (1,0) Until reach (0,10) f(0,10) = 7 value gap Increase x Until reach (7,10) Done! read x, y if x+y >= 10 N y st1 st3 N if x >= 7 read x, y; if(x+y>=10){ st1; if(x>=7){ st2; // target } } st3; y y st2 31

  32. Systematical testing Target at certain code coverage Try to understand how the code works Analyze the code structure to find out a path to go to a certain statement Analyze the code structure to find out the constraint of the inputs to let the program follow the path 32

  33. Symbolic execution Target at better statement coverage Basic idea If a statement is not covered yet, try to provide an input to go over that statement A statement is covered only when a path goes to it is covered Then, what input will cause the program to go through certain paths? Only when the input satisfy all if-conditions along the path! 33

  34. Symbolic execution: example void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } a==null F T a.Length>0 T F a[0]==123… F T

  35. How to know what input to feed in Static symbolic execution Construct a constraint for each statement, the statement will be executed when the constraint is satisfied The parameters (variables) in the constraint are inputs of the software Solve the constraint The variables used in conditions may not be inputs So they must be represented by expressions of inputs 35

  36. Static symbolic execution Basic Example Here T is the condition for the statement to be executed, (y=s) is the relationship of all variables to the inputs after the statement is executed T (y=s), s is a symbolic variable for input y = read(); y = 2 * y; if (y <= 12) fails(); else success(); print ("OK"); T (y=2*s) T (y=2*s) T^y<=12 (y=2*s) T^!(y<=12) (y=2*s) T^y<=12 (y=2*s) | T^!(y<=12) (y=2*s) 36

  37. Static symbolic execution Generating test cases y = read(); T (y=s), s is the input y = 2 * y; T (y=2*s) if (y <= 12) T (y=2*s) fails(); T^y<=12 (y=2*s) s<=6 -> 3 else success(); T^!(y<=12) (y=2*s) s>6 -> 8 print ("OK"); T^y<=12 (y=2*s) | T^!(y<=12) (y=2*s) 37

  38. Constraint Solver Solve constraints -> provide a value set satisfying the constraint Float values Linear programming if the constraints are all linear Boolean values SAT problem Integer values Can be deduced to SAT String values SMT problem Mostly NPC problems Can be deduced from/to SAT problem 38

  39. Problems of static symbolic execution Path explosion Remember n branches will cause 2n paths Infinite paths for unbounded loops Calculate constraints on all paths is infeasible for real software Too complex constraint The constraint gets very complex for large programs Not to mention the resolving part is NPC 39

  40. Chaining and Goal-oriented Approach Not all branches are actually relevant If the outcome of a branch will not affect whether a statement will be executed We should not waste time on the branch A classification of branches Critical branches The branch must not be taken if target is executed Non-essential branches Other branches 40

  41. Example of branch classification Read a, b if a > 0 Read a, b; if (a > 0){ a = a + b; } if (b >= 10){ st2; //target } N-E a = a+b; N-E if b >= 10 C st2 End 41

  42. Chaining Find a path to reach the target code Start with the start node and the target node <read a,b>, <target> Avoid critical branches: b>=10 Choose b = 10 42

  43. A different example b=0 if a > 0 Read a; b = 0; if (a > 0){ b = a + b; } if (b >= 10){ st2; //target } N-E b = a+b; N-E if b >= 10 C st2 End 43

  44. Chaining Find a path to reach the target code Start with the start node and the target node <read a>, <target> Avoid critical branches: b>=10 , no solution Go to definition statement of b: b = 0; no solution b = a + b; -> a + b>=10 44

  45. Chaining: next steps <read a>, <b=a+b, a+b>=10>, <target> Avoid critical branches: a>0, a+b>=10 go to definition of a, b b = 0; read a; -> solve the constraint: a = 10 45

  46. Chaining Compared to static symbolic execution: Reduce the branches to be considered Reduce the definitions to be considered More scalable than static symbolic execution Problems Still not scalable enough Spend much time to generating only 1 test case to cover a certain statement 46

  47. Dynamic symbolic execution Koushik Sen et al. 2005 One of the most important progress in software engineering in the 21st century Basic idea Actually run the software Generate constraints as the program runs Flip constraints to reach other statements 47

  48. Dynamic symbolic execution Choose next path Code to generate inputs for: Solve Execute&Monitor void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } Data null {} {0} {123…} Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890 Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==1234567890 Negated condition a==null F T a.Length>0 T F a[0]==123… Done: There is no statement left. F T 48

  49. Dynamic symbolic execution Advantages No need to analysis the whole system before perform testing All constraints and expressions can be executed along one execution path Can handle library method calls Testing as the analysis taking place Disadvantages Overhead in testing 49

  50. Review of testing General guidelines Levels of testing Unit testing, higher level testing, GUI testing Test coverage Code, input, mutation Regression testing Non-function testing Security testing, performance testing Test automation 50

More Related