korat automated testing based on java predicates n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Korat: Automated Testing Based on Java Predicates PowerPoint Presentation
Download Presentation
Korat: Automated Testing Based on Java Predicates

Loading in 2 Seconds...

play fullscreen
1 / 37

Korat: Automated Testing Based on Java Predicates - PowerPoint PPT Presentation


  • 79 Views
  • Uploaded on

Korat: Automated Testing Based on Java Predicates. Chandrasekhar Boyapati 1 , Sarfraz Khurshid 2 , and Darko Marinov 3. 1 University of Michigan Ann Arbor 2 University of Texas at Austin 3 University of Illinois at Urbana-Champaign. Examples of Structurally Complex Data. red-black tree.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Korat: Automated Testing Based on Java Predicates' - salali


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
korat automated testing based on java predicates

Korat: Automated Testing Based on Java Predicates

Chandrasekhar Boyapati1, Sarfraz Khurshid2, and Darko Marinov3

1University of Michigan Ann Arbor

2University of Texas at Austin

3University of Illinois at Urbana-Champaign

examples of structurally complex data
Examples of Structurally Complex Data

red-black tree

XML document

1

<library> <book year=2001> <title>T1</title> <author>A1</author> </book> <book year=2002> <title>T2</title> <author>A2</author> </book> <book year=2003> <title>T3</title> <author>A3</author> </book></library>

/library/book[@year<2003]/title

0

3

2

testing setup
Testing Setup

pass

1

0

3

2

fail

2

2

0

0

3

3

3

0

inputs

outputs

  • examples of code under test
    • abstract data type
      • input/output: data structure
    • XML processing program
      • input/output: XML document

testing

oracle

testgenerator

code

manual test generation
Manual Test Generation

pass

1

0

3

2

fail

2

2

0

0

3

3

3

0

inputs

outputs

  • drawbacks of manual generation
    • labor-intensive and expensive
    • vulnerable to systematic bias that eliminates certain kinds of inputs
  • testgenerator
  • manual

testing

oracle

code

automated test generation
Automated Test Generation

pass

1

0

3

2

fail

2

2

0

0

3

3

3

0

inputs

outputs

  • challenges of automated test generation
    • describing test inputs
    • (efficient) test generation
    • checking output
  • testgenerator
  • automated

testing

oracle

code

running example
Running Example

B0: 3

root

N0: 2

right

left

N1: 1

N2: 3

class BST {

Node root;

int size;

static class Node {

Node left, right;

int value;

}

}

void remove(int i) { … }

example valid inputs
Example Valid Inputs

B0: 3

B0: 3

B0: 3

B0: 3

B0: 3

N0: 1

N0: 2

root

root

root

root

root

right

right

left

N0: 1

N0: 3

N0: 3

N1: 1

N2: 3

N1: 3

right

left

left

left

N1: 2

N1: 1

N1: 2

N2: 2

right

right

left

N2: 3

N2: 2

N2: 1

  • trees with exactly 3 nodes
example invalid inputs
Example Invalid Inputs

B0: 2

B0: 3

N0: 3

N0: 2

root

root

right

right

left

left

N1: 1

N1: 1

N2: 2

N2: 3

B0: 3

root

N0: 2

right

left

N1: 1

N2: 3

  • object graphs violating some validity property
validity properties for example
Validity Properties for Example

underlying graph is a tree(no sharing between subtrees)

correct number of nodes reachable from root

node values orderedfor binary search

B0: 3

N0: 2

root

right

left

N1: 1

N2: 3

describing validity properties
Describing Validity Properties
  • Korat uses a predicate written in standard implementation language (Java, C#…)
    • takes an input that can be valid or invalid
    • returns a boolean indicating validity
  • advantages
    • familiar language
    • existing development tools
    • predicates often already present
  • challenge: generate tests from predicates
example predicate
Example Predicate

boolean repOk(BST t) {

return isTree(t) && hasCorrectSize(t) && isOrdered(t);

}

boolean isTree(BST t) {

if (t.root == null) returntrue; // empty tree

Set visited = new HashSet(); visited.add(t.root);

List workList = new LinkedList(); workList.add(t.root);

while (!workList.isEmpty()) {

Node current = (Node)workList.removeFirst();

if (current.left != null) {

if (!visited.add(current.left)) return false; // sharing

workList.add(current.left); }

if (current.right != null) {

if (!visited.add(current.right)) return false; // sharing

workList.add(current.right); }

}

return true; // no sharing

}

input space
Input Space

B0: 2

B0: 3

B0: 3

B0: 3

B0: 3

B0: 3

N0: 1

N0: 2

N0: 3

root

root

right

right

left

left

N0: 1

N0: 3

N0: 3

root

root

root

root

N1: 1

N1: 1

N2: 2

N2: 3

N1: 3

B0: 1

B0: 3

right

N0: 1

root

B0: 3

left

B0: 1

B0: 1

N1: 2

N1: 1

N1: 2

root

right

N2: 2

left

left

N0: 2

N0: 1

root

left

root

left

right

B0: 0

N0: 2

B0: 1

N0: 1

root

N2: 3

N2: 2

N2: 1

left

right

right

right

N0: 1

left

root

N1: 1

N2: 3

right

right

left

N1: 1

N2: 3

  • all possible object graphs with a BST root(obeying type declarations)
bounded exhaustive generation
Bounded-Exhaustive Generation
  • finitization bounds input space
    • number of objects
    • values of fields
  • generate all valid inputs up to given bound
    • eliminates systematic bias
    • finds all errors detectable within bound
  • avoid isomorphic inputs
    • reduces the number of inputs
    • preserves capability to find all errors
example finitization
Example Finitization
  • specifies number of objects for each class
    • 1 object for BST: { B0 }
    • 3 objects for Node: { N0, N1, N2 }
  • specifies set of values for each field
    • sets consist of objects and literals
    • for root, left, right: { null, N0, N1, N2 }
    • for value: { 1, 2, 3 }
    • for size: { 3 }
example input space
Example Input Space

B0

N0

N2

N1

root

size

left

right

value

left

right

value

left

right

value

N0

3

N1

N1

2

null

null

1

null

null

3

B0

N0

N2

N1

root

size

left

right

value

left

right

value

left

right

value

null

null

null

null

null

null

null

1

1

1

N0

N0

N0

N0

N0

N0

N0

2

2

2

N1

N1

N1

N1

N1

N1

N1

3

3

3

N2

N2

N2

N2

N2

N2

N2

  • 1 BST object, 3 Node objects: total 11 fields

3

4 * 1 * (4 * 4 * 3)3 > 218 inputs, only 5 valid

example input
Example Input

B0

N0

N2

N1

root

size

left

right

value

left

right

value

left

right

value

N0

3

N1

N1

2

null

null

1

null

null

3

B0: 3

root

N0: 2

right

left

N1: 1

N2: 3

  • each input is a valuation of fields
generation problem
Generation Problem
  • given
    • predicate
    • finitization
  • generate
    • all nonisomorphic valid inputs within finitization
  • simple “solution”
    • enumerate entire input space
    • run predicate on each input
    • generate input if predicate returns true
    • infeasible for sparse input spaces (#valid << #total)
    • must reason about behavior of predicate
example execution
Example Execution

B0: 3

root

N0: 2

right

left

N1: 1

N2: 3

[ B0.root, N0.left, N0.right ]

[ B0.root ]

[ B0.root, N0.left ]

boolean repOk(BST t) {

return isTree(t) && …;

}

boolean isTree(BST t) {

if (t.root == null) return true;

Set visited = new HashSet(); visited.add(t.root);

List workList = new LinkedList(); workList.add(t.root);

while (!workList.isEmpty()) {

Node current = (Node)workList.removeFirst();

if (current.left != null) {

if (!visited.add(current.left)) return false;

workList.add(current.left); }

if (current.right != null) {

if (!visited.add(current.right)) return false;

workList.add(current.right); }

}

return true;

}

field accesses:

[ ]

failed execution
Failed Execution

B0

B0

N0

N0

N2

N2

N1

N1

root

root

size

size

left

left

right

right

value

value

left

left

right

right

value

value

left

left

right

right

value

value

N0

N0

3

-

N1

N1

N1

N1

2

-

null

-

null

-

1

-

null

-

null

-

3

-

  • failed after few accesses for a concrete input
  • would fail for all inputs with partial valuation

1 * 3 * 4 * 4 * 3 * 4 * 4 * 3 > 212

key idea
Key Idea
  • observe execution of predicate
  • record field accesses
  • prune large chunks of input space for each failed execution
  • use backtracking to efficiently enumerate valid inputs
search step
Search Step

B0

B0

N0

N0

N2

N2

N1

N1

root

root

size

size

left

left

right

right

value

value

left

left

right

right

value

value

left

left

right

right

value

value

N0

N0

3

3

N1

N1

N2

N1

2

2

null

null

null

null

1

1

null

null

null

null

3

3

  • backtracking on [ B0.root, N0.left, N0.right ]
  • produces next candidate input
next input
Next Input

B0

N0

N2

N1

root

size

left

right

value

left

right

value

left

right

value

N0

3

N1

N2

2

null

null

1

null

null

3

  • execution returns true, generate this input

B0: 3

root

N0: 2

right

left

N1: 1

N2: 3

korat
Korat

repOk

finitization

  • solver for executable predicates
  • Korat systematically searches input space
    • encodes inputs as vectors of integers
    • creates candidate object-graph inputs
    • monitors execution of predicate
    • prunes search by backtracking on field accesses
    • avoids isomorphic inputs

allnonisomorphicvalid inputs

Korat

isomorphic inputs
Isomorphic Inputs

B0: 3

N1: 2

root

right

left

N2: 1

N0: 3

B0: 3

root

N0: 2

right

left

N1: 1

N2: 3

  • equivalent for all code and all properties
  • example for trees: permutation of nodes
  • n! isomorphic trees => generation infeasible
  • removing isomorphic inputs from test suites
    • significantly reduces the number of tests
    • does not reduce quality

isomorphism definition
Isomorphism Definition
  • inputs: rooted and labeled (object) graphs
    • edges have fields
    • nodes have object identities
  • isomorphism with respect to object identitites
  • definition: inputs I and I’ consisting of objects from a set O are isomorphic iff: O->O. o,o’ reachable in I. ffields(o).

o.f = o’ in I => (o).f = (o’) in I’ and

o.f = p in I => (o).f = p in I’

example nonisomorphic trees
Example Nonisomorphic Trees

B0: 3

B0: 3

root

root

B0: 3

N0: 2

N0: 2

root

right

left

right

left

N0: 3

N1: 1

N2: 3

N1: 1

N2: 3

right

left

N1: 1

N2: 2

  • different edges or primitives, not only nodes
nonisomorphic generation
Nonisomorphic Generation
  • simple “solution”
    • generate all inputs
    • filter out isomorphic inputs
  • Korat does not require filtering
    • generate only one (candidate) input from each isomorphism class
    • only the lexicographically smallest input
    • search increments some field values for >1
correctness
Correctness
  • Korat’s input generation
    • sound
      • no invalid input
    • complete
      • at least one valid input from each isomorphism class
    • optimal
      • at most one (valid) input from each isomorphism class
example specification
Example Specification

class BST {

Node root; int size;

static class Node {

Node left, right; int value;

}

//@ invariant repOk();

//@ precondition true;

//@ postcondition !contains(i) && …;

void remove(int i) { … }

boolean contains(int i) { … }

}

testing scenario
Testing Scenario

root

B0: 3

invariantrepOk()

pass

right

N0: 1

postcondition!contains(i) && …

preconditiontrue

fail

right

N1: 2

N2: 3

root

B0: 3

right

N0: 1

N1: 2

, 3

and

and

Korat

remove

finitization

korat structure generation
Korat Structure Generation

linked data structures and array-based data structures

korat structure generation1
Korat Structure Generation

very large input spaces

korat structure generation2
Korat Structure Generation

pruning based on filed accesses very effective

korat structure generation3
Korat Structure Generation

correct number of nonisomorphic structures (Sloane’s)

korat structure generation4
Korat Structure Generation

800Mhz Pentium III Sun’s Java 2 SDK 1.3.1 JVM

korat method testing
Korat Method Testing

several operations for each data structure

conclusions
Conclusions
  • Korat automates specification-based testing
    • uses method precondition to generate all nonisomorphic test inputs
      • prunes search space using field accesses
    • invokes the method on each input and uses method postcondition as a test oracle
  • Korat prototype uses JML specifications
  • Korat efficiently generates complex data structures including some from JCF