Relaxing the testcase assumptions in genprog
Download
1 / 38

Relaxing the Testcase Assumptions in GenProg - PowerPoint PPT Presentation


  • 67 Views
  • Uploaded on

Relaxing the Testcase Assumptions in GenProg. Jonathan Dorn. EVALUATE FITNESS. INPUT. DISCARD. ACCEPT. GenProg. OUTPUT. MUTATE. EVALUATE FITNESS. INPUT. DISCARD. ACCEPT. GenProg. OUTPUT. MUTATE. EVALUATE FITNESS. INPUT. DISCARD. ACCEPT. GenProg. OUTPUT. MUTATE.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Relaxing the Testcase Assumptions in GenProg' - wynona


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Relaxing the testcase assumptions in genprog

Relaxing the Testcase Assumptionsin GenProg

Jonathan Dorn


Genprog

EVALUATE FITNESS

INPUT

DISCARD

ACCEPT

GenProg

OUTPUT

MUTATE


Genprog1

EVALUATE FITNESS

INPUT

DISCARD

ACCEPT

GenProg

OUTPUT

MUTATE


Genprog2

EVALUATE FITNESS

INPUT

DISCARD

ACCEPT

GenProg

OUTPUT

MUTATE


Testcases in genprog

EVALUATE FITNESS

Testcases in GenProg


Testcases in genprog1

Indicate which functionality to repair.

Indicate which functionality to retain.

Negative Testcases

Positive Testcases

Testcases in GenProg


Testcase assumptions

  • Testcasesexist.

  • They are correct.

  • They are comprehensive.

Testcase Assumptions


Testcase difficulties

  • Human-written testcases are expensive.

    • Development and maintenance costs.

    • Legacy code or remote deployments.

  • “Complete coverage” approached but rarely achieved.

Testcase Difficulties


How can we help

Machine-generatedtestcases!

How Can We Help?


Agenda

  • Automatic Testcase Generation

  • Evaluation

  • Preliminary Results

Agenda


Automatic testcases

Automatic Testcases


The oracle comparator model

Input

Output

Oracle

Comparator

Pass / Fail

The Oracle Comparator Model


Structure of a testcase

  • Test Setup

    • Network connections, opened files, etc.

    • Argument values

  • Run Function Under Test

  • Check Results

    • Return value

    • Side-effects

Structure of a Testcase


Automatic testcases1

  • Test Setup

    • Network connections, opened files, etc.

    • Argument values

  • Run Function Under Test

  • Check Results

    • Return value

    • Side-effects

Automatic Testcases


Automatic testcases2

  • Test Setup

    • Network connections, opened files, etc.

    • Argument values

  • Run Function Under Test

  • Check Results

    • Return value

    • Side-effects

Automatic Testcases


Test input generation

  • DART

  • CUTE

  • CREST

  • Symstra

  • Austin

  • Pex

Long-established research area

Test Input Generation


Concolic execution

  • Generate initial input.

  • Run test; record constraints at branches.

  • Negate one constraint.

  • Solve for new input.

  • Repeat.

Concolic Execution


Concolic execution1

  • int f(int x) {

  • if (x < 10)

  • return 1;

  • else

  • return 2;

  • }

Input: x = 456123

pred = {}

Concolic Execution


Concolic execution2

  • int f(int x) {

  • if (x < 10)

  • return 1;

  • else

  • return 2;

  • }

Input: x = 456123

pred = {}

Concolic Execution


Concolic execution3

  • int f(int x) {

  • if (x < 10)

  • return 1;

  • else

  • return 2;

  • }

Input: x = 456123

pred = {x>10}

Concolic Execution


Concolic execution4

  • int f(int x) {

  • if (x < 10)

  • return 1;

  • else

  • return 2;

  • }

Input: x = 9

pred = {}

Concolic Execution


Automatic testcases3

  • Test Setup

    • Network connections, opened files, etc.

    • Argument values 

  • Run Function Under Test

  • Check Results

    • Return value

    • Side-effects

Automatic Testcases


Automatic testcases4

  • Test Setup

    • Network connections, opened files, etc.

    • Argument values 

  • Run Function Under Test

  • Check Results

    • Return value

    • Side-effects

Automatic Testcases


Relaxing the testcase assumptions in genprog

  • Oracle: check for invariants.

  • What are the interesting invariants?

    • Checking that 1 = 1 is not useful.

    • Invariants are true for all runs of program but violated for some runs of not-quite-the-same program.

μtest


Relaxing the testcase assumptions in genprog

f = min(a,b)

  • Identify predicates that are true for all inputs.

μtest


Relaxing the testcase assumptions in genprog

μtest


Relaxing the testcase assumptions in genprog

May miss invariants if mutants do not fail.

μtest


Automatic testcases5

  • Test Setup

    • Network connections, opened files, etc.

    • Argument values 

  • Run Function Under Test

  • Check Results

    • Return value 

    • Side-effects

Automatic Testcases


Research questions

Research Questions


Evaluation

  • Only generated fixes?testcases.

    • Pretend human test cases do not exist.

  • Generated testcases + X% of original.

    • How much is human burden reduced?

Evaluation





Preliminary results3

*Invalid repair. **No repair found. fixes?

Preliminary Results


Preliminary results4

*Invalid repair. **No repair found. fixes?

Preliminary Results


Testcase assumptions1

  • Testcases fixes?exist.(concolic execution + μTEST)

  • They are correct.(compent programmer assumption)

  • They are comprehensive.(may need small number of human tests)

Testcase Assumptions


Conclusion

  • We can create fixes?testcasesautomatically to augment human-created tests.

  • Initial results suggest these tests permit repairs without comprehensive test suites.

Conclusion


Relaxing the testcase assumptions in genprog

Questions? fixes?