Efficient time aware prioritization with knapsack solvers
Download
1 / 22

- PowerPoint PPT Presentation


  • 364 Views
  • Uploaded on

Efficient Time-Aware Prioritization with Knapsack Solvers. Sara Alspaugh Kristen R. Walcott Mary Lou Soffa University of Virginia. Michael Belanich Gregory M. Kapfhammer Allegheny College. Test Suite Prioritization . Testing occurs throughout software development life cycle

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about '' - medwin


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Efficient time aware prioritization with knapsack solvers l.jpg

Efficient Time-Aware Prioritization with Knapsack Solvers

Sara AlspaughKristen R. WalcottMary Lou Soffa

University of Virginia

Michael BelanichGregory M. Kapfhammer

Allegheny College

ACM WEASEL Tech


Test suite prioritization l.jpg
Test Suite Prioritization

  • Testing occurs throughout software development life cycle

    • Challenge: time consuming and costly

  • Prioritization: reordering the test suite

    • Goal: find errors sooner in testing

      • Doesn’t consider the overall time budget

    • Alternative: time-aware prioritization

      • Goal 1: find errors sooner in testing

      • Goal 2: execute within time constraint


Motivating example l.jpg
Motivating Example

Original test suite with fault information

T1

4 faults

2 min.

T3

2 faults

2 min.

T4

6 faults

2 min.

T2

1 fault

2 min.

Assume: - Same execution time - Unique faults found

Prioritized test suite

T4

6 faults

2 min.

T1

4 faults

2 min.

T3

2 faults

2 min.

T2

1 fault

2 min.

Testing time budget: 4 minutes


The knapsack problem for time aware prioritization l.jpg

n

P

n

P

¤

c

x

·

t

t

i

i

¤

x

i

1

i

i

t

=

i

1

m

a

x

=

x

i

i

t

i

m

a

x

i

c

i

The Knapsack Problem for Time-Aware Prioritization

Maximize: , where is the code coverage of test and is either 0 or 1.

Subject to the constraint:

where is the execution time of test and is the time budget.


The knapsack problem for time aware prioritization5 l.jpg
The Knapsack Problem for Time-Aware Prioritization

Assume test cases cover unique requirements.

T1

4 lines

2 min.

T2

1 line

2 min.

Time Budget: 4 min.

T3

2 lines

2 min.

T4

5 lines

2 min.

Total Value:

Space Remaining:

5

2 min.

9

0 min.

0

4 min.


The extended knapsack problem l.jpg
The Extended Knapsack Problem

  • Value of each test case depends on test cases already in prioritization

    • Test cases may cover same requirements

T1

4 lines

2 min.

T1

0 lines

2 min.

T2

1 line

2 min.

Time Budget: 4 min.

T3

2 lines

2 min.

T4

5 lines

2 min.

Total Value:

Space Remaining:

5

2 min.

7

0 min.

0

4 min.

UPDATE


Goals and challenges l.jpg
Goals and Challenges

  • Evaluate traditional and extended knapsack solvers for use in time-aware prioritization

    • Effectiveness

      • Coverage-based metrics

    • Efficiency

      • Time overhead

      • Memory overhead

  • How does overlapping code coverage affect results of traditional techniques?

  • Is the cost of extended knapsack algorithms worthwhile?


The knapsack solvers l.jpg
The Knapsack Solvers

  • Random: select tests cases at random

  • Greedy by Ratio: order by coverage/time

    • Greedy by Value: order by coverage

    • Greedy by Weight: order by time

  • Dynamic Programming: break problem into sub-problems; use sub-problem results for main solution

  • Generalized Tabular: use large tables to store sub-problem solutions


The knapsack solvers continued l.jpg
The Knapsack Solvers (continued)

  • Core: compute optimal fractional solution then exchange items until optimal integral solution found

  • Overlap-Aware: uses a genetic algorithm to solve the extended knapsack problem for time-aware prioritization


The scaling heuristic l.jpg

j

k

³

´

t

t

¸

£

£

m

a

x

m

a

x

c

c

1

2

h

i

T

T

t

t

[

]

1

2

T

1

1

1

2

¡

x

n

x

;

:

:

:

c

c

c

T

T

x

1

¸

2

¸

¸

;

;

n

i

1

:

:

:

t

t

t

1

2

n

The Scaling Heuristic

  • Order the test cases by their coverage-to-execution-time ratio such that:

  • If , then it is possible to find an optimal solution that includes .

  • Check the inequality for each test case until it no longer holds.

  • belong in the final prioritization.


Implementation details l.jpg
Implementation Details

TestSuite(T)

Test Transformer

Program Under Test (P)

New TestSuite(T ’)

Knapsack Solver

CoverageCalculator

  • Knapsack Solver Parameters1. Selected Solver2. Reduction Preference3. Knapsack Size


Evaluation metrics l.jpg
Evaluation Metrics

  • Code coverage: Percentage of requirements executed when prioritization is run

    • Basic block coverage used

  • Coverage preservation: Proportion of code covered by prioritization versus code covered by entire original test suite

  • Order-aware coverage: Considers both the order in which test cases execute in addition to overall code coverage


Experiment design l.jpg
Experiment Design

  • Goals of experiment:

    • Measure efficiency of algorithms and scaling in terms of time and space overhead

    • Measure effectiveness of algorithms and scaling in terms of three coverage-based metrics

  • Case studies:

    • JDepend

    • Gradebook

  • Knapsack Size

    • 25, 50, and 75% of execution time of original test suite


Summary of experimental results l.jpg
Summary of Experimental Results

  • Prioritizer Effectiveness:

    • Overlap-aware solver had highest overall coverage for each time limit

      • Greedy by Value solver good for Gradebook

      • All Greedy solvers good for JDepend

  • Prioritizer Efficiency:

    • All algorithms took small amount of time and memory except for Dynamic Programming, Generalized Tabular, and Core

    • Overlap-aware solver required hours to run

    • Generalized Tabular had prohibitively large memory requirements

    • Scaling heuristic reduced overhead in some cases


Conclusions l.jpg
Conclusions

  • Most sophisticated algorithm not necessarily most effective or most efficient

  • Trade-off: effectiveness versus efficiency

  • Efficiency or effectiveness most important?

    • Effectiveness  overlap-aware prioritizer

    • Efficiency  low-overhead prioritizer

  • Prioritizer choice depends on test suite nature

    • Time versus coverage of each test case

    • Coverage overlap between test cases


Future research l.jpg
Future Research

  • Use larger case studies with bigger test suites

  • Use case studies written in other languages

  • Evaluate other knapsack solvers such as branch-and-bound and parallel solvers

  • Incorporate other metrics such as APFD

  • Use synthetically generated test suites


Thank you l.jpg

Thank you!

Questions?

http://www.cs.virginia.edu/walcott/weasel.html



ad