Efficient Time-Aware Prioritization with Knapsack Solvers - PowerPoint PPT Presentation

Efficient time aware prioritization with knapsack solvers l.jpg
Download
1 / 22

  • 342 Views
  • Uploaded on
  • Presentation posted in: Pets / Animals

Efficient Time-Aware Prioritization with Knapsack Solvers. Sara Alspaugh Kristen R. Walcott Mary Lou Soffa University of Virginia. Michael Belanich Gregory M. Kapfhammer Allegheny College. Test Suite Prioritization. Testing occurs throughout software development life cycle

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Efficient Time-Aware Prioritization with Knapsack Solvers

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Efficient time aware prioritization with knapsack solvers l.jpg

Efficient Time-Aware Prioritization with Knapsack Solvers

Sara AlspaughKristen R. WalcottMary Lou Soffa

University of Virginia

Michael BelanichGregory M. Kapfhammer

Allegheny College

ACM WEASEL Tech


Test suite prioritization l.jpg

Test Suite Prioritization

  • Testing occurs throughout software development life cycle

    • Challenge: time consuming and costly

  • Prioritization: reordering the test suite

    • Goal: find errors sooner in testing

      • Doesn’t consider the overall time budget

    • Alternative: time-aware prioritization

      • Goal 1: find errors sooner in testing

      • Goal 2: execute within time constraint


Motivating example l.jpg

Motivating Example

Original test suite with fault information

T1

4 faults

2 min.

T3

2 faults

2 min.

T4

6 faults

2 min.

T2

1 fault

2 min.

Assume:- Same execution time- Unique faults found

Prioritized test suite

T4

6 faults

2 min.

T1

4 faults

2 min.

T3

2 faults

2 min.

T2

1 fault

2 min.

Testing time budget: 4 minutes


The knapsack problem for time aware prioritization l.jpg

n

P

n

P

¤

c

x

·

t

t

i

i

¤

x

i

1

i

i

t

=

i

1

m

a

x

=

x

i

i

t

i

m

a

x

i

c

i

The Knapsack Problem for Time-Aware Prioritization

Maximize: , where is the code coverage of test and is either 0 or 1.

Subject to the constraint:

where is the execution time of test and is the time budget.


The knapsack problem for time aware prioritization5 l.jpg

The Knapsack Problem for Time-Aware Prioritization

Assume test cases cover unique requirements.

T1

4 lines

2 min.

T2

1 line

2 min.

Time Budget: 4 min.

T3

2 lines

2 min.

T4

5 lines

2 min.

Total Value:

Space Remaining:

5

2 min.

9

0 min.

0

4 min.


The extended knapsack problem l.jpg

The Extended Knapsack Problem

  • Value of each test case depends on test cases already in prioritization

    • Test cases may cover same requirements

T1

4 lines

2 min.

T1

0 lines

2 min.

T2

1 line

2 min.

Time Budget: 4 min.

T3

2 lines

2 min.

T4

5 lines

2 min.

Total Value:

Space Remaining:

5

2 min.

7

0 min.

0

4 min.

UPDATE


Goals and challenges l.jpg

Goals and Challenges

  • Evaluate traditional and extended knapsack solvers for use in time-aware prioritization

    • Effectiveness

      • Coverage-based metrics

    • Efficiency

      • Time overhead

      • Memory overhead

  • How does overlapping code coverage affect results of traditional techniques?

  • Is the cost of extended knapsack algorithms worthwhile?


The knapsack solvers l.jpg

The Knapsack Solvers

  • Random: select tests cases at random

  • Greedy by Ratio: order by coverage/time

    • Greedy by Value: order by coverage

    • Greedy by Weight: order by time

  • Dynamic Programming: break problem into sub-problems; use sub-problem results for main solution

  • Generalized Tabular: use large tables to store sub-problem solutions


The knapsack solvers continued l.jpg

The Knapsack Solvers (continued)

  • Core: compute optimal fractional solution then exchange items until optimal integral solution found

  • Overlap-Aware: uses a genetic algorithm to solve the extended knapsack problem for time-aware prioritization


The scaling heuristic l.jpg

j

k

³

´

t

t

¸

£

£

m

a

x

m

a

x

c

c

1

2

h

i

T

T

t

t

[

]

1

2

T

1

1

1

2

¡

x

n

x

;

:

:

:

c

c

c

T

T

x

1

¸

2

¸

¸

;

;

n

i

1

:

:

:

t

t

t

1

2

n

The Scaling Heuristic

  • Order the test cases by their coverage-to-execution-time ratio such that:

  • If , then it is possible to find an optimal solution that includes .

  • Check the inequality for each test case until it no longer holds.

  • belong in the final prioritization.


Implementation details l.jpg

Implementation Details

TestSuite(T)

Test Transformer

Program Under Test (P)

New TestSuite(T ’)

Knapsack Solver

CoverageCalculator

  • Knapsack Solver Parameters1. Selected Solver2. Reduction Preference3. Knapsack Size


Evaluation metrics l.jpg

Evaluation Metrics

  • Code coverage: Percentage of requirements executed when prioritization is run

    • Basic block coverage used

  • Coverage preservation: Proportion of code covered by prioritization versus code covered by entire original test suite

  • Order-aware coverage: Considers both the order in which test cases execute in addition to overall code coverage


Experiment design l.jpg

Experiment Design

  • Goals of experiment:

    • Measure efficiency of algorithms and scaling in terms of time and space overhead

    • Measure effectiveness of algorithms and scaling in terms of three coverage-based metrics

  • Case studies:

    • JDepend

    • Gradebook

  • Knapsack Size

    • 25, 50, and 75% of execution time of original test suite


Summary of experimental results l.jpg

Summary of Experimental Results

  • Prioritizer Effectiveness:

    • Overlap-aware solver had highest overall coverage for each time limit

      • Greedy by Value solver good for Gradebook

      • All Greedy solvers good for JDepend

  • Prioritizer Efficiency:

    • All algorithms took small amount of time and memory except for Dynamic Programming, Generalized Tabular, and Core

    • Overlap-aware solver required hours to run

    • Generalized Tabular had prohibitively large memory requirements

    • Scaling heuristic reduced overhead in some cases


Conclusions l.jpg

Conclusions

  • Most sophisticated algorithm not necessarily most effective or most efficient

  • Trade-off: effectiveness versus efficiency

  • Efficiency or effectiveness most important?

    • Effectiveness  overlap-aware prioritizer

    • Efficiency  low-overhead prioritizer

  • Prioritizer choice depends on test suite nature

    • Time versus coverage of each test case

    • Coverage overlap between test cases


Future research l.jpg

Future Research

  • Use larger case studies with bigger test suites

  • Use case studies written in other languages

  • Evaluate other knapsack solvers such as branch-and-bound and parallel solvers

  • Incorporate other metrics such as APFD

  • Use synthetically generated test suites


Thank you l.jpg

Thank you!

Questions?

http://www.cs.virginia.edu/walcott/weasel.html


Case study applications l.jpg

Case Study Applications


  • Login