efficient time aware prioritization with knapsack solvers l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Efficient Time-Aware Prioritization with Knapsack Solvers PowerPoint Presentation
Download Presentation
Efficient Time-Aware Prioritization with Knapsack Solvers

Loading in 2 Seconds...

play fullscreen
1 / 22

Efficient Time-Aware Prioritization with Knapsack Solvers - PowerPoint PPT Presentation


  • 376 Views
  • Uploaded on

Efficient Time-Aware Prioritization with Knapsack Solvers. Sara Alspaugh Kristen R. Walcott Mary Lou Soffa University of Virginia. Michael Belanich Gregory M. Kapfhammer Allegheny College. Test Suite Prioritization . Testing occurs throughout software development life cycle

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

Efficient Time-Aware Prioritization with Knapsack Solvers


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
efficient time aware prioritization with knapsack solvers

Efficient Time-Aware Prioritization with Knapsack Solvers

Sara AlspaughKristen R. WalcottMary Lou Soffa

University of Virginia

Michael BelanichGregory M. Kapfhammer

Allegheny College

ACM WEASEL Tech

test suite prioritization
Test Suite Prioritization
  • Testing occurs throughout software development life cycle
    • Challenge: time consuming and costly
  • Prioritization: reordering the test suite
    • Goal: find errors sooner in testing
      • Doesn’t consider the overall time budget
    • Alternative: time-aware prioritization
      • Goal 1: find errors sooner in testing
      • Goal 2: execute within time constraint
motivating example
Motivating Example

Original test suite with fault information

T1

4 faults

2 min.

T3

2 faults

2 min.

T4

6 faults

2 min.

T2

1 fault

2 min.

Assume: - Same execution time - Unique faults found

Prioritized test suite

T4

6 faults

2 min.

T1

4 faults

2 min.

T3

2 faults

2 min.

T2

1 fault

2 min.

Testing time budget: 4 minutes

the knapsack problem for time aware prioritization

n

P

n

P

¤

c

x

·

t

t

i

i

¤

x

i

1

i

i

t

=

i

1

m

a

x

=

x

i

i

t

i

m

a

x

i

c

i

The Knapsack Problem for Time-Aware Prioritization

Maximize: , where is the code coverage of test and is either 0 or 1.

Subject to the constraint:

where is the execution time of test and is the time budget.

the knapsack problem for time aware prioritization5
The Knapsack Problem for Time-Aware Prioritization

Assume test cases cover unique requirements.

T1

4 lines

2 min.

T2

1 line

2 min.

Time Budget: 4 min.

T3

2 lines

2 min.

T4

5 lines

2 min.

Total Value:

Space Remaining:

5

2 min.

9

0 min.

0

4 min.

the extended knapsack problem
The Extended Knapsack Problem
  • Value of each test case depends on test cases already in prioritization
    • Test cases may cover same requirements

T1

4 lines

2 min.

T1

0 lines

2 min.

T2

1 line

2 min.

Time Budget: 4 min.

T3

2 lines

2 min.

T4

5 lines

2 min.

Total Value:

Space Remaining:

5

2 min.

7

0 min.

0

4 min.

UPDATE

goals and challenges
Goals and Challenges
  • Evaluate traditional and extended knapsack solvers for use in time-aware prioritization
    • Effectiveness
      • Coverage-based metrics
    • Efficiency
      • Time overhead
      • Memory overhead
  • How does overlapping code coverage affect results of traditional techniques?
  • Is the cost of extended knapsack algorithms worthwhile?
the knapsack solvers
The Knapsack Solvers
  • Random: select tests cases at random
  • Greedy by Ratio: order by coverage/time
    • Greedy by Value: order by coverage
    • Greedy by Weight: order by time
  • Dynamic Programming: break problem into sub-problems; use sub-problem results for main solution
  • Generalized Tabular: use large tables to store sub-problem solutions
the knapsack solvers continued
The Knapsack Solvers (continued)
  • Core: compute optimal fractional solution then exchange items until optimal integral solution found
  • Overlap-Aware: uses a genetic algorithm to solve the extended knapsack problem for time-aware prioritization
the scaling heuristic

j

k

³

´

t

t

¸

£

£

m

a

x

m

a

x

c

c

1

2

h

i

T

T

t

t

[

]

1

2

T

1

1

1

2

¡

x

n

x

;

:

:

:

c

c

c

T

T

x

1

¸

2

¸

¸

;

;

n

i

1

:

:

:

t

t

t

1

2

n

The Scaling Heuristic
  • Order the test cases by their coverage-to-execution-time ratio such that:
  • If , then it is possible to find an optimal solution that includes .
  • Check the inequality for each test case until it no longer holds.
  • belong in the final prioritization.
implementation details
Implementation Details

TestSuite(T)

Test Transformer

Program Under Test (P)

New TestSuite(T ’)

Knapsack Solver

CoverageCalculator

  • Knapsack Solver Parameters1. Selected Solver2. Reduction Preference3. Knapsack Size
evaluation metrics
Evaluation Metrics
  • Code coverage: Percentage of requirements executed when prioritization is run
    • Basic block coverage used
  • Coverage preservation: Proportion of code covered by prioritization versus code covered by entire original test suite
  • Order-aware coverage: Considers both the order in which test cases execute in addition to overall code coverage
experiment design
Experiment Design
  • Goals of experiment:
    • Measure efficiency of algorithms and scaling in terms of time and space overhead
    • Measure effectiveness of algorithms and scaling in terms of three coverage-based metrics
  • Case studies:
    • JDepend
    • Gradebook
  • Knapsack Size
    • 25, 50, and 75% of execution time of original test suite
summary of experimental results
Summary of Experimental Results
  • Prioritizer Effectiveness:
    • Overlap-aware solver had highest overall coverage for each time limit
      • Greedy by Value solver good for Gradebook
      • All Greedy solvers good for JDepend
  • Prioritizer Efficiency:
    • All algorithms took small amount of time and memory except for Dynamic Programming, Generalized Tabular, and Core
    • Overlap-aware solver required hours to run
    • Generalized Tabular had prohibitively large memory requirements
    • Scaling heuristic reduced overhead in some cases
conclusions
Conclusions
  • Most sophisticated algorithm not necessarily most effective or most efficient
  • Trade-off: effectiveness versus efficiency
  • Efficiency or effectiveness most important?
    • Effectiveness  overlap-aware prioritizer
    • Efficiency  low-overhead prioritizer
  • Prioritizer choice depends on test suite nature
    • Time versus coverage of each test case
    • Coverage overlap between test cases
future research
Future Research
  • Use larger case studies with bigger test suites
  • Use case studies written in other languages
  • Evaluate other knapsack solvers such as branch-and-bound and parallel solvers
  • Incorporate other metrics such as APFD
  • Use synthetically generated test suites
thank you
Thank you!Questions?

http://www.cs.virginia.edu/walcott/weasel.html