Loading in 5 sec....

Establishing Theoretical Minimal Sets of Mutants ICST 2014PowerPoint Presentation

Establishing Theoretical Minimal Sets of Mutants ICST 2014

- By
**enid** - Follow User

- 130 Views
- Uploaded on

Download Presentation
## PowerPoint Slideshow about ' Establishing Theoretical Minimal Sets of Mutants ICST 2014' - enid

**An Image/Link below is provided (as is) to download presentation**

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

### Establishing Theoretical Minimal Sets of MutantsICST 2014

Paul Ammann

Joint work with

Marcio Eduardo Delamaro

Jeff Offutt

April 1, 2014

Outline

- The situation
- Researchers use mutation analysis to evaluate test selection strategies

- The problem
- What do mutation scores mean?

- The model
- Motivating idea: Minimal mutant sets don’t have redundant mutants
- Need to define notion of redundancy

- Main result: Dynamic subsumption = Minimal mutant sets

- Motivating idea: Minimal mutant sets don’t have redundant mutants
- Reduced mutation: Is it close to minimal?
- Apply model to Siemens suite

- Result: Huge gap
- Good news: That’s an opportunity!

Researchers Use Mutation Analysis to Evaluate Test Selection Strategies

Test Set C

Select Test Sets with

Test Selection Strategies

Carefully Chosen

Artifacts

Test Set B

Test Set A

Deep Analysis

Measure

“Good”

Tests

Test Selection

Strategy C

Test Selection

Strategy B

Test Selection

Strategy A

Exactly What Does A Score of 91% Mean?

The Problem With Mutation Scores Strategies

Mutation Scores for 3 Test Sets

Evaluate 3 Test Sets with 4 Mutants:

A: {t1, t2}

B: {t2, t5}

C: {t3}

Bscores 75%

Is that good?

Let’s Add Some More Mutants Strategies

The same tests kill m3 and m6. We say that T does not distinguish m3 from m6

Every test kills m8

What’s the point?

Ditto for m5 and m9

Mutation Scores for 3 Test Sets

Evaluate 3 Test Sets with 10 Mutants:

A: {t1, t2}

B: {t2, t5}

C: {t3}

NowBscores 90%!

Did B just get better?

Let’s Throw Away Some Mutants Strategies

Evaluate 3 Test Sets with 2 Mutants:

A: {t1, t2}

B: {t2, t5}

C: {t3}

Mutation Scores for 3 Test Sets

Now B scores 100%

Did B get even better?

All Together Now Strategies

Evaluate 3 Test Sets with Various Mutants:

A: {t1, t2}

B: {t2, t5}

C: {t3}

Cumulative Scores

Is Blousy or good?

What about C?

What Makes a Mutant Redundant? Strategies

Basic Idea:

Throwing away a redundant mutant has no effect on the minimal test sets.

Choose M = {m1, m2, m3, m4}

Choose T= {t1, t2, t3, t4, t5}

Minimal test sets wrtM and T: {t1, t2}, {t1, t3}, {t4}

Try removing m4: M4 = M - {m4}

Minimal test sets wrtM4 and T: {t1, t2}, {t1, t3}, {t2, t5}, {t3, t5}, {t4}

A change, so m4 is not redundant

Try removingm3: M3 = M - {m3}

Minimal test sets wrtM3 and T: {t1}, {t4}

A change, so m3 is not redundant

Try removing m1: M1 = M - {m1}

Minimal test sets wrtM1 and T: {t1, t2}, {t1, t3}, {t4}

No change, so m1is redundant

Ditto for M2 = M - {m2}

Minimal Sets of Mutants Strategies

- Definition
- M is minimal if it does not contain redundant mutants

- Minimal mutant sets from the definition
- Requires computing all minimal test sets, which is NP complete

- We need an efficient algorithm for finding minimal mutant sets
- Turn to dynamicsubsumption
- Subsumption with respect to a test set

- Turn to dynamicsubsumption

Dynamic StrategiesSubsumption

Test set T

Tests that kill mj

Tests that kill mk

Tests that kill mi

✔

✖

?

?

mi → mj

mi → mk

Efficiently Computing Minimal Sets of Mutants Strategies

- Formally: mxdynamically subsumesmywrtTiff
- Some testin T kills mx
- Every testin T that kills mx also killsmy

- Main result:
Mutant set M minimal wrtT

=

no dynamic subsumption in M

- Properties
- Only need to consider mutants in pairs
- Groups of mutants do not make another mutant redundant

- Fast
- Every minimal mutant set has the same cardinality
- Contrast with minimal test sets

- Only need to consider mutants in pairs

What Does This Mean in Practice? Strategies

- Apply the definitions to the Siemens test bed
- See what happens!

- 7 programs
- print_tokens
- print_tokens2
- replace
- schedule
- schedule2
- tcas
- totinfo

- Extensive hand-crafted test set

Test Characteristics Strategies

- Notes:
- 512 is an artifact of the Proteum tool
- Our approach applies with any test set

- Most tests used were also distinguished
- Minimal test set size modest compared to number of tests used

- 512 is an artifact of the Proteum tool

Mutant Characteristics Strategies

- “Killed Mutants” means those killed by the test set of size 512
- Vast majority of remainder are equivalent

- Most mutants are redundant!
- Tiny fraction of mutants are actually minimal wrt 512 tests!
- print_tokens: Killing the right 28 mutants guarantees killing all 3711

How Good Are Reduced Mutation Strategies? Strategies

- We considered five approaches to reduced mutation
- STMT: Statement deletion (Proteum SSDL)
- ROR: Relation operators (Proteum ORRN)
- CON: Replace scalars with constants (Proteum CCSR)
- 5RND: 5% Random selection of all mutants
- SELECT: “Selective” mutation (Proteum OOAN, OLLN, ORRN, OLNG)

- Method:
- Choose test sets adequate for each reducedmutation approach
- wrt test sets analyzed earlier

- Compute mutation score
- Against all mutants
- Against minimal mutant set
- Equivalent mutants hand-identified and removed

- Choose test sets adequate for each reducedmutation approach

Reduced Mutation Scores: Raw vs. Minimal Strategies

- Notes:
- Table entries: Raw Mutation Score: Minimal Mutation Score
- Raw Reduced mutation scores make test strategies look good
- Minimal Reduced mutation scores do not

Closer Look: StrategiesRaw vs. Minimal for STMT

- Raw mutation scores show little variation
- Minimal mutation scores show a lot

Reduced Mutation: Mutants vs. Tests Strategies

- Notes:
- Table entries: Number of Mutants : Size of Minimal Test Set
- Reduced approaches
- Generate many more mutants than minimal
- But not nearly enough tests

Closer Look: StrategiesMutants and Tests for STMT

- STMT usually generates too many mutants
- Unfortunately, they aren’t the right ones
- Hence, not nearly enough tests

Discussion Strategies

- Huge gap: Reduced mutation vs. minimal mutant sets
- Research opportunity!

- The problem with reduced mutation
- Reduced approaches don’t consider specific program under test
- Maybe it’s time to change that
- Can we analyze specific mutants in a specific program?

- Problem with minimal mutant sets for practical testing
- Need mutation adequate tests to compute minimal mutant sets!
- Aren’t we done at that point?

- Need mutation adequate tests to compute minimal mutant sets!
- There is a lot we don’t know about minimal mutant sets
- Let’s look at an example from yesterday’s Mutation workshop

Subsumption Strategies Graph Example: cal()

- 31 nodes of indistinguished mutants
- 7 nodes of minimal mutants
- muJava generated 145 non-equivalent mutants
- we only need 7 for given test set

- Static analysis can refine this graph

Questions? Strategies

- Contact:
- {pammann, [email protected]
- [email protected]

Download Presentation

Connecting to Server..