Establishing theoretical minimal sets of mutants icst 2014
Download
1 / 22

Establishing Theoretical Minimal Sets of Mutants ICST 2014 - PowerPoint PPT Presentation


  • 134 Views
  • Uploaded on

Establishing Theoretical Minimal Sets of Mutants ICST 2014. Paul Ammann Joint work with Marcio Eduardo Delamaro Jeff Offutt April 1, 2014. Outline. The situation Researchers use mutation analysis to evaluate test selection strategies The problem What do mutation scores mean?

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Establishing Theoretical Minimal Sets of Mutants ICST 2014' - enid


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Establishing theoretical minimal sets of mutants icst 2014

Establishing Theoretical Minimal Sets of MutantsICST 2014

Paul Ammann

Joint work with

Marcio Eduardo Delamaro

Jeff Offutt

April 1, 2014


Outline
Outline

  • The situation

    • Researchers use mutation analysis to evaluate test selection strategies

  • The problem

    • What do mutation scores mean?

  • The model

    • Motivating idea: Minimal mutant sets don’t have redundant mutants

      • Need to define notion of redundancy

    • Main result: Dynamic subsumption = Minimal mutant sets

  • Reduced mutation: Is it close to minimal?

    • Apply model to Siemens suite

  • Result: Huge gap

    • Good news: That’s an opportunity!


Researchers use mutation analysis to evaluate test selection strategies
Researchers Use Mutation Analysis to Evaluate Test Selection Strategies

Test Set C

Select Test Sets with

Test Selection Strategies

Carefully Chosen

Artifacts

Test Set B

Test Set A

Deep Analysis

Measure

“Good”

Tests

Test Selection

Strategy C

Test Selection

Strategy B

Test Selection

Strategy A

Exactly What Does A Score of 91% Mean?


The problem with mutation scores
The Problem With Mutation Scores Strategies

Mutation Scores for 3 Test Sets

Evaluate 3 Test Sets with 4 Mutants:

A: {t1, t2}

B: {t2, t5}

C: {t3}

Bscores 75%

Is that good?


Let s add some more mutants
Let’s Add Some More Mutants Strategies

The same tests kill m3 and m6. We say that T does not distinguish m3 from m6

Every test kills m8

What’s the point?

Ditto for m5 and m9

Mutation Scores for 3 Test Sets

Evaluate 3 Test Sets with 10 Mutants:

A: {t1, t2}

B: {t2, t5}

C: {t3}

NowBscores 90%!

Did B just get better?


Let s throw away some mutants
Let’s Throw Away Some Mutants Strategies

Evaluate 3 Test Sets with 2 Mutants:

A: {t1, t2}

B: {t2, t5}

C: {t3}

Mutation Scores for 3 Test Sets

Now B scores 100%

Did B get even better?


All together now
All Together Now Strategies

Evaluate 3 Test Sets with Various Mutants:

A: {t1, t2}

B: {t2, t5}

C: {t3}

Cumulative Scores

Is Blousy or good?

What about C?


What makes a mutant redundant
What Makes a Mutant Redundant? Strategies

Basic Idea:

Throwing away a redundant mutant has no effect on the minimal test sets.

Choose M = {m1, m2, m3, m4}

Choose T= {t1, t2, t3, t4, t5}

Minimal test sets wrtM and T: {t1, t2}, {t1, t3}, {t4}

Try removing m4: M4 = M - {m4}

Minimal test sets wrtM4 and T: {t1, t2}, {t1, t3}, {t2, t5}, {t3, t5}, {t4}

A change, so m4 is not redundant

Try removingm3: M3 = M - {m3}

Minimal test sets wrtM3 and T: {t1}, {t4}

A change, so m3 is not redundant

Try removing m1: M1 = M - {m1}

Minimal test sets wrtM1 and T: {t1, t2}, {t1, t3}, {t4}

No change, so m1is redundant

Ditto for M2 = M - {m2}


Minimal sets of mutants
Minimal Sets of Mutants Strategies

  • Definition

    • M is minimal if it does not contain redundant mutants

  • Minimal mutant sets from the definition

    • Requires computing all minimal test sets, which is NP complete 

  • We need an efficient algorithm for finding minimal mutant sets

    • Turn to dynamicsubsumption

      • Subsumption with respect to a test set


Dynamic subsumption
Dynamic StrategiesSubsumption

Test set T

Tests that kill mj

Tests that kill mk

Tests that kill mi

?

?

mi → mj

mi → mk


Efficiently computing minimal sets of mutants
Efficiently Computing Minimal Sets of Mutants Strategies

  • Formally: mxdynamically subsumesmywrtTiff

    • Some testin T kills mx

    • Every testin T that kills mx also killsmy

  • Main result:

    Mutant set M minimal wrtT

    =

    no dynamic subsumption in M

  • Properties

    • Only need to consider mutants in pairs

      • Groups of mutants do not make another mutant redundant

    • Fast 

    • Every minimal mutant set has the same cardinality

      • Contrast with minimal test sets


What does this mean in practice
What Does This Mean in Practice? Strategies

  • Apply the definitions to the Siemens test bed

    • See what happens!

  • 7 programs

    • print_tokens

    • print_tokens2

    • replace

    • schedule

    • schedule2

    • tcas

    • totinfo

  • Extensive hand-crafted test set


Test characteristics
Test Characteristics Strategies

  • Notes:

    • 512 is an artifact of the Proteum tool

      • Our approach applies with any test set

    • Most tests used were also distinguished

    • Minimal test set size modest compared to number of tests used


Mutant characteristics
Mutant Characteristics Strategies

  • “Killed Mutants” means those killed by the test set of size 512

    • Vast majority of remainder are equivalent

  • Most mutants are redundant!

    • Tiny fraction of mutants are actually minimal wrt 512 tests!

    • print_tokens: Killing the right 28 mutants guarantees killing all 3711


How good are reduced mutation strategies
How Good Are Reduced Mutation Strategies? Strategies

  • We considered five approaches to reduced mutation

    • STMT: Statement deletion (Proteum SSDL)

    • ROR: Relation operators (Proteum ORRN)

    • CON: Replace scalars with constants (Proteum CCSR)

    • 5RND: 5% Random selection of all mutants

    • SELECT: “Selective” mutation (Proteum OOAN, OLLN, ORRN, OLNG)

  • Method:

    • Choose test sets adequate for each reducedmutation approach

      • wrt test sets analyzed earlier

    • Compute mutation score

      • Against all mutants

      • Against minimal mutant set

      • Equivalent mutants hand-identified and removed


Reduced mutation scores raw vs minimal
Reduced Mutation Scores: Raw vs. Minimal Strategies

  • Notes:

    • Table entries: Raw Mutation Score: Minimal Mutation Score

    • Raw Reduced mutation scores make test strategies look good

    • Minimal Reduced mutation scores do not


Closer look raw vs minimal for stmt
Closer Look: StrategiesRaw vs. Minimal for STMT

  • Raw mutation scores show little variation

  • Minimal mutation scores show a lot


Reduced mutation mutants vs tests
Reduced Mutation: Mutants vs. Tests Strategies

  • Notes:

    • Table entries: Number of Mutants : Size of Minimal Test Set

    • Reduced approaches

      • Generate many more mutants than minimal

      • But not nearly enough tests


Closer look mutants and tests for stmt
Closer Look: StrategiesMutants and Tests for STMT

  • STMT usually generates too many mutants

  • Unfortunately, they aren’t the right ones

    • Hence, not nearly enough tests


Discussion
Discussion Strategies

  • Huge gap: Reduced mutation vs. minimal mutant sets

    • Research opportunity!

  • The problem with reduced mutation

    • Reduced approaches don’t consider specific program under test

    • Maybe it’s time to change that

      • Can we analyze specific mutants in a specific program?

  • Problem with minimal mutant sets for practical testing

    • Need mutation adequate tests to compute minimal mutant sets!

      • Aren’t we done at that point?

  • There is a lot we don’t know about minimal mutant sets

    • Let’s look at an example from yesterday’s Mutation workshop


Subsumption graph example cal
Subsumption Strategies Graph Example: cal()

  • 31 nodes of indistinguished mutants

  • 7 nodes of minimal mutants

    • muJava generated 145 non-equivalent mutants

    • we only need 7 for given test set

  • Static analysis can refine this graph


Questions
Questions? Strategies

  • Contact:

    • {pammann, offutt}@gmu.edu

    • delamaro@icmc.usp.br