Mutation testing ch 5
1 / 24

Mutation Testing (ch 5) - PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Mutation Testing (ch 5). Course Software Testing & Verification 2013/14 Wishnu Prasetya. Negative Test. Recall: to test a program outside its normal behavior range. Idea: mutate inputs.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Mutation Testing (ch 5)

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Mutation Testing(ch 5)

Course Software Testing & Verification



Negative Test

  • Recall: to test a program outside its normal behavior range.

  • Idea: mutate inputs.

  • For inputs described by BNF we can mutate the grammar instead (5.5.2), e.g. replace/delete/ duplicate terminals and non-terminals in a production rule.

  • Or mutate on-the-fly as we build derivation trees (so that we can control how many times a mutated rule is applied).


NLpostcode Street Area

NLpostcode Area Space Street

Area  FirstDigit Digit Digit Digit

Street  Letter Letter

FirstDigit 1 | 2 ...

Digit  0 | 1 | 2 ...

Letter  a | b | c ... | A | B | C ...

Space   | “ “ Space

Area  FirstDigit Digit Digit

Area  FirstDigit Digit DigitDigitDigit

Some coverage concepts for grammar-based mutants generation

  • (Def 5.46) Mutation operator (e.g. non-terminal deletion); (Def 5.47) mutant

  • (C 5.33) Mutation Operator Coverage (MOC) : for every mutation operator TR contains one mutant produced via this operator.

  • (C5.34) Mutation Production Cov: for every production rule and every mutation operator that can be applied to the rule, TR contains one mutant via that rule and mutation.

Using mutation as “coverage” (5.2)

  • We can “mutate” a program to “fake” a mistake; but the mutant must be a valid program.

  • (D 5.45) Ground : the original.

  • (D 5.48) Let P be a ground. A test t of P kills a mutant P’ of P if t’s output of both differ.

  • We can use mutants to measure the adequacy of our test-set: it is good if it can kill all those mutants.

  • Obviously, the strength of this scheme depends on your choice of the mutation operators.


Fundamental premise : If a program contains a fault, there will usually be a set of mutants that can only be killed by test cases that also detect that fault.

(mutations generated by the tool Mothra)

We can also do: mutation-driven testing

Fundamental premise : If a program contains a fault, there will usually be a set of mutants that can only be killed by test cases that also detect that fault.

not enough mutants percentage killed

a test-case fails

  • Things to note (see next slide) :

  • reachability of a mutation

  • infection

  • propagation

This is a simpler variation of the scheme in Fig. 5.2. The one above does not presume automated generation of test-cases.

How to kill a mutant?

P(x) { if (x<0) x++ ;

return x }

P(x) { if (x0) x++ ;

return x }

  • (reachability) you need a test-case whose execution passes the mutation.

  • You need a test-case that actually observes the effect of the mutation. Note there is a difference in:

    • (infection) “incorrect” state, right after the mutation.

    • (propagation) the infection propagates to incorrect P’s output. In principle, a test-case can only kill if there is propagation.

test() { r = P(1) ; assert (r==2) }

test() { r = P(0) ; assert (r==0) }

Another example

  • booleanisEven (int X) {

  • if (X < 0)

  • 3 X = 0 – X ;

  • ∆ 3 X = 0 ;

  • if ((double) (X/2) == ((double) X) / 2.0)

  • return true ;

  • else return false ;

  • }

test1() { assert (isEven(-2)==true) }

Issues with killing

  • No reachability  usually can easily be fixed.

  • You have propagation, but used oracle is only partial (is not strong enough).  fix the oracle.

  • You have infection, but no propagation  as in the isEven example  find another test-case.

  • What if we weaken the definition of “kill” ? We can at least see what that gives us.

Strong and weak kill

  • Let t be a test on a ground P, and P’ is a mutant. Let l be the location of the mutation in P’.

  • (D 5.49) t strongly kills P’ if its verdict on P’ and P are different. In other words, if it fails on P’.

  • ( D 5.50) t weakly kills P’ if the states right after l when executing t on P’ and P are different.

P(x) { if (x<0) x++ ;

return x }

P(x) { if (x0) x++ ;

return x }

test() { r = P(1) ; assert (r0) }

does not even weakly kill

test() { r = P(0) ; assert (r0) }

does not kill, but still weakly kill

Mutation-based coverage criteria

  • (C5.32) Given a set M of mutants, the TR is simply M: you have to kill every mutant in M.

  • C5.35 if you use “strong killing”, C5.36 for “weak killing”.

  • OA remark that in practice strong vs weak do not make much difference.

  • The strength depends on the choice of mutation operators. See 5.2.2 and 5.3.2.

  • Typical problem in mutation-based testing: it generates lots of mutants, roughly O(#dataobjs*#refs)  make testing very computation intensive!

  • O(#refs) using selective mutation (next slide)

Mutation operators

  • This depends on the used programming language.

  • We can start with some set of mutation operators, then see if we can minimize it.

  • (D5.51) Given a set O of mutation operators; a subset O1 is effective if test-cases designed specifically to kill mutants produced by O1 also kill mutants from O/ O1.

  • There have been plenty of research in this direction. For mutations over imperative constructs, see 5.2.2. Mutations on OO aspects, see 5.3.2.

Examples of mutops on imperative constructs (5.2.2)

  • ABS (Absolute Value Insertion)Modify each arithmetic (sub)expression by using functions abs(), negAbs(), and failOnZero().

  • AOR (Arithmetic Operator Replacement)Replace each occurrence of arithmetic operators +,-,*,/, % by each other; and in addition, by leftOp, and rightOp.

  • SVR (Scalar Variable Replacement)Replace each variable reference by another (type compatible and in-scope) variable.

  • BSR (Bomb Statement Replacement)Replace each statement by Bomb().

  • More... see book.

Addressing integration issues (5.3.2)

  • Faults in component integration are caused by a mismatch of assumptions; e.g. callee thought:

    • a list was sorted, caller did not do that

    • all fields were initialized, caller only initialized some

    • thought values are in kilometers, caller use miles

  • Integration mutation focuses on mutating the connections between components

    • Sometimes called “interface mutation”

    • Both caller and callee methods are considered

Four types of integration mutations

  • Change a calling method by modifying the call (e.g. deleting the call; replacing the returned obj by a type-compatible obj).

  • Change a calling method by modifying values that are sent to a called method.

  • Change a called method by modifying values that enter and leave a method

    • Includes parameters as well as variables from higher scopes (class level, package, public, etc.)

How about OO?These are typical OO features; also where typical mistakes are made

  • Encapsulation (hiding details).

Typical OO aspects

  • Inheritance: method overriding, variables shadowing, constructors overriding works differently

  • Polymorphism

    • a required object of type T can be instantiated by a subclass at the runtime

    • overloading

  • In addition to object level members we also have class level members (the ‘static’).

  • 5.3.3 lists 20 mutops targeting the above aspects.

Examples of inheritance mutops

  • OMD (Overriding Method Deletion)Each entire declaration of an overriding method is deleted.

  • OMM (Overridden Method Moving)Each call to a super.method is moved to the first and last position, and up and down one statement.

override m() { x ++ ; y = y/x ;

base.m() ; // super}

Examples of inheritance mutops

HVI – Hiding Variable Insertion

HVD – Hiding Variable Deletion


int x;

int y;


int x;

int y;


1 int x;

2 int y;


int x;

1 // int x;

int y;

2 // int y;

Examples of inheritance mutops



ATC – Actual Type Change

DTC – Declared Type Change

point p;

p = PointFactory();

 p = new colorpoint ();

point p;

colorpoint p;

p = PointFactory();


void set (int x, int y, char c);

void set (char a, int x, int y);


void set (int x, int y, int z);

void set (int x, int y);

void set (int z);

Examples of overloading mutops

AOC – Argument Order Change

ANC – Argument Number Change


point3D p ;

p.set (1, 2, ‘t’) ;

 p.set (‘t’, 1, 2) ;

point3D p;

p.set (1, 2, 3);

 p.set (2, 3);

 p.set (3);


int x = 5;

int x;


point() { … }

// point() { … }


void set (int x, int y)


this.x = x;

1 x = x;

this.y = y;

2 y = y;


Examples of “other” mutops

VID –Variable Initialization Deletion

DCD – Default Constructor Delete

TKD – This Keyword Deletion

Ok... but how useful is this !?

  • Mutation is widely considered the strongest test criterion; but also the most expensive!

  • This because the TRs of ‘other ‘criteria can be encoded by a mutation (assuming you are completely free to define your mutops), at least with respect to weak killing (other criteria impose local requirements, like weak mutation)

    • Node coverage, edge coverage, clause coverage

  • But.. e.g. correlated active clause coverage  cannot be expressed in terms of mutation killing, as the TR is expressed in terms of pairs of tests(unless we also define a concept of killing a pair of mutants)

  • Login