Mutation testing ch 5
1 / 24

Mutation Testing (ch 5) - PowerPoint PPT Presentation

  • Uploaded on

Mutation Testing (ch 5). Course Software Testing & Verification 2013/14 Wishnu Prasetya. Negative Test. Recall: to test a program outside its normal behavior range. Idea: mutate inputs.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Mutation Testing (ch 5)' - wardah

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Mutation testing ch 5

Mutation Testing(ch 5)

Course Software Testing & Verification



Negative test
Negative Test

  • Recall: to test a program outside its normal behavior range.

  • Idea: mutate inputs.

  • For inputs described by BNF we can mutate the grammar instead (5.5.2), e.g. replace/delete/ duplicate terminals and non-terminals in a production rule.

  • Or mutate on-the-fly as we build derivation trees (so that we can control how many times a mutated rule is applied).


NLpostcode Street Area

NLpostcode Area Space Street

Area  FirstDigit Digit Digit Digit

Street  Letter Letter

FirstDigit 1 | 2 ...

Digit  0 | 1 | 2 ...

Letter  a | b | c ... | A | B | C ...

Space   | “ “ Space

Area  FirstDigit Digit Digit

Area  FirstDigit Digit DigitDigitDigit

Some coverage concepts for grammar based mutants generation
Some coverage concepts for grammar-based mutants generation

  • (Def 5.46) Mutation operator (e.g. non-terminal deletion); (Def 5.47) mutant

  • (C 5.33) Mutation Operator Coverage (MOC) : for every mutation operator TR contains one mutant produced via this operator.

  • (C5.34) Mutation Production Cov: for every production rule and every mutation operator that can be applied to the rule, TR contains one mutant via that rule and mutation.

Using mutation as coverage 5 2
Using mutation as “coverage” (5.2)

  • We can “mutate” a program to “fake” a mistake; but the mutant must be a valid program.

  • (D 5.45) Ground : the original.

  • (D 5.48) Let P be a ground. A test t of P kills a mutant P’ of P if t’s output of both differ.

  • We can use mutants to measure the adequacy of our test-set: it is good if it can kill all those mutants.

  • Obviously, the strength of this scheme depends on your choice of the mutation operators.


Fundamental premise : If a program contains a fault, there will usually be a set of mutants that can only be killed by test cases that also detect that fault.

(mutations generated by the tool Mothra)

We can also do mutation driven testing
We can also do: mutation-driven testing

Fundamental premise : If a program contains a fault, there will usually be a set of mutants that can only be killed by test cases that also detect that fault.

not enough mutants percentage killed

a test-case fails

  • Things to note (see next slide) :

  • reachability of a mutation

  • infection

  • propagation

This is a simpler variation of the scheme in Fig. 5.2. The one above does not presume automated generation of test-cases.

How to kill a mutant
How to kill a mutant?

P(x) { if (x<0) x++ ;

return x }

P(x) { if (x0) x++ ;

return x }

  • (reachability) you need a test-case whose execution passes the mutation.

  • You need a test-case that actually observes the effect of the mutation. Note there is a difference in:

    • (infection) “incorrect” state, right after the mutation.

    • (propagation) the infection propagates to incorrect P’s output. In principle, a test-case can only kill if there is propagation.

test() { r = P(1) ; assert (r==2) }

test() { r = P(0) ; assert (r==0) }

Another example
Another example

  • booleanisEven (int X) {

  • if (X < 0)

  • 3 X = 0 – X ;

  • ∆ 3 X = 0 ;

  • if ((double) (X/2) == ((double) X) / 2.0)

  • return true ;

  • else return false ;

  • }

test1() { assert (isEven(-2)==true) }

Issues with killing
Issues with killing

  • No reachability  usually can easily be fixed.

  • You have propagation, but used oracle is only partial (is not strong enough).  fix the oracle.

  • You have infection, but no propagation  as in the isEven example  find another test-case.

  • What if we weaken the definition of “kill” ? We can at least see what that gives us.

Strong and weak kill
Strong and weak kill

  • Let t be a test on a ground P, and P’ is a mutant. Let l be the location of the mutation in P’.

  • (D 5.49) t strongly kills P’ if its verdict on P’ and P are different. In other words, if it fails on P’.

  • ( D 5.50) t weakly kills P’ if the states right after l when executing t on P’ and P are different.

P(x) { if (x<0) x++ ;

return x }

P(x) { if (x0) x++ ;

return x }

test() { r = P(1) ; assert (r0) }

does not even weakly kill

test() { r = P(0) ; assert (r0) }

does not kill, but still weakly kill

Mutation based coverage criteria
Mutation-based coverage criteria

  • (C5.32) Given a set M of mutants, the TR is simply M: you have to kill every mutant in M.

  • C5.35 if you use “strong killing”, C5.36 for “weak killing”.

  • OA remark that in practice strong vs weak do not make much difference.

  • The strength depends on the choice of mutation operators. See 5.2.2 and 5.3.2.

  • Typical problem in mutation-based testing: it generates lots of mutants, roughly O(#dataobjs*#refs)  make testing very computation intensive!

  • O(#refs) using selective mutation (next slide)

Mutation operators
Mutation operators

  • This depends on the used programming language.

  • We can start with some set of mutation operators, then see if we can minimize it.

  • (D5.51) Given a set O of mutation operators; a subset O1 is effective if test-cases designed specifically to kill mutants produced by O1 also kill mutants from O/ O1.

  • There have been plenty of research in this direction. For mutations over imperative constructs, see 5.2.2. Mutations on OO aspects, see 5.3.2.

Examples of mutops on imperative constructs 5 2 2
Examples of mutops on imperative constructs (5.2.2)

  • ABS (Absolute Value Insertion)Modify each arithmetic (sub)expression by using functions abs(), negAbs(), and failOnZero().

  • AOR (Arithmetic Operator Replacement)Replace each occurrence of arithmetic operators +,-,*,/, % by each other; and in addition, by leftOp, and rightOp.

  • SVR (Scalar Variable Replacement)Replace each variable reference by another (type compatible and in-scope) variable.

  • BSR (Bomb Statement Replacement)Replace each statement by Bomb().

  • More... see book.

Addressing integration issues 5 3 2
Addressing integration issues (5.3.2)

  • Faults in component integration are caused by a mismatch of assumptions; e.g. callee thought:

    • a list was sorted, caller did not do that

    • all fields were initialized, caller only initialized some

    • thought values are in kilometers, caller use miles

  • Integration mutation focuses on mutating the connections between components

    • Sometimes called “interface mutation”

    • Both caller and callee methods are considered

Four types of integration mutations
Four types of integration mutations

  • Change a calling method by modifying the call (e.g. deleting the call; replacing the returned obj by a type-compatible obj).

  • Change a calling method by modifying values that are sent to a called method.

  • Change a called method by modifying values that enter and leave a method

    • Includes parameters as well as variables from higher scopes (class level, package, public, etc.)

How about oo these are typical oo features also where typical mistakes are made
How about OO?These are typical OO features; also where typical mistakes are made

  • Encapsulation (hiding details).

Typical oo aspects
Typical OO aspects

  • Inheritance: method overriding, variables shadowing, constructors overriding works differently

  • Polymorphism

    • a required object of type T can be instantiated by a subclass at the runtime

    • overloading

  • In addition to object level members we also have class level members (the ‘static’).

  • 5.3.3 lists 20 mutops targeting the above aspects.

Examples of inheritance mutops
Examples of inheritance mutops

  • OMD (Overriding Method Deletion)Each entire declaration of an overriding method is deleted.

  • OMM (Overridden Method Moving)Each call to a super.method is moved to the first and last position, and up and down one statement.

override m() { x ++ ; y = y/x ;

base.m() ; // super}

Examples of inheritance mutops1
Examples of inheritance mutops

HVI – Hiding Variable Insertion

HVD – Hiding Variable Deletion


int x;

int y;


int x;

int y;


1 int x;

2 int y;


int x;

1 // int x;

int y;

2 // int y;

Examples of inheritance mutops2
Examples of inheritance mutops



ATC – Actual Type Change

DTC – Declared Type Change

point p;

p = PointFactory();

 p = new colorpoint ();

point p;

colorpoint p;

p = PointFactory();

Examples of overloading mutops


void set (int x, int y, char c);

void set (char a, int x, int y);


void set (int x, int y, int z);

void set (int x, int y);

void set (int z);

Examples of overloading mutops

AOC – Argument Order Change

ANC – Argument Number Change


point3D p ;

p.set (1, 2, ‘t’) ;

 p.set (‘t’, 1, 2) ;

point3D p;

p.set (1, 2, 3);

 p.set (2, 3);

 p.set (3);

Examples of other mutops


int x = 5;

 int x;


point() { … }

 // point() { … }


void set (int x, int y)


this.x = x;

1 x = x;

this.y = y;

2 y = y;


Examples of “other” mutops

VID –Variable Initialization Deletion

DCD – Default Constructor Delete

TKD – This Keyword Deletion

Ok but how useful is this
Ok... but how useful is this !?

  • Mutation is widely considered the strongest test criterion; but also the most expensive!

  • This because the TRs of ‘other ‘criteria can be encoded by a mutation (assuming you are completely free to define your mutops), at least with respect to weak killing (other criteria impose local requirements, like weak mutation)

    • Node coverage, edge coverage, clause coverage

  • But.. e.g. correlated active clause coverage  cannot be expressed in terms of mutation killing, as the TR is expressed in terms of pairs of tests(unless we also define a concept of killing a pair of mutants)