1 / 24

# Mutation Testing (ch 5) - PowerPoint PPT Presentation

Mutation Testing (ch 5). Course Software Testing & Verification 2013/14 Wishnu Prasetya. Negative Test. Recall: to test a program outside its normal behavior range. Idea: mutate inputs.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

## PowerPoint Slideshow about 'Mutation Testing (ch 5)' - wardah

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

### Mutation Testing(ch 5)

Course Software Testing & Verification

2013/14

WishnuPrasetya

• Recall: to test a program outside its normal behavior range.

• Idea: mutate inputs.

• For inputs described by BNF we can mutate the grammar instead (5.5.2), e.g. replace/delete/ duplicate terminals and non-terminals in a production rule.

• Or mutate on-the-fly as we build derivation trees (so that we can control how many times a mutated rule is applied).

NLpostcode Street Area

NLpostcode Area Space Street

Area  FirstDigit Digit Digit Digit

Street  Letter Letter

FirstDigit 1 | 2 ...

Digit  0 | 1 | 2 ...

Letter  a | b | c ... | A | B | C ...

Space   | “ “ Space

Area  FirstDigit Digit Digit

Area  FirstDigit Digit DigitDigitDigit

• (Def 5.46) Mutation operator (e.g. non-terminal deletion); (Def 5.47) mutant

• (C 5.33) Mutation Operator Coverage (MOC) : for every mutation operator TR contains one mutant produced via this operator.

• (C5.34) Mutation Production Cov: for every production rule and every mutation operator that can be applied to the rule, TR contains one mutant via that rule and mutation.

• We can “mutate” a program to “fake” a mistake; but the mutant must be a valid program.

• (D 5.45) Ground : the original.

• (D 5.48) Let P be a ground. A test t of P kills a mutant P’ of P if t’s output of both differ.

• We can use mutants to measure the adequacy of our test-set: it is good if it can kill all those mutants.

• Obviously, the strength of this scheme depends on your choice of the mutation operators.

Fundamental premise : If a program contains a fault, there will usually be a set of mutants that can only be killed by test cases that also detect that fault.

(mutations generated by the tool Mothra)

We can also do: mutation-driven testing

Fundamental premise : If a program contains a fault, there will usually be a set of mutants that can only be killed by test cases that also detect that fault.

not enough mutants percentage killed

a test-case fails

• Things to note (see next slide) :

• reachability of a mutation

• infection

• propagation

This is a simpler variation of the scheme in Fig. 5.2. The one above does not presume automated generation of test-cases.

P(x) { if (x<0) x++ ;

return x }

P(x) { if (x0) x++ ;

return x }

• (reachability) you need a test-case whose execution passes the mutation.

• You need a test-case that actually observes the effect of the mutation. Note there is a difference in:

• (infection) “incorrect” state, right after the mutation.

• (propagation) the infection propagates to incorrect P’s output. In principle, a test-case can only kill if there is propagation.

test() { r = P(1) ; assert (r==2) }

test() { r = P(0) ; assert (r==0) }

• booleanisEven (int X) {

• if (X < 0)

• 3 X = 0 – X ;

• ∆ 3 X = 0 ;

• if ((double) (X/2) == ((double) X) / 2.0)

• return true ;

• else return false ;

• }

test1() { assert (isEven(-2)==true) }

• No reachability  usually can easily be fixed.

• You have propagation, but used oracle is only partial (is not strong enough).  fix the oracle.

• You have infection, but no propagation  as in the isEven example  find another test-case.

• What if we weaken the definition of “kill” ? We can at least see what that gives us.

• Let t be a test on a ground P, and P’ is a mutant. Let l be the location of the mutation in P’.

• (D 5.49) t strongly kills P’ if its verdict on P’ and P are different. In other words, if it fails on P’.

• ( D 5.50) t weakly kills P’ if the states right after l when executing t on P’ and P are different.

P(x) { if (x<0) x++ ;

return x }

P(x) { if (x0) x++ ;

return x }

test() { r = P(1) ; assert (r0) }

does not even weakly kill

test() { r = P(0) ; assert (r0) }

does not kill, but still weakly kill

• (C5.32) Given a set M of mutants, the TR is simply M: you have to kill every mutant in M.

• C5.35 if you use “strong killing”, C5.36 for “weak killing”.

• OA remark that in practice strong vs weak do not make much difference.

• The strength depends on the choice of mutation operators. See 5.2.2 and 5.3.2.

• Typical problem in mutation-based testing: it generates lots of mutants, roughly O(#dataobjs*#refs)  make testing very computation intensive!

• O(#refs) using selective mutation (next slide)

• This depends on the used programming language.

• We can start with some set of mutation operators, then see if we can minimize it.

• (D5.51) Given a set O of mutation operators; a subset O1 is effective if test-cases designed specifically to kill mutants produced by O1 also kill mutants from O/ O1.

• There have been plenty of research in this direction. For mutations over imperative constructs, see 5.2.2. Mutations on OO aspects, see 5.3.2.

• ABS (Absolute Value Insertion)Modify each arithmetic (sub)expression by using functions abs(), negAbs(), and failOnZero().

• AOR (Arithmetic Operator Replacement)Replace each occurrence of arithmetic operators +,－,*,／, % by each other; and in addition, by leftOp, and rightOp.

• SVR (Scalar Variable Replacement)Replace each variable reference by another (type compatible and in-scope) variable.

• BSR (Bomb Statement Replacement)Replace each statement by Bomb().

• More... see book.

• Faults in component integration are caused by a mismatch of assumptions; e.g. callee thought:

• a list was sorted, caller did not do that

• all fields were initialized, caller only initialized some

• thought values are in kilometers, caller use miles

• Integration mutation focuses on mutating the connections between components

• Sometimes called “interface mutation”

• Both caller and callee methods are considered

• Change a calling method by modifying the call (e.g. deleting the call; replacing the returned obj by a type-compatible obj).

• Change a calling method by modifying values that are sent to a called method.

• Change a called method by modifying values that enter and leave a method

• Includes parameters as well as variables from higher scopes (class level, package, public, etc.)

How about OO?These are typical OO features; also where typical mistakes are made

• Encapsulation (hiding details).

• Inheritance: method overriding, variables shadowing, constructors overriding works differently

• Polymorphism

• a required object of type T can be instantiated by a subclass at the runtime

• In addition to object level members we also have class level members (the ‘static’).

• 5.3.3 lists 20 mutops targeting the above aspects.

• OMD (Overriding Method Deletion)Each entire declaration of an overriding method is deleted.

• OMM (Overridden Method Moving)Each call to a super.method is moved to the first and last position, and up and down one statement.

override m() { x ++ ; y = y/x ;

base.m() ; // super}

HVI – Hiding Variable Insertion

HVD – Hiding Variable Deletion

point

int x;

int y;

point

int x;

int y;

colorpoint

1 int x;

2 int y;

colorpoint

int x;

1 // int x;

int y;

2 // int y;

point

colorpoint

ATC – Actual Type Change

DTC – Declared Type Change

point p;

p = PointFactory();

 p = new colorpoint ();

point p;

colorpoint p;

p = PointFactory();

void set (int x, int y, char c);

void set (char a, int x, int y);

point3D

void set (int x, int y, int z);

void set (int x, int y);

void set (int z);

AOC – Argument Order Change

ANC – Argument Number Change

point

point3D p ;

p.set (1, 2, ‘t’) ;

 p.set (‘t’, 1, 2) ;

point3D p;

p.set (1, 2, 3);

 p.set (2, 3);

 p.set (3);

int x = 5;

 int x;

point

point() { … }

 // point() { … }

point

void set (int x, int y)

{

this.x = x;

1 x = x;

this.y = y;

2 y = y;

}

Examples of “other” mutops

VID –Variable Initialization Deletion

DCD – Default Constructor Delete

TKD – This Keyword Deletion

• Mutation is widely considered the strongest test criterion; but also the most expensive!

• This because the TRs of ‘other ‘criteria can be encoded by a mutation (assuming you are completely free to define your mutops), at least with respect to weak killing (other criteria impose local requirements, like weak mutation)

• Node coverage, edge coverage, clause coverage

• But.. e.g. correlated active clause coverage  cannot be expressed in terms of mutation killing, as the TR is expressed in terms of pairs of tests(unless we also define a concept of killing a pair of mutants)