Firewalls for regression testing

1 / 26

# Firewalls for regression testing - PowerPoint PPT Presentation

Firewalls for regression testing. Spring 2013. Use of the word “Firewalls”.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

## PowerPoint Slideshow about 'Firewalls for regression testing' - hyatt-higgins

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

### Firewalls for regression testing

Spring 2013

Use of the word “Firewalls”

The way the authors White, Jaber, Robinson and Rajlich use the term “Firewall” in this work is special for their work on regression testing and must not be mixed with the common use of the term “firewall” which is part of networks and security.

Why firewalls in regression testing

Most regression tests are large and time-consuming to run. The concept of a “firewall” is used to reduce the set of classes – or components – that need to be tested.

A firewall in regression testing separates the classes that depend on the class that is changed from the rest of the classes.

There are two central concepts – dependency and encoding

Dependency

The main concept to understand when we apply the firewall idea is dependency.

The main question is:

Is the component dealing with the plain values of the input

or

Must the component account for how those inputs were generated

Definitions – 1

Let C be a component with

• inputs {Ii}
• outputs {Oj}
• post-conditions {Qk}

If Q  {Qk} and I  {Ii} so that Q depends on I and on a value generated by a component C1 that is connected to I through a data flow through one or more intermediary component C2,…Cm then

• I is an encoded input
• C is an external component in scope of C2 and visa versa
Definitions – 2

No such component C1 => I is a plain input, i.e. not encoded

All inputs of C are plain => C is an I/O component

C is external in scope of D and there is no message sent from C to D => there is a hidden dependency between C and D.

I

I

c

c

c1

c2

c1

cm

I is a plain input

I is an encoded input

Dependency example – 1

Component 1 generates the key k, which is converted to an integer in component 2. Component 3 converts the integer to a hash address A. Component 4 uses the hash address to do a table look-up to generate the value f(k).

If component 2 or 3 is changed, the result r may change.

Dependency example – 2

The post condition of component 4 depends on k, generated by component 1 => input A is encoded by component 2.

Component 4 is external in scope of component 2 and visa versa.

r = f(k) but k is changed by component 2

There is a hidden dependency between 2 and 4

Example of hidden dependency

A’s post-condition is dependent on the value “colour” that is

• Generated in class I
• Encoded as an integer in class A (0 is red, 1 is yellow etc.)
• Decoded in class B (receives colour code, decodes and paints the screen)

Thus class B is external in scope of A. The dependency between A and B is hidden since no message is sent between A and B

Global variables

Activation path:

1, 2, 3, 4

Messages path:

1, 2, 3, g, 4

The output v3 from component 3 is used to modify

the global variable g which is used as input to component 4.

Component 4 is external and dependent on variable v1 and is activated by a message from component 3 to component 4 which has no data associated with it

Simple firewall example: a TFW

TFW does not examine dependencies further away than one level from changed

component

Process for determining TFW in OO systems
• Given two successive versions of an oo-system, find the difference of the two versions and identify those classes that have changed
• If any of the changed classes are in an inheritance hierarchy, also consider descendants of the changed class as changed
• For each changed class, identify all the classes that send messages to the changed class or receive messages from the changed class and include them in the TFW
• See previous slide for a graph
Extended Testing Firewall - ETW
• EFW extends the TFW by taking into account the data flows, external components and hidden dependencies (see previous slides)
• EFW is obtained by following the process:
• First apply steps 1-3 in order to obtain TFW
• Identify all data paths from and to a modified class
• Find all external classes in scope of the modified class (E in the example on next slide). Include all such classes in EFW
Empirical Study 1: TFW vs. EFW
• In order to compare the concepts of Simple (TFW) and Extended Firewall (EFW), the authors tested several builds of a TelComm system.
• 66 classes, 11000 lines of code, C++
• This system had already been tested by the developers using their standard test suite.
• Common – found using both TFW and EFW: 8 errors
• Found only using EFW: 4 errors. Two of these had not been found by the company’s test suite.
TelComm system: Severity of faults
• All faults in previous slide were of severity 3 except for the EFW fault in build 1.1 which is of severity 2
• No faults of severity 1 or 4
TelComm system: Extent of extra effort
• In the table on next slide, total time for testing is the sum of
• Analysis time – identifying the firewall
• Testing time is the time needed for
• test setup
• test execution
• analysis of test results.
• EFW testing requires approximately
• 20 – 30% more tests
• 40 – 50% more person hours
TelComm system: Extent of extra effort
• TFW: 46 person hours, 8 errors => 5.7 person hours per error found
• EFW: 58 person hours,12 errors => 4.8 person hours per error found
TelComm system: Reused vs. new tests

For TFW some testes were reused from the company’s original test suite

All extra tests needed for EFW were written from scratch.

Empirical Study 2 TFW vs. ETF
• An empirical study was also performed on a very large system
• more than 1 mill lines of code
• thousands of classes
• C++ code
• Real time system
• Windows environment
Real time system: TFW vs. EFW
• 13 common faults and 3 faults found only by EFW
• Version 1: Common faults: 3 at sev. level 4, 1 at level 3, 1 at level 2 (EFW)
• Version 2: Common faults 8 (6 at lev. 4, 2 at lev. 2),
• EFW faults, one at level 3 one at level 2
Real time system: number of tests and effort
• TFW: 93 person hours, 13 errors => 7.2 person hours per error found
• EFW: 120 person hours, 16 errors => 7.5 person hours per error found
Conclusion
• Extended Firewall testing (EFW) finds more faults than Simple Fire Wall (TFW) testing
• 50% in the first study, 23 % in the second study
• The cost in person hours per fault found is approximately the same: 5 - 8 person hours per fault
• EFW and TFW
• should only be used to test updates and changes
• assumes that the initial test suite exists and has a high quality
Reference

Most of the information on these slides are from the paper (available from It’s learning)

“Extended firewall for regression testing: an experience report”

By: Lee White, Khaled Jaber, Brian Robinson and Václac Rajlich

From: Journal of Software Maintenance and Evolution: Research and Practice, 2008, 20:419-433