testing product lines of industrial size advancements in combinatorial interaction testing n.
Download
Skip this Video
Download Presentation
Testing Product Lines of Industrial Size: Advancements in Combinatorial Interaction Testing

Loading in 2 Seconds...

play fullscreen
1 / 50

Testing Product Lines of Industrial Size: Advancements in Combinatorial Interaction Testing - PowerPoint PPT Presentation


  • 135 Views
  • Uploaded on

Testing Product Lines of Industrial Size: Advancements in Combinatorial Interaction Testing. Martin Fagereng Johansen PhD Thesis Defense, 2013-11-05. Industrial Motivation. TOMRA's Reverse Vending Machines Finale's Financial Reporting Systems ABB's Configurable Safety Module

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Testing Product Lines of Industrial Size: Advancements in Combinatorial Interaction Testing' - adolfo


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
testing product lines of industrial size advancements in combinatorial interaction testing

Testing Product Lines of Industrial Size:Advancements in Combinatorial Interaction Testing

Martin Fagereng Johansen

PhD Thesis Defense, 2013-11-05

slide3

TOMRA's Reverse Vending Machines

  • Finale's Financial Reporting Systems
  • ABB's Configurable Safety Module
  • Eclipse IDEs – Free and Open Source
about the eclipse ide
About the Eclipse IDE
  • Initiated and funded by IBM
  • Widely used by software engineers to develop software
    • Major competitor to Microsoft Visual Studio
  • Many third-party extensions
eclipse ide v3 7 0 indigo an example of a software product line
Eclipse IDE – v3.7.0 (Indigo) – An Example of a Software Product Line

The problem: Can we gain confidence that any product will work?

which products are possible
Which products are possible?

→ model its features and their relationships in a→ feature model:

356,352 possibilities!

today a test suite for each feature
Today: A Test Suite for Each Feature

http://wiki.eclipse.org/Eclipse/Testing

http://archive.eclipse.org/eclipse/downloads/drops/R-3.7-201106131736/testResults.php

faulty f eatures
Faulty Features
  • Unit tests may find faults inside a single feature.
    • n test suites required for a product line with n features.
  • What about faulty cooperation between features?
    • What if they interact incorrectly?
interaction faults
Interaction Faults
  • 2-wise interaction fault
    • reproducible by including 2 specific features
    • the others do not matter
interaction faults1
Interaction Faults
  • 3-wise interaction fault
    • reproducible by including 3 specific features
    • the others do not matter
slide15

Empirics Show:

  • Kuhn et al. 2004:
    • Almost all bugs can be attributed to the interaction of a few features.
covering arrays
Covering Arrays
  • Only a few products are needed to cover all simple interactions.
    • i.e. testing a few well-selected products might reveal almost all bugs
  • Examples (2-wise testing):
    • For the "e-shop product line" with 287 features: 21 products
    • For the Linux kernel with almost 7,000 features: 480 products
configuring feature models
Configuring Feature Models
  • Feature models can be solved by configuration:
  • …or by satisfying the corresponding Boolean formula:
    • R ∧ (A ⇒ R) ∧ (B ⇒ R) ∧ (C ⇒ A) ∧ (D ⇒ A)∧ (C ∨ D) ∧ ¬(C ∧D) ∧ (E ⇒ B) ∧ (F ⇒ B) ∧ (E ∨ F) ∧ (D ⇒ E)
    • e.g. R = 1, A = 1, B = 1, C = 0, D = 1, E = 1, F = 0
    • The SAT problem.
state of the art argument
State of the art argument
  • SAT is the classic NP-complete problem.
    • worst-case analysis (Cook 1971)
  • Configuring basic feature models
    • i.e., finding a single product of a product line
    • SPLE-SAT – Software Product Line Engineering Boolean SATisfiability
      • Includes only feature models that occur in SPLE.
  • Argument
    • SPLE-SAT = SAT, and SAT is NP-complete
    • i.e., SPLE-SAT is NP-complete
    • i.e., SPLE-SAT is impractical(unless P=NP, due to Cobham's thesis)
    • i.e. because sampling involves SPLE-SAT, sampling is impractical.
our argument
Our Argument
  • If SPLE-SAT is impractical:
    • Configuring a feature model is impractical.
    • i.e., testing product lines is of no concern.
      • If we cannot find any products, why care about their quality?
  • However, if we have a product line with products:
    • Finding them were practical.
    • We care about their quality.
    • i.e., SPLE-SAT is practical.
  • Also:
    • If a feature model is too hard to configure then it cannot serve its purpose as an SPLE artifact.
      • A customer cannot use it to customize a product to their needs.
    • i.e., SPLE-SAT is practical.
empirical investigation sat time
Empirical Investigation: SAT time
  • SPLE-SAT is very quick.
  • Even for the largest models.
  • E.g. The Linux Kernel
    • Routinely configured by hand.
conclusions as venn diagrams
Conclusions as Venn Diagrams
  • State of the Art Conclusion:
  • Our Conclusion:

SAT = SPLE-SAT

Hard SAT

what makes icpl quick
What makes ICPL quick?
  • Based on a greedy polynomial time approximation algorithm (PTAS) for the set covering problem (SCP)
    • Chvátal's algorithm (Chvátal 1979)
  • We know SPLE-SAT is quick.
    • Strategically run SPLE-SAT often and infer as much as possible.
  • Utilize modern hardware.
    • large amounts of memory (128 GB)
    • truly parallel processing (64 concurrent executions)
      • Separate out data-parallel sub-algorithms.
  • ++
comparison
Comparison
  • State of the art:
  • Our new algorithm (ICPL):
industrial context
Industrial Context
  • TOMRA's Reverse Vending Machines:
feature model of tomra rvm
Feature Model of TOMRA RVM

435,808 possibilities!

full sampling was too costly
Full Sampling was Too Costly
  • The problem
    • Too many test-products
  • Their Need:
    • Optimize the selection of 12 products.
  • Our answer:
    • Model the market situation.
    • Select the most relevant products according to that model.
better coverage with 12 products
Better Coverage with 12 Products

coverage

All Inter-actionsInteractions of market

t

slide39

?

?

interactions
Interactions

With a 3-wise covering array, we get a few products with:

With a 2-wise covering array, we get a few products with:

TestCSV succeeds for both.

Does CSV work without GEF?

CSV works with and without Web Tools.

Does CSV work with CDT?

Etc…

slide41

What Eclipse Tests Today:

2-Wise Covering Array:

possible causes
Possible Causes
  • Two (or more) features …
    • access the same resource
    • have overlapping GUI elements
      • SWTBot tests
    • have dependencies that interact wrongly
    • wait for each other (deadlock)
    • +++
potential faults found using existing test cases
Potential Faults Found using Existing Test Cases
  • Strategic application of existing tests revealed potential faults.
    • Relatively inexpensive to apply.
  • Raises confidence on success.
  • Such a large scale, fully reproducible and documented application of a product line testing technique is not found in the existing literature.
slide46

?

?

slide47

SPLCATool

SPLCATool

SPLCATool

SPLCATool

future work
Future Work
  • Further empirical study of faults in software product lines.
  • Complete application to the Eclipse IDE
    • With test cases for all features; it is possible today!
    • A good source of further empirics.
    • A good basis of further improvements.
  • Even quicker algorithms for covering array generation.
    • Less memory usage.
    • Higher degree of parallelism.
  • Improved test allocation.
    • Based on specification, model or implementation.
    • Based on meta-data such as versions.
summary
Summary
  • SPLE-SAT was investigated.
    • Realistic feature models are readily configurable.
    • Encourages the investigation into faster algorithms.
  • A fast algorithm for sampling.
    • Enables the use of sampling for product line testing.
  • Theory and algorithms for market-focused sampling.
  • One approach for automatic allocation of test cases.
    • Enables the production of a test report from (1) an implementation, (2) a test case collection and (3) feature model.
  • An automatic and scalable technique for software product line testing supported by free, open source tooling.
    • SPLCATool