1 / 26

Simplified Modeling of Combinatorial Test Spaces

Simplified Modeling of Combinatorial Test Spaces. Itai Segall , Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs. Agenda. Introduction Counters Properties Implementation Summary. Background – Restrictions. Restrictions are a crucial part of real-life combinatorial models

Download Presentation

Simplified Modeling of Combinatorial Test Spaces

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Simplified Modeling of Combinatorial Test Spaces Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs

  2. Agenda • Introduction • Counters • Properties • Implementation • Summary

  3. Background – Restrictions • Restrictions are a crucial part of real-life combinatorial models • Effort and expertise required for capturing the restrictions is high • Existing work concentrates on the ability to state complex restrictions • Typically as propositional formulas or predicates • Can be captured using classification trees

  4. The Problem Restrictions are typically: • Cumbersome to read and review • Not very generalized (exclude specific conditions) • Hard to maintain

  5. Introduction • An approach for handling some complex restrictions • We introduce two new constructs to the CTD model: • Counter parameters • Value properties • Real-life case studies

  6. Counters

  7. Counters – Example • Consider testing an upgrade process for a virtualized server • Traverses the host and VMs • Upgrades various out-of-date components • Ten VMs, each has one of four OS versions: RedHat 5.7, RedHat 6.1, AIX 6.1.3, AIX 6.1.6 • 2 versions are marked “preferred” • Triggered when at least half are out of date

  8. Counters – Example – Cont. • Parameters and values: • VM1 – RedHat5.7 / RedHat6.1 / AIX6.1.3 / AIX6.1.6 … • VM10 – RedHat5.7 / RedHat6.1 / AIX6.1.3 / AIX6.1.6

  9. Counters – Example – Cont. “Triggered when at least half are out of date”… ?!

  10. Counters – Example – Cont. • (Bad) option I: explicitly exclude all combinations in which less than half are out of date • All 653,312 of them !

  11. Counters – Example – Cont. • (Bad) option II: choose 5 VMs, and force them to be outdated • VM1 != “AIX6.1.6” && VM1 != “RedHat6.1” && VM2 != “AIX6.1.6” && VM2 != “RedHat6.1” && …VM5 != “AIX6.1.6” && VM5 != “RedHat6.1” • 252 such (terrible) restrictions

  12. Counters • A counter is a parameter that counts values in other parameters • Automatically evaluated in each test • Can be used in restrictions and in coverage requirements

  13. Counters – Back to the example • Define counter “preferred” • Counts appearances of “RedHat6.1” and “AIX6.1.6” in parameters VM1 … VM10 • One restriction that excludes: preferred > 4

  14. Auxiliary Variables • The notion can be extended to auxiliary variables • Parameters that do not define the test • but are rather a function of the defining parameters • Examples: • Sum of other numeric parameters • Count the number of duplicate values

  15. (Touching on) Implementation • Naïve implementation: Add the counter to the model, and restrictions that correctly map its values to the counted values • Much better: Represent as functions on other parameters • More details in the paper

  16. Properties

  17. Properties – Example • Consider testing a Storage Area Network (SAN) • A server (one of four types – two x86-based, two PowerPC-based) • An operating system (one of 17 versions – two Windows-based, three AIX-based, one virtualized, others Linux-based) • HBA (one of 11 types, of different manufacturers)

  18. Properties – Example – Cont. • Some OSs do not run on some processors (e.g., AIX on x86, Windows on PowerPC) • Some HBAs do not have drivers for some OSs • Some HBAs cannot connect to some processors • A total of 299 pairs to be excluded • Tedious and error-prone

  19. Properties • Properties may be defined for parameters • A value to the property is assigned for each value of the parameter • Can be used in restrictions and in coverage requirements • Represent different abstractions of the parameters

  20. Properties – Back to the example • Define property “Architecture” for parameter “Server” • The two PowerPC servers will have value PowerPC • The two x86 servers will have value x86 • Define property “OSFamily” for parameter “OS”, with values Windows, AIX, Linux

  21. Properties – Back to the example – cont. • Define properties “X_Compatible”, “P_Compatible”, “AIX_Compatible” and “Win_Compatible” for the HBA parameter • Give property values to the parameter values according to their compatibility with x86, PowerPC, AIX and Windows, respectively

  22. Properties – Back to the example – Cont. • The restrictions now disallow: • Server.Architecture == “PowerPC” && OS.OSFamily == “Windows” • Server.Architecture == “x86” && OS.OSFamily == “AIX” • Server.Architecture == “PowerPC” && ! HBA.P_Compatible • Server.Architecture == “x86” && ! HBA.X_Compatible • OS.OSFamily == “AIX” && ! HBA.AIX_Compatible • OS.OSFamily == “Windows” && ! HBA.Win_Compatible

  23. Simplified Modeling – Benefits • More readable restrictions • Easier to review • More general • Preserves time, reduces risk of errors • More maintainable • e.g., a new Windows-based OS is introduced

  24. Implementation • Our symbolic BDD-based implementation is agnostic to test-space representation • It merely requires one to symbolically represent the set of valid tests • Simple to extend to support counters & properties • More details in the paper

  25. Summary • Correctly capturing restrictions is a significant obstacle in combinatorial testing deployment • We introduced two constructs that significantly simplify restrictions in some cases • Examples based on real-life cases

  26. Thank You For Listening

More Related