03.03.2014, Copenhagen - PowerPoint PPT Presentation

myron
slide1 n.
Skip this Video
Loading SlideShow in 5 Seconds..
03.03.2014, Copenhagen PowerPoint Presentation
Download Presentation
03.03.2014, Copenhagen

play fullscreen
1 / 28
Download Presentation
03.03.2014, Copenhagen
71 Views
Download Presentation

03.03.2014, Copenhagen

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Multi-Objective Test Suite OptimizationforIncrementalProduct Family Testing(jointworkwith Hauke Baller, Sascha Lity and Ina Schaefer) 03.03.2014, Copenhagen

  2. Contents • „Whatis a goodorderingfortestingthemembersof a productfamily?“ • Extension oftheclassicalTest Suite Minimization Problem • Weights • Bounds • Variability • Approx. optimal solution(s) for Multi-Objectives via ILP Solving • Incrementalheuristicsolutionbased on dynamicprogramming toappear at ICST‘2014

  3. Test Suite Minimization Problem [Harrold et al., 1993] r3 Test Requirements r1 r2 r4 r5 r6 t1 t2 t3 t4 … Test Cases Complete Test Suite

  4. Example: Model-Based Testing • All-TransitionsCoverageR = {r1,r2, …, r8} • Complete Test Suite TS(4 Test Cases) • Minimal Complete Test Suite TS‘(1 Test Case) • Test Suites • TS = {(r1r2),(r1r4),(r3r7r8),(r3r5r6)} • TS‘= {(r1r2r3r5r4r7r8r5r6)} r1 r6 test model r2 r4 r5 r3 r7 r8

  5. Minimum Set Cover Problem • Assumptions: • All Test Cases are relevant: • Each Requirement is satisfiable: • Given: • Set of Requirements • Set of Test Cases • Sub sets satisfying Requirement • Find: • Minimum sub set such that • NP-Complete

  6. Cost/Profit Test Suite Optimization[Malishevsky et al. 2006, Xu et al. 2012, Rout et al. 2013] + + + + + = 2 1 4 3 1 2 Testing Profits pTS ≥ Profit Goal kp r3 Test Requirements r1 r2 r4 r5 r6 t1 t2 t3 t4 … Test Cases + + + = 4 2 1 4 TestingCostscTS ≤CostBoundkc Minimal Test Suite ∑=8 Adequate Test Suite ∑=7

  7. Example: Model-Based Testing withCosts/Profits Test Suite CostscTS • TS = {(r1r2),(r1r4),(r3r7r8),(r3r5r6)} • cTS = 2+2+3+3 = 10 • TS‘= {(r1r2r3r5r4r7r8r5r6)} • cTS = 9 • TS‘‘ = {(r1r4r5r2),(r3r7)} • cTS = 5+2 = 7 • Test Suite Profits pTS • TS = {(r1r2),(r1r4),(r3r7r8),(r3r5r6)} • pTS=(2+1)+(2+1)+(2+2+4)+(2+2+1)=15 • TS‘= {(r1r2r3r5r4r7r8r5r6)} • pTS=(2+1+2+2+1+2+4+2+1)=15 • TS‘‘ = {(r1r4r5r2),(r3r7)} • pTS=(2+1+2+1)+(2+2)=10 TS‘‘isAdequateTest Suite forkp= 10 r1(2) kp=10 r6(1) r2(1) r4(1) r5(2) Profits arecountedonce r3(2) r7(2) r8(4)

  8. Minimum Partial Weighted Set Cover Problem [Könemann et al., 2011] • Given: • Set of Weighted Requirements R • Set of Weighted Test Cases T • Profit goal kp • Find: • Sub set with minimal such that • pTS ≥ kp • where and

  9. Product Line Analysis

  10. Sample-basedProduct Line Analysis [Cohen et al. 2006], [Gustafsson 2007], [Oster et al., 2010], [Perrouin et al, 2010], [Kim et al. 2011], [Johansen et al., 2012], [Henard et al., 2013], [Haslinger et al., 2013], ….

  11. Family-basedProduct Line Analysis [Krishnamurti et al., 2006], [Gruler et al., 2010], [Classen et al. 2010], [Gnesi et al. 2011], [Cordy et al., 2013], …

  12. IncrementalProduct Line Analysis [Batoryet al. 2007], [Engström et al. 2011], [Lochau et al. 2012], …

  13. Family-basedProduct Line Test Suite Design • 4 ProductVariants • T={t1, …, t15} • t7=(r1r6r8r7) • t8=(r1r4r5r2) • t11=(r1r4r7) • t12=(r9r10r8) • t13=(r3r11) • t14=(r1r4r11) • t15=(r3r7)} r9(2) r10(2) r6(1) r1(2) r2(1) r4(1) r5(2) r3(2) r7(2) r8(4) r11(8)

  14. Complete Minimal Product Line Test Suite [Cichos et al. 2011] Assumptions • ifthen satisfies on • Given: • Set of Requirements R • Set of Test Cases T • Set of Product Variants P • Sub sets satisfying Requirement • Sub sets executable on product variant • Sub sets relevant for product variant • Find: • Minimum sub set such that

  15. Constraint Minimum Set Cover Problem 2 1 4 3 1 2 Testing Profits pTS≥kp rpTS≥krp r3 r1 r2 r4 r5 r6 Test Requirements t1 t2 t3 t4 … Test Cases 4 2 4 1 Testing Costs p2 p1 p3 ProductVariants ProductCosts/Profits ppTS = + + 5 2 3 ≥ Profit Goal kpp

  16. Constraint (Minimum) Partial Weighted Set Cover Problem • Given: • Set of WeightedRequirements • Set of WeightedTest Cases • Set of WeightedProduct Variants • Profit Goals • (Cost Bound • Find: • Sub set (with minimal such that • andand is an AdequatProduct Line Test Suite

  17. Integer Linear Programming (ILP) • (Integer) Decision variables • Linear objectivefunction • Linear constraints

  18. ILP forProduct Line Test Suite Optimization ppTS≥ kpp ppTS-Δpp=kpp Test Cases must beappliedto a Product t For all Requirements (of a Product) r rpTS≥krp cTS≤kc p t Binary Variables ppTS≥kpp Priotization Normalization Value p t Productselected Requirements must besatisfied pj select Test Case from Test Case Set NOTpjOROne Test Case from Test Case Set ppTS Requirementsatisfied For all Products, a Test Case must beselected Again (ri  Set)  (NOTriOR Set) Differences must not be negativ

  19. Re-FormulationasProductSelection Problem • ILP findsAdequate Test Suite (ifexists), but Problem isNP-hard • Another Point-of-View: IncrementalProductSelection r1 r2 r3 r4 r5 r6 rpTS t1 t2 t3 t4 cTS p1 p2 p3 ppTS

  20. AdequateProduct Sub Set • Given: • Set of WeightedRequirements • Set of WeightedTest Cases • Set of WeightedProduct Variants • Profit Goals • (Cost Bound • Find: • Sub set P such that an adequate test suite exists with

  21. IncrementalProductSelection p1 p1 + p3 + p5 p3 p2 p2 p4 p4 selectnextproduct profit goalsreached? costboundsexceeded? p7 p8 p5 p6 p9 noadequatesolutionfound isadequate isadequate

  22. ProductSelectionHeuristic • Are Profit Goals reached? (rpTS ≥ krp and ppTS ≥ kpp) • No: • For each unselected Product • While unsatisfied Requirements exist, select one: • With high profit and uniqueness/rareness • Select a satisfying Test Case : • With the ability to satisfy many other open Requirements for many other Products and low cost • Rank Products and Select • By: Product profit, resulting Test Case costs, and Requirement profit (weighted) • Recalculate ppTS,rpTSand cTS • Is Test budget excelled (cTS > kc) • Yes: “Error”  End • Goto 1. • Yes: “Success”  End cf. [Harrold et al., 1993] [Harrold et al., 1993] + Products ? ? ? n t n r ? ?

  23. Experimental Results • Varying: • Number of: Requirements, Test Cases, and Products • Mapping: Requirements/Test Cases and Test Cases/Products • Weights • In Total: 51 generated Data Sets • Time Limit: 24h 50, 100, and 500 Products 50, 100, and 500 Requirements 50, 100, and 500 Test Cases r1 r2 r3 r4 r5 r6 -/o/+ Heuristic is 21.5 timesfaster MeanandVariance MeanandVariance t1 t2 t3 t4 Efficencyimprovementfactor: 127.48 68.6% less accuracy p1 p2 p3 -/o/+

  24. Discussion & Future Work Summary • Heuristicproducesusefulresultsandscales • Feasibletradeoffbetweenaccuracyandefficiency Ongoingand Future Work • Backtracking ifboundsareexceeded • Search-basedoptimizationtechniques • Methodsandcriteriato find appropriateweightsandbounds • Beyond ILP: • Dynamic weightfunction, • On-the-flygenerationoftestartifacts, … • Applicationto Real-World Case Studies

  25. Industrial Case Study: Heidelberg Ion-Beam Therapy Center • DCU Case Study forBeam Cycle Creation • Complex Automation Engineering Software • Highly-Configurableand Runtime Adaptive • Re-Engineered as (D)SPL • ~ 200 Features (Boolean andNon-Boolean) • 150% Test Model ~ 640 Transitions • SPL Implementation in C~ 175.000 LOC DCU-R DCU-Z DCU-R DCU-P Gantry TreatmentRooms

  26. Model-Driven Development of DCU Code States Automata Stubs .c generate .h SFT Transition Trigger deploy insert Parameters + UserData Hand-written ECA Code + .h .c .h

  27. DCU SPL Test Model Specification + 150% SFT

  28. ThankYourForYour Attention! Multi-Mechanismen Adaption für das künftige Internet Integrated Model-basedTesting ofContinuouslyEvolving Software Product Lines