1 / 39

Cost-aware Pareto Optimal Test Suite Minimisation for Service-centric Systems

Cost-aware Pareto Optimal Test Suite Minimisation for Service-centric Systems. Mustafa Bozkurt CREST, University College London. Services Market. Gartner estimated Software as a Service (SaaS) and cloud-based business application services market as

kaoru
Download Presentation

Cost-aware Pareto Optimal Test Suite Minimisation for Service-centric Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Cost-aware Pareto Optimal Test Suite Minimisation forService-centric Systems Mustafa Bozkurt CREST, University College London

  2. Services Market Gartner estimated Software as a Service (SaaS) and cloud-based business application services market as $13.4 billion in 2011 and projected that the market will grow to $32.2 billion in 2016. M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  3. Mobile Application Market • Google Play has 800K+ apps as of February 2013. • Android users are downloading 2.5 billion apps a month  (Google announced). • Apple App Store has 900K+ apps as of June 2013. • iOS users 2 billion downloads a month (estimated).* * http://venturebeat.com/2013/06/03/three-months-from-today-google-play-will-have-more-total-app-downloads-than-apple/ M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  4. What do these numbers mean? Many of these apps rely on web services and web APIs. M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  5. Why is this important to SBSE? ScS and SaaS usually has faster release cycles compared to traditional software.* *http://blogs.gartner.com/guy-creese/2010/05/18/saas-vs-software-the-development-cycle-for-saas-is-usually-not-always-faster/ M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  6. Why is this important to SBSE? Testing frequency might be increased. M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  7. Why is this important to SBSE? Optimisation!!! M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  8. What are the concerns in testing? • Coverage • Cost (often time it takes to execute a test case) • Fault detection capability • Resource usage • Etc. M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  9. What are the concerns in ScST? • The monetary cost of invoking services during runtime testing. • Service disruptions that might be caused by extensive testing. • Effects of testing in some systems, such as stock-exchange systems, where each usage of the service requires a business transaction, that carries with it a monetary cost. M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  10. What are the concerns in ScST? Test case reliability. (Small side track for 8 slides) M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  11. Test data generation for ScS M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  12. Test data generation for ScS M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  13. Test data generation for ScS Service-centric test data generation.* * M. Bozkurt and M. Harman, “Automatically generating realistic test input from web services” in SOSE '11. M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  14. Service-centric test data generation M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  15. Service-centric test data generation ISBN generation ZIP Code generation M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  16. Reliability formulation Available: M. Bozkurt and M. Harman, “Optimisedrealistic test input generation using web services” in SSBSE 2012 M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  17. Why is data reliability is important? Human-oracle cost reduction due to the testers’ trust in the test input or its source. M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  18. Why is data reliability is important? Avoid extra testing cost caused by erroneous test inputs. M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  19. Why is data reliability is important? 12171 ZIP codes of 23 different U.S. states M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  20. Why is data reliability is important? M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  21. Proposed approach T1 T2 M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  22. Objective Functions Coverage: M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  23. Objective Functions Cost: Contract : Per-use : M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  24. Objective Functions Reliability: M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  25. Multi-objective Algorithm Hybrid NSGA-II (HNSGA-II) by Yoo and Harman† combines effectiveness of greedy algorithm and NSGA-II. † Yoo, S., Harman, M. “Using hybrid algorithm using hybrid algorithm for pareto efficient multi-objective test suite minimisation”, Journal of Systems Software, 2010 M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  26. Multi-objective Algorithm M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  27. Representation and Parameters • ECJ framework built-in NSGAII. • Pop. size 2000and number of generations 100. • 5% mutation probability for each gene. • Single-point crossover with 90% crossover probability. M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  28. Case Studies Complex (CS1)is an artificial test object with complex branching conditions and loops.* * J. Wegener, A. Baresel, and H. Sthamer, “Evolutionary test environment for automatic structural testing" Information and Software Technology, vol. 43, no. 14, pp. 841-854, 2001. M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  29. Case Studies Shipping Service (CS2)is a synthetic workflow generated for existing real-world services. M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  30. Case Studies Shipping Service (CS2) M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  31. Case Studies Test data generation sources M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  32. Case Studies 12%of the test cases for CS1 34% of the test cases for CS2 are equivalent M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  33. Results 12% of the test cases for CS1 34% of the test cases for CS2 are equivalent M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  34. Results M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  35. Results CS1 Contract based CS2 Contract based M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  36. Results CS1 Per-use CS2 Per-use M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  37. Results CS1 Contract Projected 100% coverage M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  38. Results CS2 Contract Projected 100% coverage M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

  39. Conclusion • First approach to introduce multi-objective formulation of test suite minimisation in ScST. • Reduce runtime cost of testing. (one of the major challenges in ScST) • Enable tester to make trade-offs between different aspects of a test suite such as coverage, cost and reliability. M. Bozkurt, Cost-aware Pareto Optimal Test Suite Minimisationfor Service-centric Systems GECCO 2013, July 6-10, Amsterdam, Holland

More Related