1 / 36

Lecture 2 May 27, 2014 Software Testing Research Dr. Sergiy Vilkomir

2014 REU Program at ECU Software Testing - Foundations, Tools, and Applications. Lecture 2 May 27, 2014 Software Testing Research Dr. Sergiy Vilkomir. Testing can show that defects are present, but cannot prove that there are no defects.

emmett
Download Presentation

Lecture 2 May 27, 2014 Software Testing Research Dr. Sergiy Vilkomir

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 2014 REU Program at ECUSoftware Testing - Foundations, Tools, and Applications • Lecture 2 May 27, 2014 • Software Testing Research • Dr. SergiyVilkomir

  2. Testing can show that defects are present, but cannot prove that there are no defects. A restricted number of test cases should be selected There are different ways (testing methods/strategies/criteria/approaches) to select test cases There are no method that guarantee that software is 100% correct Exhaustive testing is impossible

  3. All Combinations All Combinations: All combinations of levels from all factors must be used. Example: 3 factors: A, B, C 3 levels for each factor: red, blue, green Total number of all combinations: 33 = 27

  4. Pair-Wise Pair-Wise: A value of each level for each factor must be combined with a value from every levels for each other factor. In other words, we mustcover all possible pairs of values. Pair-wise testing provides a small set of test cases. It is a practical alternative to testing all combinations. Why? One test case covers several pairs

  5. Pair-Wise Pair-Wise: A value of each level for each factor must be combined with a value from every levels for each other factor. Exercise: Choose one pair and find a test case which covers this pair

  6. Pairwise combinatorial testing • Pairwise combination (instead of exhaustive) • Generate combinations that efficiently cover all pairs • Rationale: most failures are triggered by single values or combinations of a few values. Covering pairs reduces the number of test cases, but reveals most faults

  7. Example: Display Control The total number of combinations – 432 (3x4x3x4x3) test cases

  8. Pairwise combinations: 17 test cases Exercise: Choose one pair and find a test case which covers this pair

  9. Algorithms There are several algorithms for test case generation according to pairwise approach Several techniques are used: Latin Squares Orthogonal arrays, etc One of the well known algorithms is IPO (in-parameter-order), 1998 It is hard to generate test manually for a large number of factors and/or levels There are many tools for pairwise test case generation

  10. Pairwise tools 1. CATS (Constrained Array Test System) [Sherwood] Bell Labs. 2. OATS (Orthogonal Array Test System) [Phadke] AT&T .3AETG Telecordia Web-based, commercial 4. IPO (PairTest) [Tai/Lei] 5. TConfig [Williams] Java-applet 6. TCG (Test Case Generator) NASA 7. AllPairs Satisfice Perl script, free, GPL 8. Pro-Test SigmaZone GUI, commercial 9. CTS (Combinatorial Test Services) IBM Free for non-commercial use 10. Jenny [Jenkins] Command-line, free, public-domain 11. ReduceArray2 STSC, U.S. Air Force Spreadsheet-based, free 12. TestCover Testcover.com Web-based, commercial 13. DDA [Colburn/Cohen/Turban] 14. Test Vector Generator GUI, free 15. OA1 k sharp technology, etc

  11. Number of test cases

  12. In-class exercise 4 x 3 x 2 = 24 Dest - M: 4 x 3 = 12 Dest – Direct: 4 x 2 = 8 M – Direct: 3 x 2 = 6 Total: 26 Start with factors with max numbers of levels: Dest - M: 4 x 3 = 12 We need min 12 test cases Consider software with input parameters: • Dest: London, Paris, Chicago, NY • M: 1000, 2000, 3000 • Direct: Y, N • Create test cases according to pair-wise approach. • Number of all possible combinations? • Number of pairs? • Minimal number of test cases?

  13. In-class exercise • Dest: London, Paris, Chicago, NY • M: 1000, 2000, 3000 • Direct: Y, N

  14. In-class exercise • Dest: London, Paris, Chicago, NY • M: 1000, 2000, 3000 • Direct: Y, N

  15. In-class exercise • Dest: London, Paris, Chicago, NY • M: 1000, 2000, 3000 • Direct: Y, N

  16. t-Wise t-Wise: A value of each level for each group of t factors must be combined. Sometimes: t-way (3-way, 4-way, etc) A natural extension is to require combinations of tvalues instead of 2 t-wise is expensive and benefits are not clear Example: 4-way 30 parameters 5 values each 3,800 test cases (could be too many)

  17. Combinatorial Methods in Software Testing Presentation by Rick Kuhn, (NIST) at ECU, March 22, 2012 http://core.ecu.edu/STRG/seminars.html National Institute of Standards and Technology (NIST) http://csrc.nist.gov/groups/SNS/acts/index.html

  18. Software Testing Study Software Testing is a part of ECU MSc SE program • SENG 6265 Foundations of Software Testing • SENG 6270 Software Verification and Validation

  19. Software Testing Research New Scientific approach: investigation, analysis, comparison, justification, etc. Publications Long period of time – BS, MS, PhD …. Research - ?

  20. http://core.ecu.edu/STRG/

  21. Project 1 Mobile testing to detect device specific faults​

  22. Device-specific failures • Device-specific failures are very common for mobile software applications • An application works reliably on many smartphones and tablets, but does not work properly (i.e., fails) on some specific devices • Examples: • App normally works under some specific operating system but fails under the latest or older OS • Graphics created for high-resolution screens are not shown properly on the mobile devices with extra-high or low screen resolutions • Factors: OS, screen resolution, screen size, device type (smartphone or tablet), handset manufacturer, RAM, etc.

  23. Device-specific failures How many? There are many different mobile devices Sufficient testing is required on different mobile devices Such testing is expensive and time-consuming

  24. Android Fragmentation Visualized (July 2013) From OpenSignal, http://opensignal.com/reports/fragmentation-2013/ From OpenSignal: We have seen 11,868 distinct devices download our app in the past few months. In our report last year we saw 3,997

  25. Research task Optimal selection of mobile devices for testing • S. Vilkomir and B. Amstutz, "Using Combinatorial Approaches for Testing Mobile Applications", Proceedings of the IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops (ICSTW 2014), March 31 - April 4, 2014, Cleveland, Ohio, USA, pp. 78-83. http://core.ecu.edu/STRG/publications/Vilkomir-IWCT-2014-Proceedings.pdf • Chauncey R. Perry, Testing Mobile Applications to Detect Device Specific Faults, MS project, ECU, May 2014​

  26. Device-specific failures

  27. Device-specific failures

  28. Device-specific failures

  29. Device-specific failures

  30. Project 2 Effective Test Generation using Combinatorial Design and ​MCDC

  31. Terminology • Logical expression (the program points at which the control flow can divide into various paths) = Decision • Atomic predicates (elementary Boolean expressions), which form component parts of decisions = Conditions Example: d = ACBD d – decision A, B, C, D – conditions Conditions depend on program data, for example: A ↔ x<7 B ↔ y=3 Values of decisions and conditions are TRUE ( 1 ) or FALSE ( 0 )

  32. Condition/decision coverage Condition/decision coverage = = decision coverage + condition coverage Condition/Decision (D/CC): every decision in the program has taken all possible outcomes at least once, and every condition in each decision has taken all possible outcomes at least once 32 32

  33. Condition/decision coverage Example: d = (A  B)  (C  D) Do these test cases satisfy C/DC ? Two test cases The same outcomes - NO all possible outcomes - YES

  34. MC/DC • “Independent affect” is the main idea of Modified Condition/Decision Coverage (MC/DC) • Suggested in 1992-1994 for avionic software • DO-178B. Software Considerations in Airborne Systems and Equipment Certification. USA, 1992. • Chilenski, J. and Miller, S. Software Engineering Journal, 1994

  35. Test cases according to MC/DC d = (A  B)  C * * * * * * * * * *

  36. Project 2 • Measure combinatorial coverage of test sets that were developed using non-combinatorial methods including MCDC. • Measure MCDC coverage of CT test sets. • Propose methods for enhancing CT approaches to achieve MCDC and other source code coverage criteria. • Empirically investigate effectiveness of the proposed methods when substantial information of code is available. To the greatest extent possible, real-world software should be used, i.e., code should not be small seeded-error programs. • Decrease cost of achieving coverage criteria as compared with other approaches. • Develop a tool for test case generation according to the proposed methods

More Related