Black box test reduction using input output analysis
This presentation is the property of its rightful owner.
Sponsored Links
1 / 19

Black-Box Test Reduction Using Input-Output Analysis PowerPoint PPT Presentation


  • 106 Views
  • Uploaded on
  • Presentation posted in: General

Black-Box Test Reduction Using Input-Output Analysis. ISSTA ‘00 Patrick J. Schroeder, Bogdan Korel Department of Computer Science Illinois Institute of Technology Chicago, IL USA. Presentation Overview. 1) Motivation 2) Black-Box Testing 3) Input-Output Analysis

Download Presentation

Black-Box Test Reduction Using Input-Output Analysis

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Black box test reduction using input output analysis

Black-Box Test Reduction Using Input-Output Analysis

ISSTA ‘00

Patrick J. Schroeder, Bogdan Korel

Department of Computer Science

Illinois Institute of Technology

Chicago, IL USA

ISSTA '00


Presentation overview

Presentation Overview

1) Motivation

2) Black-Box Testing

3) Input-Output Analysis

4) Determining Input-Output Relationships

5) Conclusion

ISSTA '00


Motivation for research

Motivation for Research

  • Experience in Industry as a black-box tester of large, complex, telecommunications systems

  • Root cause analysis of customer-found faults often discovered ineffective or redundant black-box tests

  • Additional information from the software implementation often improves black-box testing (after the initial release!)

ISSTA '00


Black box testing

Black-Box Testing

  • Black-box testing is testing from a functional or behavioral perspective to ensure a program meets its specification

  • Testing usually conducted without knowledge of software implementation - system treated as a “black box”

  • Black-box test design techniques include: equivalence partitioning, boundary value analysis, cause-effect graphing, random testing

ISSTA '00


Observations on black box testing

Observations on Black-Box Testing

  • Testing from the black-box perspective generates a large number of tests (e.g. AT&T Definity PBX > 3 million tests)

  • All testers are faced with resource limitations; the number of tests must be reduced

  • Test reduction done randomly or based on “engineering judgement” may lead to ineffective or redundant tests

ISSTA '00


Combinatorial testing

Combinatorial Testing

  • Combinatorial testing: For programs with multiple inputs, one must decide how to test different combinations of selected test data values

  • The most thorough approach is to test all combinations - this is quite often impossible due to the large number of tests

  • However, not to consider testing different input combinations could lead to an unacceptable number of undetected software faults

ISSTA '00


Combinatorial explosion

X1 X2 X3 . . . Xn

Program P

Combinatorial Explosion

If n = 10 and 10 test data values are selected for each variable, the total number of combinatorial test cases is 1010

ISSTA '00


Current approaches

Current Approaches

  • Testers may try to identify “important” combinations to test using specifications, historical fault data, and engineering judgement

  • Orthogonal arrays, or other combinatorial design techniques, may be used to tests pair-wise or n-way combinations of inputs

  • Difficult to determine impact on fault-detection capability of the test set using these techniques

ISSTA '00


Definition of the problem

Definition of the Problem

  • In combinatorial testing, black-box testers face the situation where the number of possible test cases far exceeds the time and resources available to execute them

  • How can we reduce the number of combinatorial tests while maintaining the fault-detection capability of combinatorial testing?

ISSTA '00


Proposed technique

Proposed Technique

  • Observations:

    • Current techniques do not consider program output variables

    • For a class of programs, not all program inputs affect all program outputs

  • We propose the use of Input-Output (IO) Analysis to determine which program inputs affect program outputs

  • This information can be used to reduce the number of combinatorial tests while “maintaining” the fault-detection capability of the test set

ISSTA '00


Simple example using io analysis

Simple Example using IO Analysis

Input data selected using equivalence partitioning:

A={1, 3} B={N,S,E,W} C={TDC, BDM}

W Z

Total Number of Combinatorial Tests: 2 * 4 * 2 = 16

ISSTA '00


Io analysis applied

IO Analysis Applied

A={1, 3} B={N,S,E,W} C={TDC, BDM}

W Z

ISSTA '00


Io analysis applied1

IO Analysis Applied

A={1, 3} B={N,S,E,W} C={TDC, BDM}

T(W)  T(Z)

(1, N, TDC)

(1, N, BDM)

(3, N, TDC)

(3, N, BDM)

(1, S, TDC)

(1, E, TDC)

(1, W, TDC)

W Z

T(W)

(1, N, TDC)

(1, N, BDM)

(3, N, TDC)

(3, N, BDM)

T(Z)

(1, N, TDC)

(1, S, TDC)

(1, E, TDC)

(1, W, TDC)

ISSTA '00


Optimization problem

Optimization Problem

  • Create the smallest possible test set which covers all data combinations in the individual output test sets

  • Test set optimized using “Don’t Care” (DC) conditions

T(W)

(1, DC, TDC)

(1, DC, BDM)

(3, DC, TDC)

(3, DC, BDM)

T(Z)

(DC, N, DC)

(DC, S, DC)

(DC, E, DC)

(DC, W, DC)

Toptimized

(1, N, TDC)

(1, S, BDM)

(3, E, TDC)

(3, W, BDM)

ISSTA '00


Determining input output relationships

Determining Input-Output Relationships

  • Information may be gathered in a variety of ways (e.g. from specifications, code)

  • Manual collection of information difficult and time consuming

  • Automated collection possible through:

    • Static Analysis

    • Dynamic Analysis

    • Execution-oriented Analysis

ISSTA '00


Automated analysis techniques

Automated Analysis Techniques

  • Static techniques analyze a program’s source code - static program dependencies used to determine IO relationships

  • Dynamic techniques analyze a program’s execution trace across a set of test data - dynamic dependencies used to determine IO relationships

  • Execution-oriented techniques manipulate individual inputs across multiple program executions in a test harness to determine IO relationships

ISSTA '00


Technique advantages disadvantages

Technique Advantages/Disadvantages

  • Static techniques reliably determine the absence of relationships between inputs and outputs; however, they may identify relationships that are not actually present

  • Dynamic analysis and execution-oriented analysis reliably determine the presence of input-output relationships; however, they cannot guarantee that all relationships will be detected

  • Dynamic analysis and execution-oriented analysis may be used when static analysis, perhaps in over-estimation, indicates that all program inputs affect all program outputs

ISSTA '00


Experimental study results

Experimental Study Results

1) Warehouse Management System

Combinatorial Tests: 3,125

Using IO Analysis 40

2) DACS Configuration System

Combinatorial Tests22,500

Using IO Analysis 158

3) Liquidity Ratio Spreadsheet

Combinatorial Tests 1.95 Million

Using IO Analysis 625

ISSTA '00


Conclusion

Conclusion

  • For the class of program and type of problem defined, input-output analysis can reduce the number of combinatorial tests while “maintaining” the fault-detection capability of the test set

  • Research continues on identifying program types and testing situations where our approach is applicable

  • Additional experimentation with execution-oriented analysis to provide program dependencies, and other information useful in black-box testing, is underway

ISSTA '00


  • Login