Integration and system testing
This presentation is the property of its rightful owner.
Sponsored Links
1 / 87

Integration and system Testing PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Integration and system Testing . Csci565 Fall 2012. Objectives. Integration Testing Simple ATM (SATM) discuss integration testing strategies Decomposition Based Integration Testing (DBIT) Top-down Bottom-up Call Graph Based Integration Testing (CGBIT)

Download Presentation

Integration and system Testing

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Integration and system testing

Integration and system Testing


Fall 2012




  • Integration Testing

  • Simple ATM (SATM)

  • discuss integration testing strategies

    • Decomposition Based Integration Testing (DBIT)

      • Top-down

      • Bottom-up

    • Call Graph Based Integration Testing (CGBIT)

    • Path-Based Integration Testing (PBIT)


A software testing strategy spiral model

A Software Testing Strategy: Spiral Model

System Testing

validation Testing

Integration Testing

Unit Testing




System eng.

Integration testing 1

Integration Testing:1

  • If all units/components work individually, do they work as a whole when we put them together?

    • Not necessarily

  • The problem is “putting them together” or interfacing them


Problems with interfacing

Problems with interfacing

  • Integration faults often traceable to incomplete or misunderstood interface specifications

    • mismatched assumptions about other components

    • Individually acceptable imprecision may be magnified to unacceptable levels

    • Global data structures can present problems

  • Inconsistent interpretation of parameters or values

  • Mixed units (meters/yards) in Martian Lander

  • Violations of value domains, capacity, or size limits


Integration testing

Integration testing

  • Tests complete systems or subsystems composed of integrated components

  • Integration testing should be black-box testing when tests derived from the specification

  • Main difficulty is localising errors

  • Incremental integration testing reduces this problem


Approaches to integration testing

Approaches to integration testing

  • Two major approaches

    • Incremental approaches

      • The decomposition-based techniques or tree

        • Stubs/Drivers

      • Call Graph-based techniques

        • No stubs or drivers

    • Non-incremental approaches


Incremental integration testing

Incremental integration testing


Incremental approaches top down

Incremental approaches: Top-down

  • Top-down testing

    • Start with high-level system

    • integrate from the top-down replacing individual components by stubs where appropriate

      • Depth-first

      • Breadth-first

      • No-best order

        • Critical sections

        • Early skeletal version using I/O modules




  • Stubs

    • Special module to simulate some functionality

    • Its production is nontrivial task because the code may simulate a very complicated tasks

      • E.g.

        • Writing a stub performing a database table search routine

    • Creating multiple version of same stub for various reasons


Top down testing

Top-down testing


Top down complications

Top-down: Complications

  • The most common complication occurs when processing at low level hierarchy demands adequate testing of upper level

  • To overcome:

    • Either, delay many tests until stubs are replaced with actual modules (BAD)

    • Or, develop stubs that perform limited functions that simulate the actual module (GOOD)

    • Or, Integrate the software using bottom up approach


Incremental testing bottom up

Incremental testing: Bottom up

  • Bottom-up testing

    • Integrate individual components in levels until the complete system is created


Bottom up approach

Bottom-up approach

  • Starts with construction and testing with atomic modules

    • No need for stub

    • Low level components are combined into cluster (or builds) to perform a specific sub-function

    • A driver (a control program for testing) is written

      • Contain hardcoded test input, calls the module being tested, and display the results

    • Cluster is tested

    • Drivers are removed and clusters are combined moving upward in the program structure


Bottom up testing

Bottom-up testing



Guideline for integration testing

Guideline for Integration testing

  • Integrate the components that implement the most frequently used functionality

  • Perform regression testing for old features

  • Perform progression testing for new features

  • Study the Architecture design of the Application

    • identify the Critical Modules.

  • Obtain the interface designs from the Architectural team and create test cases to verify all of the interfaces in detail.

    • E.g., Interface to database/external hardware/software application must be tested in detail.


Non incremental


  • Big-bang

    • Imposes no order (GOOD)

    • Test all the units (Modules) at once (GOOD)

    • Very easy (GOOD)

    • difficult to localize the source of errors (VERY BAD)


Top down vs bottom up

Top-down Vs. Bottom-up

  • Architectural validation

    • Top-down integration testing is better at discovering errors in the system architecture

  • System demonstration

    • Top-down integration testing allows a limited demonstration at an early stage in the development

  • Test implementation

    • Often easier with bottom-up integration testing


Primary design of the atm using tree based decomposition

Primary Design of the ATM using Tree-based decomposition



Terminal I/O







Card Entry

PIN Entry


Simple atm satm

Simple ATM (SATM)

  • An ATM simple

    • Provides 15 screens for interactions

    • includes 3 function buttons

      • B1, B2, B3

    • Modeled in structural analysis

      • Data Model (ERD)

      • Functional Model (DFD)

      • Behavioral model (STD)


Figure 12 7

Figure 12.7


Figure 12 8

Figure 12.8


Figure 12 9

Figure 12.9


Figure 12 10

Figure 12.10


Entity relationship model of the satm

Entity/Relationship model of the SATM

Figure 12 12

Figure 12.12


Figure 12 13

Figure 12.13


Decomposition based strategies

Decomposition based strategies

  • Decomposition based

    • Top/down

    • Bottom up

    • Sandwich

    • Big bang


Figure 12 14

Figure 12.14


Pdl description of satm

PDL description of SATM


Integration and system testing chapter 12 13


Integration and system testing chapter 12 13


Figure 13 1

Figure 13.1


Decomposition based testing 1

Decomposition based testing:1

  • Discussion revolves around the tree-based decomposition and the order by which units are tested and combined

    • Top-to-bottom

    • Bottom-to-top

    • Sandwich

    • Big bang

  • The focus is on the structural compatibility among interfaces


Test sessions

Test Sessions

  • A test session refers to one set of tests for a specific configuration of actual code and stubs

  • The number of integration test sessions using a decomposition tree can be computed

    • Sessions=nodes – leaves + edges


Decomposition based testing 2

decomposition based testing: 2

  • For SATM system

    • 42 integration testing session (i.e., 42 separate sets of integration test cases)

      • top/down

        • (Nodes-1) stubs are needed

          • 32 stub in SATM

      • bottom/up

        • (Nodes-leaves) of drivers are needed

          • 10 drivers in SATM


Decomposition based strategies pros and con

Decomposition based strategies: Pros and con

  • Intuitively clear and understandable

    • In case of faults, most recently added units are suspected ones

  • Can be tracked against decomposition tree

  • Suggests breadth-first or depth-first traversals

  • Units are merged using the decomposition tree

    • Implies correct behavior should follow from individually correct units and interfaces

    • Stubs/Drives are major development Overhead


Call graph based integration testing

Call graph based integration testing

  • Call graph

    • A directed graph

    • Nodes corresponds to unit

    • Edges corresponds to the call

    • E.g.

      • AB (i.e., A is calling B)

  • Attempts to overcome the decomposition problem (structural)

  • Moves toward behavioral testing


Call graph based cg approaches

Call graph based(CG): approaches

  • Two main approaches based on Call Graph

    • Pair-wise integration

    • Neighborhood integration


Table 2 am

Table 2: AM


Pair wise integration

Pair-wise integration

  • The main idea is to eliminate the overhead (i.e., stub/drive)

  • Uses actual code by restricting a session testing to a pair of units in the Call Graph

    • One integration test for each edge in Call Graph

    • 40 edges means 40 integration tests for the SATM




Uses actual code

-one integration test session for each edge

-40 edges for SATM


Neighborhood integration

Neighborhood Integration

  • The neighborhood of a node refers to the nodes that are one edge away from the given nodes

    • SATM Neighborhoods

    • Number of neighborhoods can be computed:

      • 1) Neighborhoods = Interior node + source node

      • 2) Interior nodes =nodes – (source nodes + sink nodes)

        • ( x=1 if there exists leaf nodes connected directly to the root node otherwise X= 0)

  • Results a drastic reduction in the number of integration test session

    • In case of SATM (11 vs. 40)


Neighborhood integration1

Neighborhood Integration


Table 3 satm neighborhoods

Table 3: SATM Neighborhoods


Pros and cons

Pros and cons

  • Benefits (GOOD)

    • Mostly behavioral than structural

    • Eliminates sub/drive overhead

    • Works well with incremental development method such as Build and composition

  • Liabilities (BAD)

    • The fault isolation

      • E.g.,

        • Fault in one node appearing in several neighborhood


Questions about integration testing

Questions about Integration testing

  • How do perform Integration testing for non-tree based functional decomposition?

    • E.g.,

      • integration testing for OO

      • Integration testing for Client/server systems

      • Integration testing for Layered systems

      • ….


Path based integration testing

Path-based Integration Testing

  • The hybrid approach (i.e., structural and behavioral) is an ideal one integration testing

  • The focus is on the interactions among the units

    • Interfaces are structural

    • Interactions are behavioral

  • With unit testing, some path of source statements is traversed

    • What happens when there is a call to another unit?

      • Ignore the single-entry/single-exit

      • Use exist follows by an entry sequence

      • Suppress the call statement


New and extended concepts

New and extended concepts

  • Source node (begin)

    • A statement fragment at which program execution begins or resumes

    • E.g., BEGIN

  • Sink node (end)

    • A statement fragment at which program execution terminates

    • E.g., END


More on concepts

More on concepts

  • Module Execution Path (MEP)?

    • A sequence of statements that begins with a source node and ends with a sink node, with no intervening sink nodes

  • Message?

    • A programming language mechanism by which one unit transfers control to another unit

      • E.g.,

        • subroutine invocations

        • Procedure calls

        • Function references


The path based integration testing definition

The path based integration testing (definition)

  • Definition: MM-Path

    • An interleaved sequence of module execution paths(MEP) and messages

  • MM-Path can be used

    • to describe sequences of module execution paths including transfers of control among units using messages

    • To represent feasible execution paths that cross unit boundaries

    • To extent program graph

      • Where

        • nodes = MEP

        • edges = messages

  • Atomic system function (ASF)?

    • An action that is observable at the system level in terms of port input and output events (at GUI)


Figure 13 3

Figure 13.3



Integration and system testing chapter 12 13

There are seven module execution paths (MEP):



Mm path graph definition

MM-Path Graph (definition)

  • Definition: MM-Path Graph?

    • Given a set of units, their MM-Path graph is the directed graph in which

      • nodes = MEPs

      • edges = messages/returns


Figure 13 4

Figure 13.4


Satm code

SATM: Code

Satm code 2

SATM: Code 2

Code 3

Code: 3

Mm path for correct for correct pin entry

MM-Path for correct for correct PIN Entry

How long should an mm path be 1

How long should an MM-Path Be:1

  • MM-Path issues

    • How long is an MM-Path?

    • What is the endpoint?

  • The following observable behavior are used as endpoints

    • Event quiescence (event inactivity)

      • System level event

      • Happens when system is ideal/waiting

    • Message quiescence (msg inactivity)

      • Unit that sends no messages is reached (module C)

    • Data quiescence (data inactivity)

      • Happens when a sequences of processing generate a stored data that is not immediately used

      • E.g., account balance that is not used immediately


How long should an mm path be 2

How long should an MM-Path Be:2


Mm path guidelines

MM-Path Guidelines

  • MM-Path guidelines:

    • Points of

    • (data, event, or msg) are natural endpoints for an MM-path

    • Atomic System Functions (system behavior) are considered as an upper limit for MM-Paths

      • MM-Paths should not cross ASF boundaries


Pros and cons1

Pros and cons

  • hybrid approach(GOOD)

  • The approach works equally well for software testing developed by waterfall model (GOOD)

  • Testing closely coupled with actual system behavior (GOOD)

  • Identification of MM-Paths which can be offset by elimination of sub/drivers(BAD)

  • Test case generation ?


Summary of integration testing 1

Summary of integration testing: 1

  • Integration testing

    • Testing of incompatibilities and interfaces between otherwise correctly working components

    • Performed with an incomplete system using scaffolding(drivers/stubs/mocks)

      • Stubs main job is to return values to the calling program

        • Simple stub assigns constant values to the outputs


Summary of integration testing 2

Summary of integration testing:2

  • Drivers main job is to control and set up testing ; they can include hard-code values or retrieve the values from an external source

  • Tools exist to generate both stubs/drivers automatically


Summary of integration testing 3

Summary of integration testing: 3

  • Integration testing focuses on interface

    • 40% of software errors are revealed during integration testing

    • Must be built on foundation of thorough unit testing

    • Integration faults often traceable to incomplete or misunderstood interface specifications

      • Prefer prevention to detection, and make detection easier by imposing design constraints

  • Strategies tied to project build order

    • Order construction, integration, and testing to reduce cost or risk

Table integration testing

Table: integration testing?

Integrated testing life cycle

Integrated Testing Life cycle


Type of integration faults

Type of Integration Faults

  • Inconsistent interpretation of parameters or values

    • Example: Mixed units (meters/yards) in Martian Lander

  • Violations of value domains, capacity, or size limits

    • Example: Buffer overflow

  • Side effects on parameters or resources

    • Example: Conflict on (unspecified) temporary file

  • Omitted or misunderstood functionality

    • Example: Inconsistent interpretation of web hits

  • Nonfunctional properties

    • Example: Unanticipated performance issues

  • Dynamic mismatches

    • Example: Incompatible polymorphic method calls

Integration testing using software architecture sa

Integration testing Using Software Architecture (SA)

  • Software Architectural Based Integration testing

    • How software architecture can be used to conduct tests to uncover errors related to the interface?

  • How can we test if a design and implementation comply using SA?

  • How can we specify a SA such that one or more properties can be tested or verified?

  • How integration testing can be planned and controlled based on the SA?


Software architecture testing

Software architecture & Testing

  • Using Traditional approach for formalizing and automating the integration test stage is difficult

    • the selection of the test cases for the subsystems stressstructureover behavior

    • the order in which the components are incrementally combined is entirely dependent on the structure (tree or graph)


Software architecture sa testing

Software Architecture (SA) & Testing

  • Most current testing approaches are either based

    • the implementation or structural

    • information of the system

    • a requirement specification

    • system design

    • the architecture level

  • Using the properties of formal architecture description languages (ADL), architecture based test criteria can be defined




  • what to test at the software architecture?

    • At the integration and system testing level, the predominant form for defining testing criteria is based on definition/use bindings

      • each module defines or provides a set of facilities that are available to the users or requires by other modules

  • Software architecture

    • focuses on the interaction between components

    • Interaction is language independent


Testing requirements for sa

Testing Requirements for SA

  • Testing component-to-connector connectivity

  • Testing connector-to-component connectivity

  • Testing component –to-internal interfaces

  • Testing connector internal interfaces

  • Testing direct-component-to-component connectivity

  • Testing indirect-component-to-component connectivity

  • Testing whole structure connectivity

  • Testing of NFR? How to test for usability, safety, etc?


Path based testing sa

Path-Based Testing & SA?

  • SA Method

    • Used for prediction of the system-level quality

  • The approach would belong to the black box techniques of state transition testing

    • The system specification is modeled by an automata

    • The generation of test cases is aimed at covering the arcs (the transitions) and the nodes ( the states) of it

    • Create sub-graphs from SPEC representing specific views of a system

    • Use these views as a base for the definition of coverage criteria and testing strategies.

    • Select test cases to cover these sub-graphs


The advantages of using the sa to derive the automaton

The advantages of using the SA to derive the automaton

  • for state transition testing are evident:

    • we have a formal description and the automaton can be automatically derived;

    • the SA is at a high level of abstraction, thus the number of states in the automaton can be kept manageable


References acknowledgements


  • 1) Software testing by Paul C. Jargensen

  • 2) Art of Software Testing by Mayer

  • 3) Writing Good Software Engineering Research Papers (Mini-tutorial), Mary Shaw, ICSE02

  • Software engineering: Practitioner ’s approach,7th edition, 2009 by Roger Pressman:

  • Software Testing and Analysis by Mauro Pezze and Michal Young

  • Software Testing by Paul Ammann and Jeff Offutt


Writing good software engineering research papers

Writing Good Software Engineering Research Papers

  • In a research paper, the author explains to an interested reader

    • what he or she accomplished

    • how the author accomplished it

    • why the reader should care.


What kinds of questions do software engineers investigate

What kinds of questions do software engineers investigate?

  • Generally speaking, software engineering researchers seek better ways to develop and evaluate software.

    • Development

      • includes all the synthetic activities that involve creating and modifying the software, including the code, design documents, documentation, etc.

    • Evaluation

      • includes all the analytic activities associated with predicting, determining, and estimating properties of the software systems, including both functionality and extra-functional properties such as performance or reliability.


A good research paper need to answer the following

A good research paper need to answer the following:

  • What, precisely, was your contribution?

  • What question did you answer?

  • Why should the reader care?

  • What is your new result?

  • What new knowledge have you contributed that can be used elsewhere?

  • What previous work (yours or someone else’s)do you build on?

  • What do you provide as a superior alternative ?

  • How is your result different from and better than this prior work?

  • What, precisely and in detail, is your new result?

  • Why should the reader believe your result?

  • What standard should be used to evaluate your claim?


What is your new result

What is your new result?

  • Explain precisely what you have contributed to the store of software engineering knowledge.

  • how this is useful beyond your own project?


About survey paper

About survey paper

  • Example of abstract for survey paper:

    • In this report (or paper), we survey a number of X-based representations/algorithms/ tool support used to generate, identify, execute, etc.

    • Many problems with references and citation

    • Analysis (Comparisons )

  • Examples of citation:


About presentations

About Presentations

  • Each presentation will be 30 minutes long, including 5 minutes for questions/discussion.

  • The presentation will be divided into the following parts

    • Problem definition/motivation

    • What are the challenges?

    • Background literature surveyed by the authors

    • Specific technique developed in this paper

    • Weaknesses of the technique.




  • Each group will need to develop a power-point presentation

  • The presentation will be evaluated using the following criteria

    • Originality

    • Clarity

    • Organization

    • Punctuation


System testing

System testing

  • Objective of unit and integration testing was to ensure that the code implemented the design properly

  • Functional Testing

    • Black-box testing

  • The goal is to Test the Quality + functionalities

  • Works with notion of threads (or scenarios)

  • A thread is a sequence of events (or ASFs)

    • Provides a unifying view of our three level of testing

      • e.g.,

        • A scenario of normal usage

        • A stimulus/response pair

        • A sequence of machine instructions

        • A sequences of ASFs

  • Identifying threads?

    • FSM (node/edge coverage metrics)

      • Top/down

      • Bottom/up


System testing techniques

System testing techniques

  • System testing attempts to validate system-wide behaviors and properties of the entire system

    • E.g.,

      • performance,

      • security,

      • recovery, etc.


  • Login