Integration and system testing
Sponsored Links
This presentation is the property of its rightful owner.
1 / 87

Integration and system Testing PowerPoint PPT Presentation


  • 464 Views
  • Uploaded on
  • Presentation posted in: General

Integration and system Testing . Csci565 Fall 2012. Objectives. Integration Testing Simple ATM (SATM) discuss integration testing strategies Decomposition Based Integration Testing (DBIT) Top-down Bottom-up Call Graph Based Integration Testing (CGBIT)

Download Presentation

Integration and system Testing

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Integration and system Testing

Csci565

Fall 2012

H.Reza


Objectives

  • Integration Testing

  • Simple ATM (SATM)

  • discuss integration testing strategies

    • Decomposition Based Integration Testing (DBIT)

      • Top-down

      • Bottom-up

    • Call Graph Based Integration Testing (CGBIT)

    • Path-Based Integration Testing (PBIT)

H.Reza


A Software Testing Strategy: Spiral Model

System Testing

validation Testing

Integration Testing

Unit Testing

Code

Design

Requirements

System eng.


Integration Testing:1

  • If all units/components work individually, do they work as a whole when we put them together?

    • Not necessarily

  • The problem is “putting them together” or interfacing them

H.Reza


Problems with interfacing

  • Integration faults often traceable to incomplete or misunderstood interface specifications

    • mismatched assumptions about other components

    • Individually acceptable imprecision may be magnified to unacceptable levels

    • Global data structures can present problems

  • Inconsistent interpretation of parameters or values

  • Mixed units (meters/yards) in Martian Lander

  • Violations of value domains, capacity, or size limits

H.Reza


Integration testing

  • Tests complete systems or subsystems composed of integrated components

  • Integration testing should be black-box testing when tests derived from the specification

  • Main difficulty is localising errors

  • Incremental integration testing reduces this problem

H.Reza


Approaches to integration testing

  • Two major approaches

    • Incremental approaches

      • The decomposition-based techniques or tree

        • Stubs/Drivers

      • Call Graph-based techniques

        • No stubs or drivers

    • Non-incremental approaches

H.Reza


Incremental integration testing

H.Reza


Incremental approaches: Top-down

  • Top-down testing

    • Start with high-level system

    • integrate from the top-down replacing individual components by stubs where appropriate

      • Depth-first

      • Breadth-first

      • No-best order

        • Critical sections

        • Early skeletal version using I/O modules

H.Reza


Stubs

  • Stubs

    • Special module to simulate some functionality

    • Its production is nontrivial task because the code may simulate a very complicated tasks

      • E.g.

        • Writing a stub performing a database table search routine

    • Creating multiple version of same stub for various reasons

H.Reza


Top-down testing

H.Reza


Top-down: Complications

  • The most common complication occurs when processing at low level hierarchy demands adequate testing of upper level

  • To overcome:

    • Either, delay many tests until stubs are replaced with actual modules (BAD)

    • Or, develop stubs that perform limited functions that simulate the actual module (GOOD)

    • Or, Integrate the software using bottom up approach

H.Reza


Incremental testing: Bottom up

  • Bottom-up testing

    • Integrate individual components in levels until the complete system is created

H.Reza


Bottom-up approach

  • Starts with construction and testing with atomic modules

    • No need for stub

    • Low level components are combined into cluster (or builds) to perform a specific sub-function

    • A driver (a control program for testing) is written

      • Contain hardcoded test input, calls the module being tested, and display the results

    • Cluster is tested

    • Drivers are removed and clusters are combined moving upward in the program structure

H.Reza


Bottom-up testing

M

H.Reza


Guideline for Integration testing

  • Integrate the components that implement the most frequently used functionality

  • Perform regression testing for old features

  • Perform progression testing for new features

  • Study the Architecture design of the Application

    • identify the Critical Modules.

  • Obtain the interface designs from the Architectural team and create test cases to verify all of the interfaces in detail.

    • E.g., Interface to database/external hardware/software application must be tested in detail.

H.Reza


Non-incremental

  • Big-bang

    • Imposes no order (GOOD)

    • Test all the units (Modules) at once (GOOD)

    • Very easy (GOOD)

    • difficult to localize the source of errors (VERY BAD)

H.Reza


Top-down Vs. Bottom-up

  • Architectural validation

    • Top-down integration testing is better at discovering errors in the system architecture

  • System demonstration

    • Top-down integration testing allows a limited demonstration at an early stage in the development

  • Test implementation

    • Often easier with bottom-up integration testing

H.Reza


Primary Design of the ATM using Tree-based decomposition

Requirement

specifications

Terminal I/O

Mange

Session

Conduct

Transactions

Select

Transaction

Card Entry

PIN Entry

H.Reza


Simple ATM (SATM)

  • An ATM simple

    • Provides 15 screens for interactions

    • includes 3 function buttons

      • B1, B2, B3

    • Modeled in structural analysis

      • Data Model (ERD)

      • Functional Model (DFD)

      • Behavioral model (STD)

H.Reza


Figure 12.7

H.Reza


Figure 12.8

H.Reza


Figure 12.9

H.Reza


Figure 12.10

H.Reza


Entity/Relationship model of the SATM


Figure 12.12

H.Reza


Figure 12.13

H.Reza


Decomposition based strategies

  • Decomposition based

    • Top/down

    • Bottom up

    • Sandwich

    • Big bang

H.Reza


Figure 12.14

H.Reza


PDL description of SATM

H.Reza


H.Reza


H.Reza


Figure 13.1

H.Reza


Decomposition based testing:1

  • Discussion revolves around the tree-based decomposition and the order by which units are tested and combined

    • Top-to-bottom

    • Bottom-to-top

    • Sandwich

    • Big bang

  • The focus is on the structural compatibility among interfaces

H.Reza


Test Sessions

  • A test session refers to one set of tests for a specific configuration of actual code and stubs

  • The number of integration test sessions using a decomposition tree can be computed

    • Sessions=nodes – leaves + edges

H.Reza


decomposition based testing: 2

  • For SATM system

    • 42 integration testing session (i.e., 42 separate sets of integration test cases)

      • top/down

        • (Nodes-1) stubs are needed

          • 32 stub in SATM

      • bottom/up

        • (Nodes-leaves) of drivers are needed

          • 10 drivers in SATM

H.Reza


Decomposition based strategies: Pros and con

  • Intuitively clear and understandable

    • In case of faults, most recently added units are suspected ones

  • Can be tracked against decomposition tree

  • Suggests breadth-first or depth-first traversals

  • Units are merged using the decomposition tree

    • Implies correct behavior should follow from individually correct units and interfaces

    • Stubs/Drives are major development Overhead

H.Reza


Call graph based integration testing

  • Call graph

    • A directed graph

    • Nodes corresponds to unit

    • Edges corresponds to the call

    • E.g.

      • AB (i.e., A is calling B)

  • Attempts to overcome the decomposition problem (structural)

  • Moves toward behavioral testing

H.Reza


Call graph based(CG): approaches

  • Two main approaches based on Call Graph

    • Pair-wise integration

    • Neighborhood integration

H.Reza


Table 2: AM

H.Reza


Pair-wise integration

  • The main idea is to eliminate the overhead (i.e., stub/drive)

  • Uses actual code by restricting a session testing to a pair of units in the Call Graph

    • One integration test for each edge in Call Graph

    • 40 edges means 40 integration tests for the SATM

H.Reza


Pairwise

Uses actual code

-one integration test session for each edge

-40 edges for SATM

H.Reza


Neighborhood Integration

  • The neighborhood of a node refers to the nodes that are one edge away from the given nodes

    • SATM Neighborhoods

    • Number of neighborhoods can be computed:

      • 1) Neighborhoods = Interior node + source node

      • 2) Interior nodes =nodes – (source nodes + sink nodes)

        • ( x=1 if there exists leaf nodes connected directly to the root node otherwise X= 0)

  • Results a drastic reduction in the number of integration test session

    • In case of SATM (11 vs. 40)

H.Reza


Neighborhood Integration

H.Reza


Table 3: SATM Neighborhoods

H.Reza


Pros and cons

  • Benefits (GOOD)

    • Mostly behavioral than structural

    • Eliminates sub/drive overhead

    • Works well with incremental development method such as Build and composition

  • Liabilities (BAD)

    • The fault isolation

      • E.g.,

        • Fault in one node appearing in several neighborhood

H.Reza


Questions about Integration testing

  • How do perform Integration testing for non-tree based functional decomposition?

    • E.g.,

      • integration testing for OO

      • Integration testing for Client/server systems

      • Integration testing for Layered systems

      • ….

H.Reza


Path-based Integration Testing

  • The hybrid approach (i.e., structural and behavioral) is an ideal one integration testing

  • The focus is on the interactions among the units

    • Interfaces are structural

    • Interactions are behavioral

  • With unit testing, some path of source statements is traversed

    • What happens when there is a call to another unit?

      • Ignore the single-entry/single-exit

      • Use exist follows by an entry sequence

      • Suppress the call statement

H.Reza


New and extended concepts

  • Source node (begin)

    • A statement fragment at which program execution begins or resumes

    • E.g., BEGIN

  • Sink node (end)

    • A statement fragment at which program execution terminates

    • E.g., END

H.Reza


More on concepts

  • Module Execution Path (MEP)?

    • A sequence of statements that begins with a source node and ends with a sink node, with no intervening sink nodes

  • Message?

    • A programming language mechanism by which one unit transfers control to another unit

      • E.g.,

        • subroutine invocations

        • Procedure calls

        • Function references

H.Reza


The path based integration testing (definition)

  • Definition: MM-Path

    • An interleaved sequence of module execution paths(MEP) and messages

  • MM-Path can be used

    • to describe sequences of module execution paths including transfers of control among units using messages

    • To represent feasible execution paths that cross unit boundaries

    • To extent program graph

      • Where

        • nodes = MEP

        • edges = messages

  • Atomic system function (ASF)?

    • An action that is observable at the system level in terms of port input and output events (at GUI)

H.Reza


Figure 13.3

MM-path

H.Reza


There are seven module execution paths (MEP):

MM-path

H.Reza


MM-Path Graph (definition)

  • Definition: MM-Path Graph?

    • Given a set of units, their MM-Path graph is the directed graph in which

      • nodes = MEPs

      • edges = messages/returns

H.Reza


Figure 13.4

H.Reza


SATM: Code


SATM: Code 2


Code: 3


MM-Path for correct for correct PIN Entry


How long should an MM-Path Be:1

  • MM-Path issues

    • How long is an MM-Path?

    • What is the endpoint?

  • The following observable behavior are used as endpoints

    • Event quiescence (event inactivity)

      • System level event

      • Happens when system is ideal/waiting

    • Message quiescence (msg inactivity)

      • Unit that sends no messages is reached (module C)

    • Data quiescence (data inactivity)

      • Happens when a sequences of processing generate a stored data that is not immediately used

      • E.g., account balance that is not used immediately

H.Reza


How long should an MM-Path Be:2

H.Reza


MM-Path Guidelines

  • MM-Path guidelines:

    • Points of

    • (data, event, or msg) are natural endpoints for an MM-path

    • Atomic System Functions (system behavior) are considered as an upper limit for MM-Paths

      • MM-Paths should not cross ASF boundaries

H.Reza


Pros and cons

  • hybrid approach(GOOD)

  • The approach works equally well for software testing developed by waterfall model (GOOD)

  • Testing closely coupled with actual system behavior (GOOD)

  • Identification of MM-Paths which can be offset by elimination of sub/drivers(BAD)

  • Test case generation ?

H.Reza


Summary of integration testing: 1

  • Integration testing

    • Testing of incompatibilities and interfaces between otherwise correctly working components

    • Performed with an incomplete system using scaffolding(drivers/stubs/mocks)

      • Stubs main job is to return values to the calling program

        • Simple stub assigns constant values to the outputs

H.Reza


Summary of integration testing:2

  • Drivers main job is to control and set up testing ; they can include hard-code values or retrieve the values from an external source

  • Tools exist to generate both stubs/drivers automatically

H.Reza


Summary of integration testing: 3

  • Integration testing focuses on interface

    • 40% of software errors are revealed during integration testing

    • Must be built on foundation of thorough unit testing

    • Integration faults often traceable to incomplete or misunderstood interface specifications

      • Prefer prevention to detection, and make detection easier by imposing design constraints

  • Strategies tied to project build order

    • Order construction, integration, and testing to reduce cost or risk


Table: integration testing?


Integrated Testing Life cycle

H.Reza


Type of Integration Faults

  • Inconsistent interpretation of parameters or values

    • Example: Mixed units (meters/yards) in Martian Lander

  • Violations of value domains, capacity, or size limits

    • Example: Buffer overflow

  • Side effects on parameters or resources

    • Example: Conflict on (unspecified) temporary file

  • Omitted or misunderstood functionality

    • Example: Inconsistent interpretation of web hits

  • Nonfunctional properties

    • Example: Unanticipated performance issues

  • Dynamic mismatches

    • Example: Incompatible polymorphic method calls


Integration testing Using Software Architecture (SA)

  • Software Architectural Based Integration testing

    • How software architecture can be used to conduct tests to uncover errors related to the interface?

  • How can we test if a design and implementation comply using SA?

  • How can we specify a SA such that one or more properties can be tested or verified?

  • How integration testing can be planned and controlled based on the SA?

H.Reza


Software architecture & Testing

  • Using Traditional approach for formalizing and automating the integration test stage is difficult

    • the selection of the test cases for the subsystems stressstructureover behavior

    • the order in which the components are incrementally combined is entirely dependent on the structure (tree or graph)

H.Reza


Software Architecture (SA) & Testing

  • Most current testing approaches are either based

    • the implementation or structural

    • information of the system

    • a requirement specification

    • system design

    • the architecture level

  • Using the properties of formal architecture description languages (ADL), architecture based test criteria can be defined

H.Reza


issues

  • what to test at the software architecture?

    • At the integration and system testing level, the predominant form for defining testing criteria is based on definition/use bindings

      • each module defines or provides a set of facilities that are available to the users or requires by other modules

  • Software architecture

    • focuses on the interaction between components

    • Interaction is language independent

H.Reza


Testing Requirements for SA

  • Testing component-to-connector connectivity

  • Testing connector-to-component connectivity

  • Testing component –to-internal interfaces

  • Testing connector internal interfaces

  • Testing direct-component-to-component connectivity

  • Testing indirect-component-to-component connectivity

  • Testing whole structure connectivity

  • Testing of NFR? How to test for usability, safety, etc?

H.Reza


Path-Based Testing & SA?

  • SA Method

    • Used for prediction of the system-level quality

  • The approach would belong to the black box techniques of state transition testing

    • The system specification is modeled by an automata

    • The generation of test cases is aimed at covering the arcs (the transitions) and the nodes ( the states) of it

    • Create sub-graphs from SPEC representing specific views of a system

    • Use these views as a base for the definition of coverage criteria and testing strategies.

    • Select test cases to cover these sub-graphs

H.Reza


The advantages of using the SA to derive the automaton

  • for state transition testing are evident:

    • we have a formal description and the automaton can be automatically derived;

    • the SA is at a high level of abstraction, thus the number of states in the automaton can be kept manageable

H.Reza


References/Acknowledgements

  • 1) Software testing by Paul C. Jargensen

  • 2) Art of Software Testing by Mayer

  • 3) Writing Good Software Engineering Research Papers (Mini-tutorial), Mary Shaw, ICSE02

  • Software engineering: Practitioner ’s approach,7th edition, 2009 by Roger Pressman:

  • Software Testing and Analysis by Mauro Pezze and Michal Young

  • Software Testing by Paul Ammann and Jeff Offutt

H.Reza


Writing Good Software Engineering Research Papers

  • In a research paper, the author explains to an interested reader

    • what he or she accomplished

    • how the author accomplished it

    • why the reader should care.

H.Reza


What kinds of questions do software engineers investigate?

  • Generally speaking, software engineering researchers seek better ways to develop and evaluate software.

    • Development

      • includes all the synthetic activities that involve creating and modifying the software, including the code, design documents, documentation, etc.

    • Evaluation

      • includes all the analytic activities associated with predicting, determining, and estimating properties of the software systems, including both functionality and extra-functional properties such as performance or reliability.

H.Reza


A good research paper need to answer the following:

  • What, precisely, was your contribution?

  • What question did you answer?

  • Why should the reader care?

  • What is your new result?

  • What new knowledge have you contributed that can be used elsewhere?

  • What previous work (yours or someone else’s)do you build on?

  • What do you provide as a superior alternative ?

  • How is your result different from and better than this prior work?

  • What, precisely and in detail, is your new result?

  • Why should the reader believe your result?

  • What standard should be used to evaluate your claim?

H.Reza


What is your new result?

  • Explain precisely what you have contributed to the store of software engineering knowledge.

  • how this is useful beyond your own project?

H.Reza


About survey paper

  • Example of abstract for survey paper:

    • In this report (or paper), we survey a number of X-based representations/algorithms/ tool support used to generate, identify, execute, etc.

    • Many problems with references and citation

    • Analysis (Comparisons )

  • Examples of citation:

H.Reza


About Presentations

  • Each presentation will be 30 minutes long, including 5 minutes for questions/discussion.

  • The presentation will be divided into the following parts

    • Problem definition/motivation

    • What are the challenges?

    • Background literature surveyed by the authors

    • Specific technique developed in this paper

    • Weaknesses of the technique.

H.Reza


PowerPoint

  • Each group will need to develop a power-point presentation

  • The presentation will be evaluated using the following criteria

    • Originality

    • Clarity

    • Organization

    • Punctuation

H.Reza


System testing

  • Objective of unit and integration testing was to ensure that the code implemented the design properly

  • Functional Testing

    • Black-box testing

  • The goal is to Test the Quality + functionalities

  • Works with notion of threads (or scenarios)

  • A thread is a sequence of events (or ASFs)

    • Provides a unifying view of our three level of testing

      • e.g.,

        • A scenario of normal usage

        • A stimulus/response pair

        • A sequence of machine instructions

        • A sequences of ASFs

  • Identifying threads?

    • FSM (node/edge coverage metrics)

      • Top/down

      • Bottom/up

H.Reza


System testing techniques

  • System testing attempts to validate system-wide behaviors and properties of the entire system

    • E.g.,

      • performance,

      • security,

      • recovery, etc.

H.Reza


  • Login