System testing
This presentation is the property of its rightful owner.
Sponsored Links
1 / 15

System Testing PowerPoint PPT Presentation


  • 55 Views
  • Uploaded on
  • Presentation posted in: General

System Testing. Earlier we have stated the two views of testing: To show that the software works (according to specification) To detect as many defects (and have them fixed prior to releasing to the users) as possible

Download Presentation

System Testing

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


System testing

System Testing

  • Earlier we have stated the two views of testing:

    • To show that the software works(according to specification)

    • To detect as many defects(and have them fixed prior to releasing to the users) as possible

  • In system testing we are more interested in showing that the system, as a whole with all the “major” specifications – especially cross-component, is working.

  • Still Interested in knowing which “major” function is not working. (But this should be rare if we completed functional and component test properly.)

    • We are not focused on all types of exception during systems test, but want to make sure that when multiple paths (threads) are executed, it still works according to the requirements.

  • System testing is a blend of functional (black-box) and structural (white-box) testing - - - testing (mostly main) threads of system behavior.


Thread

Thread

  • A thread is a sequence of activities that are observable.

  • There are many levels of granularity for a thread:

    • A sequence of instructions as in a DU-pathfor unit test

    • A flow of control among a set of related modules as in a MM-path in an integration path.

    • A sequence of “atomic” functionsthat contributes to the accomplishment of a major functionality as described in a requirements specification.

  • In system testing, we are interested in the

    • a) high-level (cross-functional) threadsand

    • b) multiple threads.


Examples of threads from the atm problem p27 of the text

Examples of threads from the ATM Problem (p27 of the text)

Unit

Testing

  • Entry of a digit: (port entry of a numerical digit keystroke and a port out of a digit echo on the screen.

    • This is a minimal “atomic system function” and qualifies as a thread of interest at the unit test level.

  • Entry of a PIN number: (this entails a set of activities from screen request of a PIN with interleaved sequence of digit entry until completion - - - possibly including cancellation and up to three re-tries).

    • This is a sequence of atomic system functions that also qualifies as athread, probably fit for integration testing or unit testing levels.

  • A simple ATM transaction: (this includes a set of functions such as 1) Card entry, 2) PIN entry, 3) choice of transaction type (deposit, withdraw, account detail processing, etc.)

    • This is a sequence of atomic system functions that is a threadaccomplishes a major function, fit for integration testing or system testinglevels .

  • Multiple ATM transactions: (this includes a complete user session where several different transactions- - - query, withdraw, deposit of different accounts - - - may transpire.)

    • This is a sequence of different/multiple threads and is definitely in the domain of system testing.

System

Testing


Some definitions related to thread

Some definitions related to thread

  • An atomic system function (ASF)is an action that is observable at the system level in terms of port entry and port output.

  • An ASF graphis a directed graph of a ASF represented system where the nodes are the ASF’s and the edges represent the sequential flow.

  • A source ASFis an ASF that is a source node in an ASF graph.

  • A sink ASFis an ASF that is a sink node in an ASF graph.

  • A system threadis a path from a source ASF to a sink ASF in the ASF graph of the system.

  • The thread graph of a systemis a directed graph where the

    • a) nodes are the threads of the system and

    • b) the edges represent the sequential execution flow from one thread to another.


Requirements specification for system testing

Requirements Specification for System Testing

  • Requirements specification served as a tool (input) for, mainly black-box testing, but also white-box testing even at the unit testing level; it is even more important as a source of developing test cases for system testing.

  • There are many ways to express requirements and many ways to view the specification. As a whole the requirements specification may be viewed a set of inter-related elements (“basis” for requirements):

    • Data (information and information structure)

    • Actions (functional tasks)

    • Devices (for input, output, and storage)

    • Events (triggers and combination of action and data)

    • Threads (business flow or process flow, involving events)

Only thing missing is the non-functional attributesuch as quality, performance, etc.


Multiple ways of viewing specification basis elements

Multiple Ways of Viewing Specification “Basis” elements

Used mostly for

early spec and development of system devices

Data

Contextual

model

Actions

Behavior

model

Devices

Often used for requirements

specifications

Events

Structural

model

Threads

Used mostly for

development

There are many tools for expressing these models: (e.g.)

- state transition diagram ( or Finite State Machine)

- E-R diagram

- Petri net


Using finite state machine or state transition diagram

Using Finite State Machine or State Transition Diagram

Welcome

Card entry

(state A)

State Input Next State Output

A correct card B screen 2

A bad card A screen 1; card

B good pin # C screen 5

B failed pin # A screen 4

wrong card

Display screen 1;

Eject card

Correct card

Display screen 2

Pin

Entry

(state B)

Failed Pin Process

Display screen 4

Successful

Pin Process

Tabular form

Display screen 5

Choosing

Transaction

(state C)

Graphical form


Finite state machine at a deeper level for pin processing

Finite State Machine at a “deeper” level for PIN processing

Welcome

Card entry

(state A)

Incorrect Pin/

cancel

wrong card

Display screen 4, 1

Display screen 1;

Eject card

Correct card

Display screen 2

1st Pin

Entry

(state B)

Incorrect Pin/

cancel

2nd Pin

Entry

(state D)

Incorrect Pin/

cancel

3rd Pin

Entry

(state E)

Display screen 3, 2

Successful

Pin Process

Display screen 3, 2

Correct Pin

Display screen 5

Choosing

Transaction

(state C)

Display screen 5

Correct Pin

Display screen 5


Threads with finite state machine

Threads with Finite State Machine

  • If we can model the requirements specifications with the finite state machine model, then:

    • A set of transitions (with inputs and outputs) can be a source of threads for system test because it covers:

      • Events ( state and input)

      • Action (state transition and output)

      • Data (inputs and outputs)

      • Thread (series of state transitions)

  • The drawbacks are:

    • that not all specifications “easily” fit this model

    • that the determination of what depth levelshould the finite state machine be for system test; very low a level is more appropriate for unit or integration test


Test coverage metrics with finite state machine

Test Coverage Metrics with Finite State Machine

  • Every node (state) is covered.

    • This is a very minimal coverage, much like every statement is covered (recall this from structural path testing)

    • In our 1st state transition chart - - - this is equivalent to covering states A, B, and C with correct card and successful PIN processing.

  • Every node and edge (transition) is covered

    • This is a more complete coverage if the finite state machine is carried to deep enough level.


Strategies for thread testing finite state machine represented systems

Strategies for Thread testing Finite State Machine represented systems

  • Event based (state-input based):

    • Each input event occurs

    • A “typical” or “common” sequence of input events occur

    • Each input event occurs in every “relevant” data context

    • For a given context, all “inappropriate” input events occur

    • For a given context, all possible (both appropriate & inappropriate) inputs events occur

  • Event based (state-output based):

    • Each output event occurs

    • Each output event occurs for every “cause”

  • Port-based (both input and output):

    • Cover all input and out ports


Strategies for thread testing data represented systems

Strategies for thread Testing Data Represented Systems

  • Some software system requirements are better represented with E-R diagram.

  • For E-R modeling used for specifications, the threads may be identified via:

    • Looking at the cardinality of every relationship

      • One-one

      • One-many

      • Many-one

      • Many - many

    • Looking at the participation factor of every relationship

    • Looking at functional dependencies among relationships

      • e.g. loaning a book that’s not available

More importantly --- we want to make sure that the DB “access” is tested,

along with threads such as a) shutdown -restartb) backup-recovery

c) multiple access threads


Using operational profile when interested in effectiveness

Using Operational Profile when interested in“Effectiveness”

  • The effectiveness of testing is often measured in terms of problems found versus effort spent or

    • (# of defects found) / person hours spent

    • (# of defects found) / number of test cases

  • Historically, most of the failures also tend to fall in small parts of the system (80% failures occur in 20% of the system - - - e.g. the most heavily traversed threads).

    • Thus if we can collect the operational profiles of the users, we can identify the most heavily used threads.

    • Thisstrategy of system testing with operational profile can improve our test efficiency.

Not just the number of

problems found


User operational profile example na ve user versus experienced user

User-Operational Profile Example(naïve user versus experienced user)

(20/20)

(20/20)

prob = .25 √

term

naïve users

(20/80)

default f1

f4

(5/5)

prob = .063

(5/10)

f4

term

Start

(80 users)

(5/10)

(5/5)

prob = .063

(10/60)

f1

f6

term

experienced

users (60/80)

(40/40)

(40/40)

(40/60)

prob = .50√√

term

f7

f2

Choice of

f1,f2,f3

(10/60)

(3/10)

(3/3)

prob = .037

term

f4

f3

(7/10)

(7/7)

prob = .087

Note that 2 usage cases take

up 75% of operational profile !

f5

term


Software changes and regression test

Software changesand Regression Test

  • In the life span of a software, we can anticipate changes that result in multiple cycles of fixes and release. Thus it needs to be retested to ensure that what worked before still works (did not get regressed).

  • A common strategy is to re-run the complete system test over as the regression test

  • A more economical strategy for regression test is to develop (1) new threads for the new areas and(2) only include those old threads that are affected by the new release. (Yes, there may be an element of risk)

Think about the

“neighborhood”

testing strategy-last lecture


  • Login