Test purposes adapting the notion of specification to testing
This presentation is the property of its rightful owner.
Sponsored Links
1 / 33

Test purposes: adapting the notion of specification to testing PowerPoint PPT Presentation


  • 82 Views
  • Uploaded on
  • Presentation posted in: General

Test purposes: adapting the notion of specification to testing. Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble, France. ASE2001, Nov 27-29 2001, San Diego. Given a black box And its specification.

Download Presentation

Test purposes: adapting the notion of specification to testing

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Test purposes adapting the notion of specification to testing

Test purposes:adapting the notion of specification to testing

Yves Ledru,

L. du Bousquet, P. Bontron,

O. Maury, C. Oriat, M.-L. Potet

LSR/IMAG

Grenoble, France

ASE2001, Nov 27-29 2001, San Diego


Conformance testing

Given a black box

And its specification

Test cases compare the black box to its specification

?sqrt(4)

sqrt

!2

accept

X = sqrt(X)*sqrt(X)

and

sqrt(X) >= 0

Conformance testing


Complex test cases

Unfortunately, complex applications

often result in complex test cases!

?Money_Transfer(1000000€)

e-Bank

Money_transfer

!OK

accept

Complex test cases


Specifying test cases

It then makes sense to specify these complex test cases

« I want a test case that will execute the Money_transfer procedure successfully! »

It is the notion of test purpose!

?Money_Transfer(1000000€)

!OK

accept

Specifying test cases


The cote project

The COTE project

  • COTE:

    • conformance testing of components vs UML specifications

    • Softeam, France Telecom, Gemplus, IRISA, LSR/IMAG

    • Languages and tools to express and exploit test purposes and test cases

    • (compilers from test cases to target technologies)

  • This work comes from a brainstorming activity around the notion of test purpose within the COTE project


The notion of test purpose

The notion of test purpose

Test purpose :

Description of a precise goal of thetest case, in terms of exercising a particular execution path or verifying the compliance with a specific requirement

[ISO IEC 9646-1 …conformance testing methodology and framework…]

  • This notion is used:

    • As a structuring element in industrial test suites

    • In some tools based on formal specifications(SAMSTAG, TGV, TorX)


Test purposes in tgv

Test purposes in TGV

  • TGV is a tool developed jointly by IRISA (Rennes) and Vérimag (Grenoble)

  • Test synthesis tool

  • Given

    • A specification of the system under test

    • A test purpose

      It generates one corresponding test case.


Example coffee machine

Pay 1 or 2 coins

?coin(1)

?coin(2)

?coin(1)

Ask for sugar (optional)

?sugar

?coffee

Ask for tea or coffee

?tea

?coffee

?tea

Get the beverage

!coffee

!tea

!tea

!sugar

!coffee

!sugar

Loop

Example : coffee machine


A sample synthesis

Dynamic specification

Test purpose

Test case

?coin(1)

?coin(1)

?coin(2)

?coin(2)

?coin(2)

?coin(1)

?coin(1)

?sugar

?sugar

?coffee

?coffee

?coffee

?tea

?coffee

?tea

?tea

?coffee

?tea

!coffee

!coffee

!tea

accept

!coffee

!tea

!tea

!tea

!sugar

!coffee

!coffee

!sugar

!coffee

!sugar

!sugar

A sample synthesis

TGV


Test purposes as specification

?coin(1)

?coin(2)

?coin(1)

?coin(1)

?coin(2)

?coin(1)

?sugar

?coffee

?sugar

?tea

?coffee

?coffee

?tea

!coffee

!coffee

accept

!tea

!coffee

!sugar

!coffee

Test purposes as specification

  • A test purpose (P) is an abstraction of test cases

  • In TGV, a partial sequence of the test cases

  • A test purpose specifies a set of test cases


Benefits of abstraction

!coffee

accept

Benefits of abstraction

  • Capture the essence of the test(here test the delivery of coffee)

  • Shorter to state than the full test case(clerical completion left to the tool)

  • Robust to evolutions of the dynamic specification(test cases are affected more often than test purposes, e.g. pay 3 instead of 2)


Relations between test purposes p test cases c and specifications s

Relations between test purposes (P), test cases (C) and specifications (S)


Weakly refines c p

?coin(2)

?coffee

!coffee

!coffee

accept

!coffee

test case not conforming

to the specification

Weakly_refines(C,P)

  • A test case « refines » the test purpose(if it includes the partial sequence)

  • But several irrelevant test cases refine the test purpose also.Þ Weak refinement!


Refines c p s

?coin(1)

?coin(2)

?coin(2)

?coin(1)

?sugar

?coffee

?coffee

?tea

?coffee

?tea

!coffee

accept

!coffee

!tea

!tea

!coffee

!sugar

!coffee

!sugar

refines(C,P,S)

  • The refinement takes place in the context of the specification

refines


Weakly refines vs refines

weakly_refines vs refines

  • Refines is a ternary relation

  • Weakly_refines is defined as

    weakly_refines(C,P) ÛE S ¥ refines(C,P,S)

weakly_refines

Test case

(C)

Test Purpose

(P)

refines

Specification

(S)


Semantics for refines c p s

Semantics for refines(C,P,S)

  • Depending on the languages used for C, P, and S, a semantics can be defined for refines(C,P,S)

  • Semantics of refines in TGV: C is one of the paths of the synchronous product of P and S

  • See also, in the paper:semantics of a test refinement relation in the context of the B method


Semantics of weakly refines

Semantics of weakly_refines

  • As a consequence:

    weakly_refines(C,P) ÛE S ¥ refines(C,P,S)

    gives a semantics to weakly_refines on top of the semantics of refines


Conforms c s

Test case

(C)

Test Purpose

(P)

refines

conforms

Specification

(S)

conforms(C,S)

conforms(C,S) ÛE P ¥ refines(C,P,S)

  • Conforms checks that the test case corresponds to the specification


Consistent p s

Test case

(C)

Test Purpose

(P)

refines

consistent

Specification

(S)

consistent(P,S)

consistent(P,S) ÛE C ¥ refines(C,P,S)

  • Consistent checks if the test purpose is compatible with the specification


Summary

weakly_refines

Test case

(C)

Test Purpose

(P)

refines

conforms

consistent

Specification

(S)

Summary

  • If refines(C,P,S) has a semantics,it gives a semantics to the binary relations!

  • In some cases, it is possible to systematically construct the third element of the relation


Test specification vs software specification

Test specificationvssoftware specification


Test purposes adapting the notion of specification to testing

Software

Specification

(SS)

refinesSW

Program

(M)

Test case

(C)

Test Purpose

(P)

refines

Specification

(S)


Automated software engineering

Automated Software Engineering

  • Formal specs as a basis for software tools:

    • Verification of specification consistency

    • Program synthesis

    • Program verification


Specification consistency

!sugar

?coin(*)

!coffee

reject

accept

Specification consistency

often demonstrated by proving:

E M ¥ refinesSW(M,SS)

(there exists a model which refines the specification)

corresponds to

consistent(P,S) ÛE C ¥ refines(C,P,S)

(a test purpose makes sense if there exists a test case)

Tool

Detection of inconsistent

purposes

(e.g. !sugar comes after !coffee)

avoids useless searches


Synthesis

Synthesis

Major theme at ASE

synthesisSW(SS)  M

corresponds to

test synthesis

e.g., TGV can be seen as a function

synthesisTGV(P,S)  C

Good news for tools

The search domain is restricted by S

(more decidable?, less complex?)


Verification

Verification

checks refinesSW(M,SS)(program M is a valid refinement of specification SS)

corresponds to3 possible test case verifications

  • refines(C,P,S)C is a refinement of P in the context of S

  • weakly_refines(C,P)C is a refinement of P in some context

  • conforms(C,S)C is a test case for S


Tools for test verification

Tools for test verification

  • Synthesis involves searching a huge space of solutions

  • If verification is cheaper than synthesis,it provides a natural complement.

  • Two examples:

    • Evolution of specifications

    • Test case retrieval


Evolution of specifications

?tea

?coffee

?choc

Evolution of specifications

  • Evolution of S into S’

    • Given a set of (Ci,Pi) for S

    • Verify refines(Ci,Pi,S’) for each (Ci,Pi)

    • When it fails, verify conforms(Ci,S’)

    • A second failure reveals regression of S’

  • Example:

    No need to recompute existing test cases if the coffee machinealso serves hot chocolate


Test case retrieval

?coin(2)

?coffee

!coffee

accept

?coin(2)

!coffee

accept

Test case retrieval

Given a large set of test cases

  • Retrieve a test that weakly_refines a new test purpose

  • Minimize the set by identifying test cases that weakly_refine several test purposes

weakly_refines

weakly_refines


Related work and conclusion

Related workand conclusion


Test selection criteria

1st category: domain independent

Coverage criteria

Fault models (mutation testing)

Analysis rules, strategies

Random testing

2nd category: exploits domain knowledge

Test purposes(based on reqts)

Operational profiles

Must be consistent with the reference

Test selection criterion

(aka test requirement)

Reference

(specification,

program)

Synthesis tool

Test case(s)

Test selection criteria


Conclusion

Conclusion

  • Test purposes are a specification of test cases!

  • 4 relations between C, P, and S

    • Not specific to a given tool or language

    • Not specific to an application domain (TGV: reactive systems, B: transf. systems)

  • Tools: from ASE to Automated Test Eng.

    • Based on these 4 relations

    • Test verification complements test synthesis

    • Applications to specification evolution, test case retrieval,…


Interactive coffee crafting

Interactive Coffee Crafting


  • Login