Test purposes: adapting the notion of specification to testing

Download Presentation

Test purposes: adapting the notion of specification to testing

Loading in 2 Seconds...

- 94 Views
- Uploaded on
- Presentation posted in: General

Test purposes: adapting the notion of specification to testing

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Test purposes:adapting the notion of specification to testing

Yves Ledru,

L. du Bousquet, P. Bontron,

O. Maury, C. Oriat, M.-L. Potet

LSR/IMAG

Grenoble, France

ASE2001, Nov 27-29 2001, San Diego

Given a black box

And its specification

Test cases compare the black box to its specification

?sqrt(4)

sqrt

!2

accept

X = sqrt(X)*sqrt(X)

and

sqrt(X) >= 0

Unfortunately, complex applications

often result in complex test cases!

?Money_Transfer(1000000€)

e-Bank

Money_transfer

!OK

accept

It then makes sense to specify these complex test cases

« I want a test case that will execute the Money_transfer procedure successfully! »

It is the notion of test purpose!

?Money_Transfer(1000000€)

!OK

accept

- COTE:
- conformance testing of components vs UML specifications
- Softeam, France Telecom, Gemplus, IRISA, LSR/IMAG
- Languages and tools to express and exploit test purposes and test cases
- (compilers from test cases to target technologies)

- This work comes from a brainstorming activity around the notion of test purpose within the COTE project

Test purpose :

Description of a precise goal of thetest case, in terms of exercising a particular execution path or verifying the compliance with a specific requirement

[ISO IEC 9646-1 …conformance testing methodology and framework…]

- This notion is used:
- As a structuring element in industrial test suites
- In some tools based on formal specifications(SAMSTAG, TGV, TorX)

- TGV is a tool developed jointly by IRISA (Rennes) and Vérimag (Grenoble)
- Test synthesis tool
- Given
- A specification of the system under test
- A test purpose
It generates one corresponding test case.

Pay 1 or 2 coins

?coin(1)

?coin(2)

?coin(1)

Ask for sugar (optional)

?sugar

?coffee

Ask for tea or coffee

?tea

?coffee

?tea

Get the beverage

!coffee

!tea

!tea

!sugar

!coffee

!sugar

Loop

Dynamic specification

Test purpose

Test case

?coin(1)

?coin(1)

?coin(2)

?coin(2)

?coin(2)

?coin(1)

?coin(1)

?sugar

?sugar

?coffee

?coffee

?coffee

?tea

?coffee

?tea

?tea

?coffee

?tea

!coffee

!coffee

!tea

accept

!coffee

!tea

!tea

!tea

!sugar

!coffee

!coffee

!sugar

!coffee

!sugar

!sugar

TGV

?coin(1)

?coin(2)

?coin(1)

?coin(1)

?coin(2)

?coin(1)

?sugar

?coffee

?sugar

?tea

?coffee

?coffee

?tea

!coffee

!coffee

accept

!tea

!coffee

!sugar

!coffee

- A test purpose (P) is an abstraction of test cases
- In TGV, a partial sequence of the test cases
- A test purpose specifies a set of test cases

!coffee

accept

- Capture the essence of the test(here test the delivery of coffee)
- Shorter to state than the full test case(clerical completion left to the tool)
- Robust to evolutions of the dynamic specification(test cases are affected more often than test purposes, e.g. pay 3 instead of 2)

Relations between test purposes (P), test cases (C) and specifications (S)

?coin(2)

?coffee

!coffee

!coffee

accept

!coffee

test case not conforming

to the specification

- A test case « refines » the test purpose(if it includes the partial sequence)
- But several irrelevant test cases refine the test purpose also.Þ Weak refinement!

?coin(1)

?coin(2)

?coin(2)

?coin(1)

?sugar

?coffee

?coffee

?tea

?coffee

?tea

!coffee

accept

!coffee

!tea

!tea

!coffee

!sugar

!coffee

!sugar

- The refinement takes place in the context of the specification

refines

- Refines is a ternary relation
- Weakly_refines is defined as
weakly_refines(C,P) ÛE S ¥ refines(C,P,S)

weakly_refines

Test case

(C)

Test Purpose

(P)

refines

Specification

(S)

- Depending on the languages used for C, P, and S, a semantics can be defined for refines(C,P,S)
- Semantics of refines in TGV: C is one of the paths of the synchronous product of P and S
- See also, in the paper:semantics of a test refinement relation in the context of the B method

- As a consequence:
weakly_refines(C,P) ÛE S ¥ refines(C,P,S)

gives a semantics to weakly_refines on top of the semantics of refines

Test case

(C)

Test Purpose

(P)

refines

conforms

Specification

(S)

conforms(C,S) ÛE P ¥ refines(C,P,S)

- Conforms checks that the test case corresponds to the specification

Test case

(C)

Test Purpose

(P)

refines

consistent

Specification

(S)

consistent(P,S) ÛE C ¥ refines(C,P,S)

- Consistent checks if the test purpose is compatible with the specification

weakly_refines

Test case

(C)

Test Purpose

(P)

refines

conforms

consistent

Specification

(S)

- If refines(C,P,S) has a semantics,it gives a semantics to the binary relations!
- In some cases, it is possible to systematically construct the third element of the relation

Test specificationvssoftware specification

Software

Specification

(SS)

refinesSW

Program

(M)

Test case

(C)

Test Purpose

(P)

refines

Specification

(S)

- Formal specs as a basis for software tools:
- Verification of specification consistency
- Program synthesis
- Program verification

!sugar

?coin(*)

!coffee

reject

accept

often demonstrated by proving:

E M ¥ refinesSW(M,SS)

(there exists a model which refines the specification)

corresponds to

consistent(P,S) ÛE C ¥ refines(C,P,S)

(a test purpose makes sense if there exists a test case)

Tool

Detection of inconsistent

purposes

(e.g. !sugar comes after !coffee)

avoids useless searches

Major theme at ASE

synthesisSW(SS) M

corresponds to

test synthesis

e.g., TGV can be seen as a function

synthesisTGV(P,S) C

Good news for tools

The search domain is restricted by S

(more decidable?, less complex?)

checks refinesSW(M,SS)(program M is a valid refinement of specification SS)

corresponds to3 possible test case verifications

- refines(C,P,S)C is a refinement of P in the context of S
- weakly_refines(C,P)C is a refinement of P in some context
- conforms(C,S)C is a test case for S

- Synthesis involves searching a huge space of solutions
- If verification is cheaper than synthesis,it provides a natural complement.
- Two examples:
- Evolution of specifications
- Test case retrieval

?tea

?coffee

?choc

- Evolution of S into S’
- Given a set of (Ci,Pi) for S
- Verify refines(Ci,Pi,S’) for each (Ci,Pi)
- When it fails, verify conforms(Ci,S’)
- A second failure reveals regression of S’

- Example:
No need to recompute existing test cases if the coffee machinealso serves hot chocolate

?coin(2)

?coffee

!coffee

accept

?coin(2)

!coffee

accept

Given a large set of test cases

- Retrieve a test that weakly_refines a new test purpose
- Minimize the set by identifying test cases that weakly_refine several test purposes

weakly_refines

weakly_refines

Related workand conclusion

1st category: domain independent

Coverage criteria

Fault models (mutation testing)

Analysis rules, strategies

Random testing

2nd category: exploits domain knowledge

Test purposes(based on reqts)

Operational profiles

Must be consistent with the reference

Test selection criterion

(aka test requirement)

Reference

(specification,

program)

Synthesis tool

Test case(s)

- Test purposes are a specification of test cases!
- 4 relations between C, P, and S
- Not specific to a given tool or language
- Not specific to an application domain (TGV: reactive systems, B: transf. systems)

- Tools: from ASE to Automated Test Eng.
- Based on these 4 relations
- Test verification complements test synthesis
- Applications to specification evolution, test case retrieval,…