Yves le traon cl mentine nebut
Download
1 / 48

Automatic test generation from system requirements and application to software product lines - PowerPoint PPT Presentation


  • 501 Views
  • Uploaded on

Automatic test generation from system requirements and application to software product lines Génération automatique de tests à partir des exigences et application aux lignes de produits logicielles. Yves Le Traon - Clémentine NEBUT. Outline. Introduction Requirement models Test generation

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Automatic test generation from system requirements and application to software product lines' - Angelica


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Yves le traon cl mentine nebut l.jpg

Automatic test generation from system requirements and application to software product linesGénération automatique de tests à partir des exigences et application aux lignes de produits logicielles

Yves Le Traon - Clémentine NEBUT


Outline l.jpg
Outline application to software product lines

  • Introduction

  • Requirement models

  • Test generation

  • Conclusions and future work

Introduction


Modern software issues l.jpg
Modern software issues application to software product lines

  • Software size grows up

    • testing methods have to adapt to the change of scale

  • System requirements ...

    • … evolve very often

      • Nokia : 69% of the requirements modified, 22% modified twice

      • need to build quickly new tests from the new requirements

    • … are in natural language

      • need of a formalization to apply automatic test generation techniques

  • Product line concept

    • Common requirements + variations

    • new testing methods

Introduction


Testing product lines l.jpg
Testing product lines application to software product lines

Classical approach

Test(P1)

Test(Pn)

  • Cost of the test generationlinear in n

  • Cost of a new product: heavy

Pn

P1

n products

Ideal approach

test

  • make the cost of the test generation decrease

  • factorize testing task

  • rapid validation of a new product

test

test

test

test

+ test

+ test

+ test

+ test

Introduction


Our objectives l.jpg

Make the cost of the system test generation decrease: application to software product lines

automatic test generation

from the requirements

Constraints:

1. complexity (size) of real software

2. compatibility with usual industrial practice

3. adaptability to the product lines context

Our objectives

Introduction


Automatic test generation classical functional approaches l.jpg
Automatic test generation: application to software product linesclassical functional approaches

Requirements

Beh. model

Test cases

Paths extraction

Oracle ? Feasibility ?

LTS, IOLTS, FSM, EFSM, ...

Z, B, SDL,...

Coverage criteria

Categoriesidentification

Partitionsidentification

Testselection

Categories

Choices

Test cases

  • Using behavioral models

  • Category-Partition approaches

Introduction

All transitions, all nodes, ...

1. Complexity ?

2. Industrial practice ?

3. Product lines context ?


Test generation from uml use cases l.jpg
Test generation from UML use cases application to software product lines

« Use cases are a means for specifying required usages of a system ».

OMG, UML 2.0.

  • Fröhlich et al (2000)

    • from cockburn-formatted use cases to state machines

  • Ryser et al (2000)

    • from use cases to statecharts, dependency charts

  • Riebish et al (2002)

    • statistical testing

  • Basanieri et al (2002)

    • cow_suite approach

  • Briand et al (2002)

    • TOTEM approach

Introduction


Existing approaches to test product lines l.jpg
Existing approaches to test product lines application to software product lines

  • Scarce !

  • Few automatic approaches

  • From use cases:

    • a systematic approach (Kamsties et al.)

    • PLUTO, an approach based on category/partition (Bertolino et al.)

Introduction


Overview of the proposed approach l.jpg
Overview of the proposed approach application to software product lines

requirement 1.1 "Register a book"

the "book" becomes "registered" after the "librarian" did "register" the "book".

the "book" is "available" after the "librarian" did "register" the "book".

Requirements

Requirement model

simulation

[connect(p1), plan(p1,m1)] [connect(p1), plan(p1,m1), open(p1,m1), close(p1,m1)]

Test objectives

Test scenarios

Test cases

Introduction


Outline10 l.jpg
Outline application to software product lines

Requirements

Requirement model

Test objectives

Test scenarios

Test cases

  • Introduction

  • Requirement models

  • Test generation

  • Conclusions and future work

Requirements models


Enhanced use case model l.jpg
Enhanced use case model application to software product lines

VirtualMtg

enter

plan

consult

A manager

open

manager

leave

user

A meeting

close

speak

The meeting existsand is organizedby the manager

Param

The meeting doesnot already exist

hand over

Plan

moderator

UC1

Pre

Post

connect

  • Enhancement of the UML use case model:

    • Parameters

      • actors or business concepts

      • represent the involved entities

    • Contracts

      • precondition and postcondition

      • handling the parameters

Requirements models


A use case contract language l.jpg
A use case contract language application to software product lines

p:participant

m:meeting

planned(m) andmanager(p,m)

not planned(m)

Plan

  • First order logic expressions

    • Boolean properties (predicates) = name+typed parameters

      • Ex: planned(m:meeting)

        manager(u:participant,m:meeting)

    • Enumerated properties

    • Classical boolean operators (and, or, implies, not)

    • Quantifiers (forall, exists)

  • Benefits:

    • formalization of the use cases

    • dependencies between the use cases can be deduced

Requirements models


Deducing dependencies example l.jpg
Deducing dependencies: example application to software product lines

#use case OPEN

UC open(u:participant;m:mtg)

pre created(m) and moderator(u,m) and not closed(m) andnot opened(m) and connected(u)

post opened(m)

Requirements models

#use case CLOSE

UC close(u:participant; m:mtg)

pre opened(m) and moderator(u,m)

post...

OPEN(u1,m1);CLOSE(u1,m1) is a correct sequence


Dealing with requirements in controlled natural language l.jpg
Dealing with requirements application to software product linesin controlled natural language

requirement 1.1 "Register a book"

the "book" becomes "registered" after the "librarian" did "register" the "book".

Requirements

  • A requirement description language (RDL)

    • Semantics given in terms of enhanced use cases

  • Benefits

    • compatible with the industrial practice

    • formalization of the natural language requirements

    • better traceability requirements / analysis models / tests

Requirements models

Requirement model


Test generation l.jpg
Test generation application to software product lines

Requirements

simulation

Requirement model

Test objectives

Test scenarios

Test cases

  • Introduction

  • Requirement models

  • Test generation

  • Conclusions and future work

Test generation


From requirements to test objectives l.jpg
From requirements to test objectives application to software product lines

Requirements

simulation

Requirement model

Test objectives

Test scenarios

Test cases

  • Introduction

  • Requirement models

  • Test generation

  • Conclusions and future work

Test generation


Simulation of a use case model l.jpg

Definition of the simulation state application to software product lines

Need of:

initial state

entities involved in the simulation

Simulation of a use case model

?

Plan(p1,m1)

connected(p1), connected(p2), planned(m1),manager(p1,m1)

  • Applying an instantiated use case:

    • checking the precondition

    • modifying the current state

connected(p1), connected(p2)

Test generation

p1,p2:participant

m1,m2:meeting

Plan(p:participant, m:meeting)

pre not planned(m) and connected(p)

post planned(m) and manager(p,m)


The use case transition system ucts l.jpg
The Use Case Transition System application to software product lines(UCTS)

connected(p1), created(m1), manager(p1,m1), moderator(p1,m1)

Test generation

open(p1,m1)

connected(p1), created(m1), manager(p1,m1),moderator(p1,m1)opened(m1)

enter(p1,m1)

close(p1,m1)

connected(p1), created(m1), manager(p1,m1), moderator(p1,m1)opened(m1), entered(p1,m1)

connected(p1), created(m1), manager(p1,m1), moderator(p1,m1)closed(m1)

close(p1,m1)


Principles of the test objective generation l.jpg
Principles of the test objective generation application to software product lines

Test objectives

{UC1(p1,p2), UC3(p2),UC4(p1)}

{UC3(p1),UC1(p2,p2)}

  • Test objective

    = path of the UCTS

    = sequence of instantiated use cases

  • Generating test objectives

    • Extracting short paths in the UCTS

    • Extracting a « reasonable » number of paths

    • Test criteria

      • 4 structural criteria

      • 1 semantic criterion

      • 1 robustness criterion

Test generation

UCTS

Testcriteria


4 structural test criteria l.jpg
4 structural test criteria application to software product lines

  • All transitions

  • All nodes

  • All instantiated use cases

  • All instantiated use cases and all nodes

C()

Test generation

N1

A(x1)

A(x2)

B(x1,y1)

N2

N3

C()

C()

B(x2,y1)

A(x2)

A(x1)

B(x2,y1)

N4

N5

C()

C()

B(x1,y1)

  • A(x:X)

  • B(x:X,y:Y)

  • C()

x1,x2:X

y1:Y


A semantic test criterion l.jpg
A semantic test criterion application to software product lines

B(x1)

{P(x1),Q(x2)}

A(x1)

B(x2)

{Q(x1),Q(x2)}

{P(x1)}

UC A(x:X)

pre P(x)

post not P(x) and Q(x)

UC B(x:X)

pre P(x) or Q(x)

post not Q(x)

B(x2)

B(x1)

B(x1)

A(x1)

x1,x2:X

{Q(x2)}

{Q(x1)}

B(x2)

B(x1)

{}

  • All precondition terms

    • Applying a use case in several different configurations making its precondition true

Test generation

  •  3 configurations

    • not P(x) and Q(x)

    • P(x) and not Q(x)

    • P(x) and Q(x)


A robustness criterion l.jpg
A robustness criterion application to software product lines

.

.

.

Correct path

Non specified action

  • Generate paths leading to an invalid application of the use case

  • Criterion

    • similar to all precondition terms (all the configurations that make the precondition fail)

Test generation


From test objectives to test scenarios l.jpg
From test objectives to test scenarios application to software product lines

Requirements

Requirement model

Test objectives

Test scenarios

Test cases

Test generation

simulation


A gap to bridge l.jpg
A gap to bridge ... application to software product lines

[connect(p1),plan(p1,m1)]

?

Test generation


Assumptions on the scenarios l.jpg
Assumptions on the scenarios application to software product lines

guards

Use caseparameters

Scenariosparameters

Effect onthe system

Test generation

{Nominal}

{Exceptional}


Principles of the test scenarios generation l.jpg
Principles of the test scenarios generation application to software product lines

SCA1

UCA

SCA2

Tests scenarios

SCB1

UCB

ISCA1

ISCA2

ISCA1

ISCA2

ISCB1

ISCB1

ISCB2

ISCB2

SCB2

= Strong sequential composition

Test generation

Test objective

Test scenario

  • Substitution of instantiated use cases by instantiated scenarios

  • Sequential composition

  • Cartesian product of sets of scenarios

IUCA

substitution

IUCB


Analysis of the verdicts l.jpg
Analysis of the verdicts application to software product lines

Initialization(test data)

Use cases

Scenarios

Generation

Test scenarios / test cases

Execution

Pass Verdict

Fail Verdict

Inconclusive Verdict

Errordetected

Test generation


Verdict fail example l.jpg
Verdict Fail: example application to software product lines

Violated => Error detected

:ATM

p1:user

Deposit(account1, amount1=40)

Constraint:account.amount> amount

p1:user

:ATM

Withdraw(account1, amount2=30)

Assert:account.amount>0


Analysis of the verdicts29 l.jpg
Analysis of the verdicts application to software product lines

Initialization(test data)

Use cases

Scenarios

Generation

Test scenarios / test cases

Execution

Pass Verdict

Fail Verdict

Inconclusive Verdict

Non-executed testscenario detected

Errordetected

Test generation


Verdict inconclusive example l.jpg
Verdict Inconclusive: example application to software product lines

Violated=> inconclusive

:ATM

Deposit(account1, amount3=20)

Constraint:account.amount> amount

:ATM

Withdraw(account1, amount2=30)

Assert:account.amount>0


Analysis of the verdicts31 l.jpg
Analysis of the verdicts application to software product lines

Initialization(test data)

Use cases

Scenarios

Generation

Test scenarios / test cases

Execution

Pass Verdict

Fail Verdict

Inconclusive Verdict

[ uncovereduse case scenarios]

Non-executed testscenario detected

Errordetected

Test generation


From test scenarios to test cases l.jpg
From test scenarios to test cases application to software product lines

in a RequirementDescription Language

Requirements

an enhanced use case model

simulation

Requirement model

based on simulation, using test criteria

Test objectives

possible use of test synthesis tools

Test scenarios

Test cases

Test generation


Test scenarios and test cases l.jpg

General case: application to software product linesTest scenario  test case

Condition : incomplete use case scenarios

parameters

wildcards

lack of certain messages

For realistic systems and product lines

Particular case: Test scenario = test case

Condition : complete use case scenarios

For simple systems

Test scenarios and test cases

Test generation

Need to treat the test scenarios to obtain test cases

Test synthesis tools


Test synthesis using the umlaut tgv toolkit l.jpg
Test synthesis: application to software product linesusing the UMLAUT/TGV toolkit

UMLSpec.

Sequencediagram

?e

!a

!a

LTS

LTS

!d

!d

!c

?b

?e

?b

Test synthesis

!a

Test case(LTS)

!c

?b

Test case(UML)

  • UMLAUT / TGV

    • On the fly test synthesis

  • Generation ofBehavioral Test Patterns

    • positive scenario

    • negative scenarios

    • prefix scenario

Test generation


Test synthesis using behavioral test patterns l.jpg
Test synthesis using application to software product linesbehavioral test patterns

nominal

exceptional

UC1

Use cases

nominal

exceptional

UC2

Selection

Positive scenario

Negative scenarios (optional)

Prefix scenarios (optional)

General

Design

Main classes

Interfaces…

BehavioralTest Patterns

specification

Test cases

synthesis

behTP1

behTP2

Detailed

Design

Evolution

  • UML model:

  • class diagram

  • state machines

  • object diagram

Test generation


Experiments and results l.jpg
Experiments and Results application to software product lines

Experiments with TAS

Requirements

Requirement model

Academical experiments

Test objectives

Applicationto the PL

Test scenarios

Ongoing

integration with the AGATHA tool

Test cases

Experiments and results


Academic experiments l.jpg

3 case studies application to software product lines

FTP client

ATM

Virtual meeting

Code repartition

Code Coverage

Criteria comparison

Academic experiments

Code repartition

Code coverage

for the virtual meeting example

Dead code9%

Robustness code

w.r.t. env

18%

# cov. statements

# test cases

Robustness code

w.r.t. spec

8%

Nominal code

65 %

Code covered with APT + robustness criterion

Experiments and results


Experiments with tas l.jpg
Experiments with TAS application to software product lines

  • Two components of weapon navigation system (Mirage 2000-9 and Rafale)

  • Translation of the requirements from English to RDL

  • Use of the simulator:

    • completion of implicit parts of the requirements

Experiments and results

  • cannot be translated(arithmetic, real-time)

20%

10%

could be translated(limit of the prototype tool)

70%

translated


Application to the product lines context l.jpg
Application to the Product Lines context application to software product lines

  • 3 types of variation points

    • 0 or 1 variant = optional

    • 1 out of n variants = choice (xor)

    • m out of n variants = multiple choice (or)

  • What can vary in the use case model ?

    • use cases

    • parameters

    • contracts

    • scenarios

Experiments and results


Variation in use cases models l.jpg
Variation in use cases models application to software product lines

  • How to represent variation ?

    • Tagging the variant model elements

      • VP_name{variant_list}

    • Examples :

      • UC Record (p:participant, m:meeting) {VP_Recording{true}}

      • UC enter(u:participant;m:meeting)

        pre connected(u) and opened(m)

        pre private(m) implies authorized(u,m){VPMeetingType(private)}

        post entered(u,m)

Experiments and results

Alternative variation point

Multiple variation point


Scenarios in the product lines context l.jpg
Scenarios in the Product Lines context application to software product lines

Test cases

synthesis

behTP1

behTP2

P1

P2

P3

  • Scenarios for product lines:

    • must be generic

      • use of parameters

      • use of wildcards

      • omit certain details

    • necessity of the test synthesis step

Experiments and results


Outline42 l.jpg
Outline application to software product lines

  • Introduction

  • Requirement models

  • Test generation

  • Conclusions and future work

Conclusion


Conclusions and benefits l.jpg

Cutting down the test generation application to software product linescost

Adapted to the product lines context

Compatible with usual industrial practice

Facing thecomplexity (size)of real software

Conclusions and benefits

Requirements

Conclusion

Improvementof therequirements

Requirement model

Using only early requirements

Test objectives

Using detailed requirements

Scenarios

Test scenarios

As soon as a UML model is available

UML model

Test cases


Conclusion l.jpg
Conclusion application to software product lines

  • Testing tools

    • simulator, test generator

  • Industrial interactions

  • Dissemination

    • ASE, ISSRE, ...

Conclusion

Collaboration between THALES, the CEA and the INRIA

European projects


Future work l.jpg
Future work application to software product lines

  • Improving the efficiency of the generated tests

    • new criteria taking into account the constraints of the scenarios

    • test data

  • Other ways to model the variability

  • Extra-functional requirements

    • models ?

    • test generation ?

  • Traceability issues in model-driven approaches

    • traceability models

    • traceability from requirements to tests

Conclusion


Model transformations and traceability l.jpg

All the metamodels are based on the MOF application to software product lines

Objectives:

models built by transformation models

homogeneous management of traceability

Model transformations and traceability

Requirements metamodel

requirement 1.1 "Register a book"

the "book" becomes "registered" after the "librarian" did "register" the "book".

the "book" is "available" after the "librarian" did "register" the "book".

Test objectives

metamodel

[connect(p1), plan(p1,m1)] [connect(p1), plan(p1,m1), open(p1,m1), close(p1,m1)]

UCTS metamodel

System static metamodel

Test scenarios

metamodel

Test cases

metamodel

Configuration metamodel

Conclusion

Enhanced Use case metamodel


Questions l.jpg
Questions application to software product lines

Questions


Slide48 l.jpg
... application to software product lines


ad