model program based black box testing l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Model Program Based Black-Box Testing PowerPoint Presentation
Download Presentation
Model Program Based Black-Box Testing

Loading in 2 Seconds...

play fullscreen
1 / 24

Model Program Based Black-Box Testing - PowerPoint PPT Presentation


  • 171 Views
  • Uploaded on

Model Program Based Black-Box Testing. Margus Veanes Foundations of Software Engineering Microsoft Research, Redmond. System-level black-box testing. Requires model of system behavior Behavior is often reactive/nondeterministic Implementation is multi-threaded or distributed

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Model Program Based Black-Box Testing' - hayward


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
model program based black box testing

Model Program Based Black-Box Testing

Margus Veanes

Foundations of Software Engineering Microsoft Research, Redmond

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

system level black box testing
System-level black-box testing
  • Requires model of system behavior
  • Behavior is often reactive/nondeterministic
    • Implementation is multi-threaded or distributed
    • Thread scheduling hard to control
  • State space is typically infinite
    • Objects, unbounded value types
  • Traditional FSM-based testing does often not scale

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

model based testing
Model based testing

Model

Provides expected

results for

Generates

PassFail

User

Info

Test Cases

Test Oracle

Provides

actual results for

Are run by

Implementation

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

our approach
Our approach
  • Behavior is described by model programs
    • Written in Spec# or AsmL
    • Describe reactive system behavior as model automata
    • Use alternating simulation for conformance
  • Model exploration, validation and model-based testing are provided by the Spec Explorer tool developed at MSR
    • Supports scenario control
    • Provides offline as well as online testing
    • Views test cases as game strategies
    • Checks conformance violations

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

the big picture
The big picture

Modeling

Model program

Scenario control

Online testing

Constrained model

Test generation

Test cases

Test execution

Coding

Implementation

Pass/Fail

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

conformance checking component view

IUT

Conformance checkingcomponent view

Controllable actions

Controllable actions

Constrained

model

Refinement

checker

IUT wrapper

Object

mapping

Observableactions

Observable actions

Model

c1

d1

Pass/Fail

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

example chat

Constrained

chat model

IUT wrapper

Controllable

actions:

Login

Send

IUT

Refinement

checker

Object

mapping

b.Send(“hi”)

Chat server

b

Bob

m

Mary

m.Send(“hello”)

b.Send(“bye”)

b.Rcv(“hello”)

m.Rcv(“hi”)

Observable actions:

Receive

b.Rcv(“hello”)

m.Rcv(“hi”)

m.Rcv(“bye”)

m.Rcv(“bye”)

b.Rcv(“hello”)

Example: Chat

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

refinement relation

b.Send(“hi”)

b.Send(“hi”)

m.Send(“hello”)

m.Send(“hello”)

m.Receive(b,“hi”)

b.Receive(m,“hello”)

m.Receive(b,“hi”)

b.Receive(m,“hello”)

b.Receive(m,“hello”)

Refinement relation
  • Conformance between the model and the system under test is alternating refinement

Spec

System

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

example valid chat scenario

B.Send(“bye”)

B.Send(“hi”)

M.Send(“hello”)

M.Receive(B,“hi”)

M.Receive(B,“bye”)

B.Receive(M,“hello”)

Example: Valid chat scenario

Service should guarantee local consistency of message delivery (FIFO wrt sender)

Send is a controllable action

Receive is an observable action

Bob

Server

Mary

Time

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

model programs and automata
Model programs and automata
  • States are mappings of variables to values
    • first-order structures
  • Initial state is given by initial assignment to variables
  • Actions are defined by method invocations
    • Actions are either controllable or observable
  • An action m(v) is enabled is a state Sif the precondition of m(v) is true in S
  • An action m(v)transitions from state S1 to stateS2 if m(v) is enabled in S1and its execution in S1 yields S2

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

model program chat service
Model program : Chat service

State:

class Client {

bool entered = false;

Map<Client, Seq<string>> unreceivedMsgs = Map{};

}

Actions:

class Client {

void Send(string message)

requires entered; {

foreach (Client c inenumof(Client), c != this, c.entered)

c.unreceivedMsgs[this] += Seq{message};}

void Receive(Client sender, string message)

requires sender != this &&

unreceivedMsgs[sender].Length > 0 &&

unreceivedMsgs[sender].Head == message;

{

unreceivedMsgs[sender] = unreceivedMsgs[sender].Tail;}

}

Controllable:

Observable:

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

online testing
Online testing
  • For large reactive systems deriving an exhaustive test-suite is infeasible
  • Merge test generation and test execution into one process
  • Online testing is a form of model-based stress testing
  • Purely random exploration is wasteful
    • Use scenario control to direct online testing

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

exploration scenario control
Exploration & Scenario control
  • Exploration is the process of unfolding a rule-based model program into a transition system
    • Actions move the system from state to state
    • State comes from variables containing values
    • Compound data types are possible (sets, maps, sequences, etc…)
    • Objects are identities
  • Scenario control
    • Imposes limits on exploration
      • Using filters, state groupings, scenario actions , …

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

example demo exploration of a coffee machine
Example Demo: exploration of a coffee machine

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

scenario control
Scenario control
  • In general the transition system is infinite, but we can impose limits
  • Goal
    • Create a state space of manageable size that satisfies a given testing goal
  • Two main tasks
    • Restrict action parameter domains to interesting values
    • Restrict the state space to interesting states
  • Note: The two tasks are not necessarily independent!

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

scenario control cont
Scenario control (cont.)
  • The following techniques are used in Spec Explorer
    • State filters
    • Stopping conditions
    • State groupings
    • Search order
    • Enabling conditions on actions
    • Scenario actions
  • Usually a combination of all (or some) of the techniques is needed to achieve desired effect

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

example of scenario actions chat initialization
Example of scenario actions: Chat initialization
  • Use a parameterized scenario action Start.
    • Invoking Start(n) produces a sequence of nCreate() invocations followed by nEnter() invocations

[Action(Kind=Scenario)]

void Start(int nrOfMembers)

requires enumof(Client) == Set{};

{

Seq<Client> clients = Seq{};

// 1-- The given number of

// clients are created

for (int i = 0; i < nrOfMembers; i++)

clients += Seq{Create()};

// 2-- all clients enter the session

for (int i = 0; i < nrOfMembers; i++)

clients[i].Enter();

}

Create()/c0;

Create()/c1;

c0.Enter();

c1.Enter();

Start(2)

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

test generation
Test generation
  • The result of exploration is a finite transition system with observable and controllable actions : a model automaton A.
  • Testing can be viewed as a game between two players
    • Testing tool (TT)
      • Making controllable transitions
    • Implementation under test (IUT)
      • Making observable transitions
  • The “rules of the game” are dictated by A
  • Tests are generated from A. Test cases are strategies
    • A strategy tells what move TT should make in a state where its actions are enabled
    • The purpose of a strategy may be to achieve transition coverage or to reach a certain goal state

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

demo chat
Demo: chat

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

experiences
Experiences
  • Most tool features driven by demands of internal users at MS (mostly testers)
  • Models help discover more bugs during modeling (design bugs) than testing
    • Testers do not get credit for finding those bugs !!!
  • During testing models help discover deep system-level bugs where manual test scenarios don’t
    • Such bugs are hard to understand and fix

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

experiences cont
Experiences (cont.)
  • Bugs appear in both models and implementations (the ratio is roughly 50-50)
  • Code coverage is a poor measure for testing concurrent software, often a single execution thread provides the same coverage.
  • New versions of implementation usually require only local changes to models, whereas manual tests often need to be rewritten completely
  • The tool is built on .NET but has been used to test distributed C++ applications. Most test harnesses at MS are built in managed code.

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

outlook
Outlook
  • User wish list:
    • Language independence for modeling
      • Use C#, VB, …
    • Better control over exploration
      • Scenario-oriented models
    • Model composition
      • Model features separately and compose the models
    • Model validation
      • Validate properties on models

Drives the focus of current research at FSE

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

related foundational work
Related foundational work
  • Abstract state machines
  • Interface automata
  • Input-Output COnformance (IOCO) theory
  • Markov Decision Process (MDP) theory

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

thank you
Thank you!
  • FSE web site:
    • http://research.microsoft.com/foundations
  • Downloading Spec Explorer:
    • http://research.microsoft.com/specexplorer

Questions?

Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands