slide1
Download
Skip this Video
Download Presentation
A Guest Lecture on Testing-Based Software and System Reliability Evaluation (Part 2)

Loading in 2 Seconds...

play fullscreen
1 / 38

Example - PowerPoint PPT Presentation


  • 200 Views
  • Uploaded on

CSE 565 Software Verification, Validation, and Testing. A Guest Lecture on Testing-Based Software and System Reliability Evaluation (Part 2). Dr. Yinong Chen. Basic Input Domain Models. Examples: MacWilliams73, Brown&Lipow75, Nelson78. .

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Example' - Sharon_Dale


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

CSE 565

Software Verification, Validation, and Testing

A Guest Lecture on

Testing-Based Software and SystemReliability Evaluation

(Part 2)

Dr. Yinong Chen

slide2

Basic Input Domain Models

Examples: MacWilliams73, Brown&Lipow75, Nelson78.

Software reliability is defined as the probabilityR(N) = Prob{no of failures over N application runs}

where N is the exposure period whose time unit is the number of application runs.

Assuming that input cases are selected independently, then R(N) can be expressed as R(N) = (R(1))N = RN where, R R(1) is the expected reliability per application run.

Now the question is how to estimate R.

slide3

Estimate R

R, the reliability per test run, can be defined by the ratio of the number of test runs in which failures are observed and the total number of test runs when infinite number of different input cases are applied for test runs:

R = 1 – F = 1 –

Because of test time limit only a subset of the entire input domain can be applied to test the program in practice. Thus the reliability per test run, R, is usually estimated by

example
Example

Test the program 10 000 time (test runs)

Five (5) failures are observed

F = 5 / 10 000 = 1 / 2 000

R = 1 - 1 / 2 000 = 0.9995

Reliability in N

application

runs are

R(N) = RN

slide5

MacWilliams 73 and Brown&Lipow 75

With partition and profiling

MacWilliams 73: the s input cases are selected randomly from the input domain.

Brown&Lipow 75: the input domain is partitioned into m classes. If si input cases are selected from class Ci and fi failures are observed, the reliability can be calculated by

where P(Ci) is a probability function reflecting the input profile in terms of classes.

example6
Example

The input domain is partitioned into 10 sub-domains.

ˆ

ˆ

=

-

=

1 – 0.020938 = 0.979063

R

1

F

slide7

Case Study 1Anti-lock Braking System (ABS)

  • Requirement:
  • To obtain the maximum braking effect
  • Algorithm:
  • Define (or measure) the wheel diameter;
  • Measure the wheel rotations per seconds rps;
  • Compute the wheel velocity wv;
  • Measure the body velocity bv;
  • Error detection and action:
  • if (bv > wv), reduce braking force
  • else if (bv < wv), reduce acceleration force
  • else “no action”
slide8

Sample Code in C++

#include <iostream>

using namespace std;

const float mile_inch = 63360;

const float pi = 3.1416;

float wheel_diameter = 15; // inches

float wheel_sensor() {

float rps;

cout << "get rotations per second: " << endl;

rps = ReadWheelRotationSensor();

return rps;

}

slide9

float wheel_velocity(float rps) {

float wv;

wv = (pi * wheel_diameter * rps * 3600)/mile_inch;

return wv;

}

float body_velocity() {

float bv;

cout << "get miles per hour: " << endl;

bv = ReadBodySpeedSensor();

return bv;

}

slide10

void error_detection(float wv, float bv) {

if (abs(bv - wv) < 0.01)

cout << "no action" << endl;

else

if (bv > wv)

cout << "reduce brake force!" << endl;

else

cout << "reduce acceleration force!" << endl;

}

void evaluation() {

float rps, wv, bv;

rps = wheel_sensor();

wv = wheel_velocity(rps);

bv = body_velocity();

error_detection (wv, bv);

}

void main() {

for (i = 1, I < 10000, i++)

evaluation();

}

slide11

Apply the input domain reliability model

  • Write the random function that simulates: ReadWheelRotationSensor();
  • Write the random function that simulates: ReadBodySpeedSensor();
  • Test the program and collect date
  • Apply the input domain model to evaluate the reliability of the program
slide12

Case Study:

Testing greatest common divisor program

Source:

Y. Chen, W.T. Tsai, Introduction to programming languages: Programming in C, C++, Scheme, Prolog, C#, and SOA, second edition, Kendall/Hunt Publishing Company, 2006, ISBN 0-7575-2974-7.

Section 1.5.2.

slide13

greatest common divisor program

#include <stdio.h>

int gcd (int n0, int m0) { // n0  0, m0  0

int n, m; // n0 or m0  0

n = n0; m = m0;

while (n != 0 && n != m) {

if (n < m) m = m - n;

else n = n - m; }

return m;

}

void main() {

int i, j, k;

scanf("%d\n%d", &i, &j); //input

k = gcd(i, j); // call gcd

printf("%d\n", k); // output

}

slide14

Random Test Case Generation

Input Output k

(i, j) = (6, 9) 3

(i, j) = (10, 5) 5

(i, j) = (0, 4) 4

(i, j) = (5, 7) 1

(i, j) = (8, 29) 1

The program produces correct outputs for all these different test cases.

Can we claim that the program is correct?

slide15

Example: Input Domain Partition Analysis

  • Input Domain Analysis:
  • The program takes two integers as input.
  • The branches of the program are controlled by the relative values of the two integers.
  • Input Domain Partitioning:
  • Partition each integer input into three groups: < 0, = 0, and > 0.
  • For this program, < 0 is not allowed. The group that has only one value is called boundary value.
  • Considering the semantics of the program, prime numbers and nonprime numbers play a role in the program. Thus, the positive integers are further divided into prime and nonprime numbers.
slide16

Input Domain Partition

Based on the analysis, we have a partition of:

i: [0]; [2, 3, 5, 7, 11, …]; [4, 6, 8, 9, …]

j: [0]; [2, 3, 5, 7, 11, …]; [4, 6, 8, 9, …]

The combination of the two inputs generates following cases:

(0, 0): This case is not allowed according to the specification.

(0, 2), (0, 3), (0, 5), (0, 7), (0, 11), …

(0, 4), (0, 6), (0, 8), (0, 9), …

(2, 0), (2 2), (2, 3), (2, 5), (2, 5), …

(2, 4), (2, 6), (2, 8), (2, 9), …

(3, 0), (3, 2), (3, 3), (3, 5), (3, 7), …

. . .

(9, 0), (9, 2), (9, 3), (9, 5), (9, 7), …

slide17

Coverage Consideration: Branch Coverage

no

n != 0?

yes

no

n != m?

yes

no

n < m?

yes

m = m - n;

n = n - m;

return m;

Exit

(0, 2),(0, 3),(0, 9),(0, 10): 

(2, 2),(3, 3), (9, 9),(10, 10): 

(2, 3),(2, 9), (2, 10): ...

(3, 2),(9, 2),10, 2),(10, 3),(10, 9): ...

(2, 0),(3, 0), (9, 0),(10, 0): ...

slide18

Testing the Program Using Test Cases

Let\'s trace the program with (i, j) = (2, 0):

(i, j) = (2, 0)

(2-0, 0)

(2-0, 0)

...

A dead loop occurs – a design fault is found

Inputs Output k

(i, j) = (0, 2) 2

(i, j) = (2, 2) 2

(i, j) = (2, 3) 1

(i, j) = (3, 2) 1

(i, j) = (2, 0) ?

apply the partition based input domain reliability model
Apply the partition-based input domain reliability model
  • Modify the main program, so that it systematically generate (large number of) input cases from different sub-domain;
  • Test the gcd function and collect data
  • Apply the partition-based input domain model to evaluate the reliability of the program
slide20

SUMMARY SOFTWARE RELIABILITY MODELS

  • Basic concepts and terminology
  • Faults and failure rates
  • Reliability R(t) and availability A(t)
  • Software reliability models
    • Classifications
    • Time-domain models
    • Fault count model
    • Input-domain models
    • Fault seeding model
  • Sample code of real-time ABS software
    • Sample code of real-time ABS software
    • Partition-based input domain testing
slide21

Modeling Complex Systems

(Software and Hardware)

  • A large system can be decomposed into smaller components.
  • Evaluate the reliability of the components;
  • Evaluate the reliability of the system based on known component reliabilities
    • Combinatorial Models
    • Markov Models
slide22

Markov Models

Markov models are more generic than combinatorial models.

They can handle repairs and much more complex situations.

Assumption:

• Any component may in one the two states: working or failed;

• Probability of state transition depends only on the current state.

ß

Failure rates and repair rates are constants.

ß

Transition probability is proportional to the time that the component stays at a state.

ß

Exponential distribution of the reliability/availability

slide23

ß

Construct differential equations

(2)

ß

Solve the equations to obtain the probability in each state

(3)

(4)

ß

The reliability or availability is the sum of the probabilities of working states.

Steps of Applying Markov Models

A system consists of multiple components

ß

Construct state transition diagram

(1)

slide24

System

l

0

1

m

A(t) = p0(t)

Step 1: Construct state transition diagram

Example 1: Simplex system with repair

slide25

Module A

Module B

Voter

Input

Output

Module C

Step 1: Construct state transition diagram

Example 2: Reliability of TMR system with repair

slide26

b

4a

4b

00

(failed nodes, failed links)

a

10

01

3a+4b

4a+3b

11

20

02

Step 1: Construct state transition diagram

Example 3: A ring system with different node and link failure rates a and b. Assumethat the system fails if any two or more than components failed.

Failed

slide27

Step 2: Construct differential equations

= –l·p0 (t) + m·p1 (t)

= l·p0 (t)–m·p1 (t)

A(t) = p0(t)

The question is how to obtain the probability of each state.

p0 (t + Dt) = (1 –l · Dt)  · p0 (t) + m · Dt ·p1 (t)

p1 (t + Dt) = l · Dt ·p0 (t) + (1 –m · Dt)  · p1 (t)

Solve the differential equations to obtain (p0 (t), p1 (t)).

slide28

Step 2: Construct differential equations

= –l·p0 (t) + m·p1 (t)

p0

p1

-l m

l -m

= l·p0 (t)–m·p1 (t)

=

l

0

1

m

slide29

Step 3: Solve differential equations

p0

p1

-l m

l -m

m

l

-

l

+

m

p0

(

)

t

(

t

)

=

+

e

=

l

+

m

l

+

m

l

m

-

l

+

m

p1

(

)

t

(

t

)

=

+

e

l

+

m

l

+

m

  • There are many different ways to solve differential equations
  • LaPlace Transformation
  • Tools like MatLab or Mathematica
slide30

m

l

-

l

+

m

(

)

t

A

(

t

)

=

+

e

l

+

m

l

+

m

p0

p0

(

(

t

t

)

)

=

=

Step 4: Find the Probabilities of Working States

m

l

-

l

+

m

p0

(

)

t

(

t

)

=

+

e

l

+

m

l

+

m

l

m

-

l

+

m

p1

(

)

t

(

t

)

=

+

e

l

+

m

l

+

m

If m = 0, the probability at p0 represents the reliability

m

l

-

l

+

m

=

-

l

(

)

t

t

R

(

t

)

=

+

e

e

l

+

m

l

+

m

slide31

Step 2: Construct differential equations (Find the pattern)

a12

1

2

a21

a13

a23

a31

a32

a14

a41

3

a25

a52

a34

a35

a53

a43

a45

4

5

a54

p1

P2

p3

p4

p5

dp

(

t

)

aij

=

2

dt

dp

(

t

)

5

dt

slide32

Step 2: Construct differential equations (Find the pattern)

In general, assume a STD has n states and is fully connected. Any state has n incoming and n outgoing transitions:

aij 0 is the transition rate from state i to j.

For i, j = 1, 2, ..., n, and i ≠ j.

slide33

Step 2: Construct differential equations (Find the pattern)

where

The probability in state j at t + Dt

= the probability in state j at t+ incoming prob – outgoing prob

Math manipulation: Divide Dt on both sides, let Dt 0

slide34

Let Dt 0

dp

(

t

)

(

)

(

)

(

)

n

n

n

j

=

a

-

×

a

=

a

-

×

b

å

p

(

t

)

p

(

t

)

å

å

p

(

t

)

p

(

t

)

p

(

t

)

i

ij

j

ji

i

ij

j

j

dt

j

1

1

1

=

=

=

i

i

i

¹

¹

¹

i

j

i

j

i

j

(

)

(

)

n

n

a

-

a

D

å

p

(

t

)

å

p

(

t

)

t

+

D

-

D

D

D

p

(

t

t

)

t

t

t

i

ij

j

ji

j

1

1

=

=

i

i

where

¹

¹

i

j

i

j

=

Step 2: Construct differential equations (More detail of the previous slide)

slide35

p1

p2

p3

pn

b1

a21

a31

a41

an1

a12

b2

a32

a42

an2

dp

(

t

)

=

2

a13

a23

b3

a43

an3

dt

dp

(

t

)

n

bn

a1n

a2n

a3n

a4n

dt

where

Step 2: Construct differential equations (found the pattern)

slide36

3

l

Example 1: Apply the Pattern

-3

l

m

0

æ

ö

ç

T

=

-(2l+m)

÷

0

ç

÷

è

0

ø

0

2l

R(t) = p1(t) + p2(t)

slide37

4a

4b

00

10

01

3a+4b

4a+3b

11

-

4

(

a

+

b

)

4

b

4

a

0

0

æ

0

ö

ç

÷

-

(

4

a

+

3

b

)

0

4

a

+

0

3

b

ç

÷

T

=

0

-

(

3

a

+

4

b

)

÷

ç

0

ç

÷

0

0

è

3

a

+

4

b

ø

Example 2

1

2

3

4

R(t) = p1(t) + p2(t) + p3(t)

summary
SUMMARY
  • Basic concepts of reliability and reliability modeling
  • Hardware reliability models
  • Software reliability models
  • System reliability models consisting of multiple components
    • Combinatorial models
    • Markov models
ad