Download Presentation
A Guest Lecture on Testing-Based Software and System Reliability Evaluation (Part 2)

Loading in 2 Seconds...

1 / 38

# Example - PowerPoint PPT Presentation

CSE 565 Software Verification, Validation, and Testing. A Guest Lecture on Testing-Based Software and System Reliability Evaluation (Part 2). Dr. Yinong Chen. Basic Input Domain Models. Examples: MacWilliams73, Brown&amp;Lipow75, Nelson78. .

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

## PowerPoint Slideshow about 'Example' - Sharon_Dale

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

CSE 565

Software Verification, Validation, and Testing

A Guest Lecture on

Testing-Based Software and SystemReliability Evaluation

(Part 2)

Dr. Yinong Chen

Basic Input Domain Models

Examples: MacWilliams73, Brown&Lipow75, Nelson78.

Software reliability is defined as the probabilityR(N) = Prob{no of failures over N application runs}

where N is the exposure period whose time unit is the number of application runs.

Assuming that input cases are selected independently, then R(N) can be expressed as R(N) = (R(1))N = RN where, R R(1) is the expected reliability per application run.

Now the question is how to estimate R.

Estimate R

R, the reliability per test run, can be defined by the ratio of the number of test runs in which failures are observed and the total number of test runs when infinite number of different input cases are applied for test runs:

R = 1 – F = 1 –

Because of test time limit only a subset of the entire input domain can be applied to test the program in practice. Thus the reliability per test run, R, is usually estimated by

Example

Test the program 10 000 time (test runs)

Five (5) failures are observed

F = 5 / 10 000 = 1 / 2 000

R = 1 - 1 / 2 000 = 0.9995

Reliability in N

application

runs are

R(N) = RN

MacWilliams 73 and Brown&Lipow 75

With partition and profiling

MacWilliams 73: the s input cases are selected randomly from the input domain.

Brown&Lipow 75: the input domain is partitioned into m classes. If si input cases are selected from class Ci and fi failures are observed, the reliability can be calculated by

where P(Ci) is a probability function reflecting the input profile in terms of classes.

Example

The input domain is partitioned into 10 sub-domains.

ˆ

ˆ

=

-

=

1 – 0.020938 = 0.979063

R

1

F

Case Study 1Anti-lock Braking System (ABS)

• Requirement:
• To obtain the maximum braking effect
• Algorithm:
• Define (or measure) the wheel diameter;
• Measure the wheel rotations per seconds rps;
• Compute the wheel velocity wv;
• Measure the body velocity bv;
• Error detection and action:
• if (bv > wv), reduce braking force
• else if (bv < wv), reduce acceleration force
• else “no action”

Sample Code in C++

#include <iostream>

using namespace std;

const float mile_inch = 63360;

const float pi = 3.1416;

float wheel_diameter = 15; // inches

float wheel_sensor() {

float rps;

cout << "get rotations per second: " << endl;

rps = ReadWheelRotationSensor();

return rps;

}

float wheel_velocity(float rps) {

float wv;

wv = (pi * wheel_diameter * rps * 3600)/mile_inch;

return wv;

}

float body_velocity() {

float bv;

cout << "get miles per hour: " << endl;

bv = ReadBodySpeedSensor();

return bv;

}

void error_detection(float wv, float bv) {

if (abs(bv - wv) < 0.01)

cout << "no action" << endl;

else

if (bv > wv)

cout << "reduce brake force!" << endl;

else

cout << "reduce acceleration force!" << endl;

}

void evaluation() {

float rps, wv, bv;

rps = wheel_sensor();

wv = wheel_velocity(rps);

bv = body_velocity();

error_detection (wv, bv);

}

void main() {

for (i = 1, I < 10000, i++)

evaluation();

}

Apply the input domain reliability model

• Write the random function that simulates: ReadWheelRotationSensor();
• Write the random function that simulates: ReadBodySpeedSensor();
• Test the program and collect date
• Apply the input domain model to evaluate the reliability of the program

Case Study:

Testing greatest common divisor program

Source:

Y. Chen, W.T. Tsai, Introduction to programming languages: Programming in C, C++, Scheme, Prolog, C#, and SOA, second edition, Kendall/Hunt Publishing Company, 2006, ISBN 0-7575-2974-7.

Section 1.5.2.

greatest common divisor program

#include <stdio.h>

int gcd (int n0, int m0) { // n0  0, m0  0

int n, m; // n0 or m0  0

n = n0; m = m0;

while (n != 0 && n != m) {

if (n < m) m = m - n;

else n = n - m; }

return m;

}

void main() {

int i, j, k;

scanf("%d\n%d", &i, &j); //input

k = gcd(i, j); // call gcd

printf("%d\n", k); // output

}

Random Test Case Generation

Input Output k

(i, j) = (6, 9) 3

(i, j) = (10, 5) 5

(i, j) = (0, 4) 4

(i, j) = (5, 7) 1

(i, j) = (8, 29) 1

The program produces correct outputs for all these different test cases.

Can we claim that the program is correct?

Example: Input Domain Partition Analysis

• Input Domain Analysis:
• The program takes two integers as input.
• The branches of the program are controlled by the relative values of the two integers.
• Input Domain Partitioning:
• Partition each integer input into three groups: < 0, = 0, and > 0.
• For this program, < 0 is not allowed. The group that has only one value is called boundary value.
• Considering the semantics of the program, prime numbers and nonprime numbers play a role in the program. Thus, the positive integers are further divided into prime and nonprime numbers.

Input Domain Partition

Based on the analysis, we have a partition of:

i: [0]; [2, 3, 5, 7, 11, …]; [4, 6, 8, 9, …]

j: [0]; [2, 3, 5, 7, 11, …]; [4, 6, 8, 9, …]

The combination of the two inputs generates following cases:

(0, 0): This case is not allowed according to the specification.

(0, 2), (0, 3), (0, 5), (0, 7), (0, 11), …

(0, 4), (0, 6), (0, 8), (0, 9), …

(2, 0), (2 2), (2, 3), (2, 5), (2, 5), …

(2, 4), (2, 6), (2, 8), (2, 9), …

(3, 0), (3, 2), (3, 3), (3, 5), (3, 7), …

. . .

(9, 0), (9, 2), (9, 3), (9, 5), (9, 7), …

Coverage Consideration: Branch Coverage

no

n != 0?

yes

no

n != m?

yes

no

n < m?

yes

m = m - n;

n = n - m;

return m;

Exit

(0, 2),(0, 3),(0, 9),(0, 10): 

(2, 2),(3, 3), (9, 9),(10, 10): 

(2, 3),(2, 9), (2, 10): ...

(3, 2),(9, 2),10, 2),(10, 3),(10, 9): ...

(2, 0),(3, 0), (9, 0),(10, 0): ...

Testing the Program Using Test Cases

Let\'s trace the program with (i, j) = (2, 0):

(i, j) = (2, 0)

(2-0, 0)

(2-0, 0)

...

A dead loop occurs – a design fault is found

Inputs Output k

(i, j) = (0, 2) 2

(i, j) = (2, 2) 2

(i, j) = (2, 3) 1

(i, j) = (3, 2) 1

(i, j) = (2, 0) ?

Apply the partition-based input domain reliability model
• Modify the main program, so that it systematically generate (large number of) input cases from different sub-domain;
• Test the gcd function and collect data
• Apply the partition-based input domain model to evaluate the reliability of the program

SUMMARY SOFTWARE RELIABILITY MODELS

• Basic concepts and terminology
• Faults and failure rates
• Reliability R(t) and availability A(t)
• Software reliability models
• Classifications
• Time-domain models
• Fault count model
• Input-domain models
• Fault seeding model
• Sample code of real-time ABS software
• Sample code of real-time ABS software
• Partition-based input domain testing

Modeling Complex Systems

(Software and Hardware)

• A large system can be decomposed into smaller components.
• Evaluate the reliability of the components;
• Evaluate the reliability of the system based on known component reliabilities
• Combinatorial Models
• Markov Models

Markov Models

Markov models are more generic than combinatorial models.

They can handle repairs and much more complex situations.

Assumption:

• Any component may in one the two states: working or failed;

• Probability of state transition depends only on the current state.

ß

Failure rates and repair rates are constants.

ß

Transition probability is proportional to the time that the component stays at a state.

ß

Exponential distribution of the reliability/availability

ß

Construct differential equations

(2)

ß

Solve the equations to obtain the probability in each state

(3)

(4)

ß

The reliability or availability is the sum of the probabilities of working states.

Steps of Applying Markov Models

A system consists of multiple components

ß

Construct state transition diagram

(1)

System

l

0

1

m

A(t) = p0(t)

Step 1: Construct state transition diagram

Example 1: Simplex system with repair

Module A

Module B

Voter

Input

Output

Module C

Step 1: Construct state transition diagram

Example 2: Reliability of TMR system with repair

b

4a

4b

00

(failed nodes, failed links)

a

10

01

3a+4b

4a+3b

11

20

02

Step 1: Construct state transition diagram

Example 3: A ring system with different node and link failure rates a and b. Assumethat the system fails if any two or more than components failed.

Failed

Step 2: Construct differential equations

= –l·p0 (t) + m·p1 (t)

= l·p0 (t)–m·p1 (t)

A(t) = p0(t)

The question is how to obtain the probability of each state.

p0 (t + Dt) = (1 –l · Dt)  · p0 (t) + m · Dt ·p1 (t)

p1 (t + Dt) = l · Dt ·p0 (t) + (1 –m · Dt)  · p1 (t)

Solve the differential equations to obtain (p0 (t), p1 (t)).

Step 2: Construct differential equations

= –l·p0 (t) + m·p1 (t)

p0

p1

-l m

l -m

= l·p0 (t)–m·p1 (t)

=

l

0

1

m

Step 3: Solve differential equations

p0

p1

-l m

l -m

m

l

-

l

+

m

p0

(

)

t

(

t

)

=

+

e

=

l

+

m

l

+

m

l

m

-

l

+

m

p1

(

)

t

(

t

)

=

+

e

l

+

m

l

+

m

• There are many different ways to solve differential equations
• LaPlace Transformation
• Tools like MatLab or Mathematica

m

l

-

l

+

m

(

)

t

A

(

t

)

=

+

e

l

+

m

l

+

m

p0

p0

(

(

t

t

)

)

=

=

Step 4: Find the Probabilities of Working States

m

l

-

l

+

m

p0

(

)

t

(

t

)

=

+

e

l

+

m

l

+

m

l

m

-

l

+

m

p1

(

)

t

(

t

)

=

+

e

l

+

m

l

+

m

If m = 0, the probability at p0 represents the reliability

m

l

-

l

+

m

=

-

l

(

)

t

t

R

(

t

)

=

+

e

e

l

+

m

l

+

m

Step 2: Construct differential equations (Find the pattern)

a12

1

2

a21

a13

a23

a31

a32

a14

a41

3

a25

a52

a34

a35

a53

a43

a45

4

5

a54

p1

P2

p3

p4

p5

dp

(

t

)

aij

=

2

dt

dp

(

t

)

5

dt

Step 2: Construct differential equations (Find the pattern)

In general, assume a STD has n states and is fully connected. Any state has n incoming and n outgoing transitions:

aij 0 is the transition rate from state i to j.

For i, j = 1, 2, ..., n, and i ≠ j.

Step 2: Construct differential equations (Find the pattern)

where

The probability in state j at t + Dt

= the probability in state j at t+ incoming prob – outgoing prob

Math manipulation: Divide Dt on both sides, let Dt 0

Let Dt 0

dp

(

t

)

(

)

(

)

(

)

n

n

n

j

=

a

-

×

a

=

a

-

×

b

å

p

(

t

)

p

(

t

)

å

å

p

(

t

)

p

(

t

)

p

(

t

)

i

ij

j

ji

i

ij

j

j

dt

j

1

1

1

=

=

=

i

i

i

¹

¹

¹

i

j

i

j

i

j

(

)

(

)

n

n

a

-

a

D

å

p

(

t

)

å

p

(

t

)

t

+

D

-

D

D

D

p

(

t

t

)

t

t

t

i

ij

j

ji

j

1

1

=

=

i

i

where

¹

¹

i

j

i

j

=

Step 2: Construct differential equations (More detail of the previous slide)

p1

p2

p3

pn

b1

a21

a31

a41

an1

a12

b2

a32

a42

an2

dp

(

t

)

=

2

a13

a23

b3

a43

an3

dt

dp

(

t

)

n

bn

a1n

a2n

a3n

a4n

dt

where

Step 2: Construct differential equations (found the pattern)

3

l

Example 1: Apply the Pattern

-3

l

m

0

æ

ö

ç

T

=

-(2l+m)

÷

0

ç

÷

è

0

ø

0

2l

R(t) = p1(t) + p2(t)

4a

4b

00

10

01

3a+4b

4a+3b

11

-

4

(

a

+

b

)

4

b

4

a

0

0

æ

0

ö

ç

÷

-

(

4

a

+

3

b

)

0

4

a

+

0

3

b

ç

÷

T

=

0

-

(

3

a

+

4

b

)

÷

ç

0

ç

÷

0

0

è

3

a

+

4

b

ø

Example 2

1

2

3

4

R(t) = p1(t) + p2(t) + p3(t)

SUMMARY
• Basic concepts of reliability and reliability modeling
• Hardware reliability models
• Software reliability models
• System reliability models consisting of multiple components
• Combinatorial models
• Markov models