CSE 565 Software Verification, Validation, and Testing. A Guest Lecture on TestingBased Software and System Reliability Evaluation (Part 2). Dr. Yinong Chen. Basic Input Domain Models. Examples: MacWilliams73, Brown&Lipow75, Nelson78. .
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Software Verification, Validation, and Testing
A Guest Lecture on
TestingBased Software and SystemReliability Evaluation
(Part 2)
Dr. Yinong Chen
Examples: MacWilliams73, Brown&Lipow75, Nelson78.
Software reliability is defined as the probabilityR(N) = Prob{no of failures over N application runs}
where N is the exposure period whose time unit is the number of application runs.
Assuming that input cases are selected independently, then R(N) can be expressed as R(N) = (R(1))N = RN where, R R(1) is the expected reliability per application run.
Now the question is how to estimate R.
R, the reliability per test run, can be defined by the ratio of the number of test runs in which failures are observed and the total number of test runs when infinite number of different input cases are applied for test runs:
R = 1 – F = 1 –
Because of test time limit only a subset of the entire input domain can be applied to test the program in practice. Thus the reliability per test run, R, is usually estimated by
Test the program 10 000 time (test runs)
Five (5) failures are observed
F = 5 / 10 000 = 1 / 2 000
R = 1  1 / 2 000 = 0.9995
Reliability in N
application
runs are
R(N) = RN
MacWilliams 73 and Brown&Lipow 75
With partition and profiling
MacWilliams 73: the s input cases are selected randomly from the input domain.
Brown&Lipow 75: the input domain is partitioned into m classes. If si input cases are selected from class Ci and fi failures are observed, the reliability can be calculated by
where P(Ci) is a probability function reflecting the input profile in terms of classes.
The input domain is partitioned into 10 subdomains.
ˆ
ˆ
=

=
1 – 0.020938 = 0.979063
R
1
F
Case Study 1Antilock Braking System (ABS)
#include <iostream>
using namespace std;
const float mile_inch = 63360;
const float pi = 3.1416;
float wheel_diameter = 15; // inches
float wheel_sensor() {
float rps;
cout << "get rotations per second: " << endl;
rps = ReadWheelRotationSensor();
return rps;
}
float wheel_velocity(float rps) {
float wv;
wv = (pi * wheel_diameter * rps * 3600)/mile_inch;
return wv;
}
float body_velocity() {
float bv;
cout << "get miles per hour: " << endl;
bv = ReadBodySpeedSensor();
return bv;
}
void error_detection(float wv, float bv) {
if (abs(bv  wv) < 0.01)
cout << "no action" << endl;
else
if (bv > wv)
cout << "reduce brake force!" << endl;
else
cout << "reduce acceleration force!" << endl;
}
void evaluation() {
float rps, wv, bv;
rps = wheel_sensor();
wv = wheel_velocity(rps);
bv = body_velocity();
error_detection (wv, bv);
}
void main() {
for (i = 1, I < 10000, i++)
evaluation();
}
Apply the input domain reliability model
Testing greatest common divisor program
Source:
Y. Chen, W.T. Tsai, Introduction to programming languages: Programming in C, C++, Scheme, Prolog, C#, and SOA, second edition, Kendall/Hunt Publishing Company, 2006, ISBN 0757529747.
Section 1.5.2.
greatest common divisor program
#include <stdio.h>
int gcd (int n0, int m0) { // n0 0, m0 0
int n, m; // n0 or m0 0
n = n0; m = m0;
while (n != 0 && n != m) {
if (n < m) m = m  n;
else n = n  m; }
return m;
}
void main() {
int i, j, k;
scanf("%d\n%d", &i, &j); //input
k = gcd(i, j); // call gcd
printf("%d\n", k); // output
}
Input Output k
(i, j) = (6, 9) 3
(i, j) = (10, 5) 5
(i, j) = (0, 4) 4
(i, j) = (5, 7) 1
(i, j) = (8, 29) 1
The program produces correct outputs for all these different test cases.
Can we claim that the program is correct?
Example: Input Domain Partition Analysis
Based on the analysis, we have a partition of:
i: [0]; [2, 3, 5, 7, 11, …]; [4, 6, 8, 9, …]
j: [0]; [2, 3, 5, 7, 11, …]; [4, 6, 8, 9, …]
The combination of the two inputs generates following cases:
(0, 0): This case is not allowed according to the specification.
(0, 2), (0, 3), (0, 5), (0, 7), (0, 11), …
(0, 4), (0, 6), (0, 8), (0, 9), …
(2, 0), (2 2), (2, 3), (2, 5), (2, 5), …
(2, 4), (2, 6), (2, 8), (2, 9), …
(3, 0), (3, 2), (3, 3), (3, 5), (3, 7), …
. . .
(9, 0), (9, 2), (9, 3), (9, 5), (9, 7), …
Coverage Consideration: Branch Coverage
no
n != 0?
yes
no
n != m?
yes
no
n < m?
yes
m = m  n;
n = n  m;
return m;
Exit
(0, 2),(0, 3),(0, 9),(0, 10):
(2, 2),(3, 3), (9, 9),(10, 10):
(2, 3),(2, 9), (2, 10): ...
(3, 2),(9, 2),10, 2),(10, 3),(10, 9): ...
(2, 0),(3, 0), (9, 0),(10, 0): ...
Testing the Program Using Test Cases
Let's trace the program with (i, j) = (2, 0):
(i, j) = (2, 0)
(20, 0)
(20, 0)
...
A dead loop occurs – a design fault is found
Inputs Output k
(i, j) = (0, 2) 2
(i, j) = (2, 2) 2
(i, j) = (2, 3) 1
(i, j) = (3, 2) 1
(i, j) = (2, 0) ?
SUMMARY SOFTWARE RELIABILITY MODELS
(Software and Hardware)
Markov models are more generic than combinatorial models.
They can handle repairs and much more complex situations.
Assumption:
• Any component may in one the two states: working or failed;
• Probability of state transition depends only on the current state.
ß
Failure rates and repair rates are constants.
ß
Transition probability is proportional to the time that the component stays at a state.
ß
Exponential distribution of the reliability/availability
Construct differential equations
(2)
ß
Solve the equations to obtain the probability in each state
(3)
(4)
ß
The reliability or availability is the sum of the probabilities of working states.
Steps of Applying Markov Models
A system consists of multiple components
ß
Construct state transition diagram
(1)
l
0
1
m
A(t) = p0(t)
Step 1: Construct state transition diagram
Example 1: Simplex system with repair
Module B
Voter
Input
Output
Module C
Step 1: Construct state transition diagram
Example 2: Reliability of TMR system with repair
4a
4b
00
(failed nodes, failed links)
a
10
01
3a+4b
4a+3b
11
20
02
Step 1: Construct state transition diagram
Example 3: A ring system with different node and link failure rates a and b. Assumethat the system fails if any two or more than components failed.
Failed
Step 2: Construct differential equations
= –l·p0 (t) + m·p1 (t)
= l·p0 (t)–m·p1 (t)
A(t) = p0(t)
The question is how to obtain the probability of each state.
p0 (t + Dt) = (1 –l · Dt) · p0 (t) + m · Dt ·p1 (t)
p1 (t + Dt) = l · Dt ·p0 (t) + (1 –m · Dt) · p1 (t)
Solve the differential equations to obtain (p0 (t), p1 (t)).
Step 2: Construct differential equations
= –l·p0 (t) + m·p1 (t)
p0
p1
l m
l m
= l·p0 (t)–m·p1 (t)
=
l
0
1
m
Step 3: Solve differential equations
p0
p1
l m
l m
m
l

l
+
m
p0
(
)
t
(
t
)
=
+
e
=
l
+
m
l
+
m
l
m

l
+
m
p1
(
)
t
(
t
)
=
+
e
l
+
m
l
+
m
l

l
+
m
(
)
t
A
(
t
)
=
+
e
l
+
m
l
+
m
p0
p0
(
(
t
t
)
)
=
=
Step 4: Find the Probabilities of Working States
m
l

l
+
m
p0
(
)
t
(
t
)
=
+
e
l
+
m
l
+
m
l
m

l
+
m
p1
(
)
t
(
t
)
=
+
e
l
+
m
l
+
m
If m = 0, the probability at p0 represents the reliability
m
l

l
+
m
=

l
(
)
t
t
R
(
t
)
=
+
e
e
l
+
m
l
+
m
Step 2: Construct differential equations (Find the pattern)
a12
1
2
a21
a13
a23
a31
a32
a14
a41
3
a25
a52
a34
a35
a53
a43
a45
4
5
a54
p1
P2
p3
p4
p5
dp
(
t
)
aij
=
2
dt
…
dp
(
t
)
5
dt
Step 2: Construct differential equations (Find the pattern)
In general, assume a STD has n states and is fully connected. Any state has n incoming and n outgoing transitions:
aij 0 is the transition rate from state i to j.
For i, j = 1, 2, ..., n, and i ≠ j.
Step 2: Construct differential equations (Find the pattern)
where
The probability in state j at t + Dt
= the probability in state j at t+ incoming prob – outgoing prob
Math manipulation: Divide Dt on both sides, let Dt 0
Let Dt 0
dp
(
t
)
(
)
(
)
(
)
n
n
n
j
=
a

×
a
=
a

×
b
å
p
(
t
)
p
(
t
)
å
å
p
(
t
)
p
(
t
)
p
(
t
)
i
ij
j
ji
i
ij
j
j
dt
j
1
1
1
=
=
=
i
i
i
¹
¹
¹
i
j
i
j
i
j
(
)
(
)
n
n
a

a
D
å
p
(
t
)
å
p
(
t
)
t
+
D

D
D
D
p
(
t
t
)
t
t
t
i
ij
j
ji
j
1
1
=
=
i
i
where
¹
¹
i
j
i
j
=
Step 2: Construct differential equations (More detail of the previous slide)
p1
p2
p3
…
pn
b1
a21
a31
a41
an1
a12
b2
a32
a42
an2
dp
(
t
)
=
2
a13
a23
b3
a43
an3
dt
…
dp
(
t
)
n
bn
a1n
a2n
a3n
a4n
dt
where
Step 2: Construct differential equations (found the pattern)
4b
00
10
01
3a+4b
4a+3b
11

4
(
a
+
b
)
4
b
4
a
0
0
æ
0
ö
ç
÷

(
4
a
+
3
b
)
0
4
a
+
0
3
b
ç
÷
T
=
0

(
3
a
+
4
b
)
÷
ç
0
ç
÷
0
0
è
3
a
+
4
b
ø
Example 2
1
2
3
4
R(t) = p1(t) + p2(t) + p3(t)