A Guest Lecture on Testing-Based Software and System Reliability Evaluation (Part 2) - PowerPoint PPT Presentation

Slide1 l.jpg
Download
1 / 38

CSE 565 Software Verification, Validation, and Testing. A Guest Lecture on Testing-Based Software and System Reliability Evaluation (Part 2). Dr. Yinong Chen. Basic Input Domain Models. Examples: MacWilliams73, Brown&Lipow75, Nelson78. .

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

A Guest Lecture on Testing-Based Software and System Reliability Evaluation (Part 2)

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Slide1 l.jpg

CSE 565

Software Verification, Validation, and Testing

A Guest Lecture on

Testing-Based Software and SystemReliability Evaluation

(Part 2)

Dr. Yinong Chen


Slide2 l.jpg

Basic Input Domain Models

Examples: MacWilliams73, Brown&Lipow75, Nelson78.

Software reliability is defined as the probabilityR(N) = Prob{no of failures over N application runs}

where N is the exposure period whose time unit is the number of application runs.

Assuming that input cases are selected independently, then R(N) can be expressed asR(N) = (R(1))N = RNwhere, R R(1) is the expected reliability per application run.

Now the question is how to estimate R.


Slide3 l.jpg

Estimate R

R, the reliability per test run, can be defined by the ratio of the number of test runs in which failures are observed and the total number of test runs when infinite number of different input cases are applied for test runs:

R = 1 – F = 1 –

Because of test time limit only a subset of the entire input domain can be applied to test the program in practice. Thus the reliability per test run, R, is usually estimated by


Example l.jpg

Example

Test the program 10 000 time (test runs)

Five (5) failures are observed

F = 5 / 10 000 = 1 / 2 000

R = 1 - 1 / 2 000 = 0.9995

Reliability in N

application

runs are

R(N) = RN


Slide5 l.jpg

MacWilliams 73 and Brown&Lipow 75

With partition and profiling

MacWilliams 73: the s input cases are selected randomly from the input domain.

Brown&Lipow 75: the input domain is partitioned into m classes. If si input cases are selected from class Ci and fi failures are observed, the reliability can be calculated by

where P(Ci) is a probability function reflecting the input profile in terms of classes.


Example6 l.jpg

Example

The input domain is partitioned into 10 sub-domains.

ˆ

ˆ

=

-

=

1 – 0.020938 = 0.979063

R

1

F


Slide7 l.jpg

Case Study 1Anti-lock Braking System (ABS)

  • Requirement:

  • To obtain the maximum braking effect

  • Algorithm:

  • Define (or measure) the wheel diameter;

  • Measure the wheel rotations per seconds rps;

  • Compute the wheel velocity wv;

  • Measure the body velocity bv;

  • Error detection and action:

  • if (bv > wv), reduce braking force

  • else if (bv < wv), reduce acceleration force

  • else “no action”


Slide8 l.jpg

Sample Code in C++

#include <iostream>

using namespace std;

const float mile_inch = 63360;

const float pi = 3.1416;

float wheel_diameter = 15;// inches

float wheel_sensor() {

float rps;

cout << "get rotations per second: " << endl;

rps = ReadWheelRotationSensor();

return rps;

}


Slide9 l.jpg

float wheel_velocity(float rps) {

float wv;

wv = (pi * wheel_diameter * rps * 3600)/mile_inch;

return wv;

}

float body_velocity() {

float bv;

cout << "get miles per hour: " << endl;

bv = ReadBodySpeedSensor();

return bv;

}


Slide10 l.jpg

void error_detection(float wv, float bv) {

if (abs(bv - wv) < 0.01)

cout << "no action" << endl;

else

if(bv > wv)

cout << "reduce brake force!" << endl;

else

cout << "reduce acceleration force!" << endl;

}

void evaluation() {

float rps, wv, bv;

rps = wheel_sensor();

wv = wheel_velocity(rps);

bv = body_velocity();

error_detection (wv, bv);

}

void main() {

for (i = 1, I < 10000, i++)

evaluation();

}


Slide11 l.jpg

Apply the input domain reliability model

  • Write the random function that simulates: ReadWheelRotationSensor();

  • Write the random function that simulates: ReadBodySpeedSensor();

  • Test the program and collect date

  • Apply the input domain model to evaluate the reliability of the program


Slide12 l.jpg

Case Study:

Testing greatest common divisor program

Source:

Y. Chen, W.T. Tsai, Introduction to programming languages: Programming in C, C++, Scheme, Prolog, C#, and SOA, second edition, Kendall/Hunt Publishing Company, 2006, ISBN 0-7575-2974-7.

Section 1.5.2.


Slide13 l.jpg

greatest common divisor program

#include <stdio.h>

int gcd (int n0, int m0) {// n0  0, m0  0

int n, m;// n0 or m0  0

n = n0; m = m0;

while (n != 0 && n != m) {

if(n < m) m = m - n;

else n = n - m; }

return m;

}

void main() {

int i, j, k;

scanf("%d\n%d", &i, &j);//input

k = gcd(i, j); // call gcd

printf("%d\n", k); // output

}


Slide14 l.jpg

Random Test Case Generation

InputOutput k

(i, j) = (6, 9)3

(i, j) = (10, 5)5

(i, j) = (0, 4)4

(i, j) = (5, 7)1

(i, j) = (8, 29)1

The program produces correct outputs for all these different test cases.

Can we claim that the program is correct?


Slide15 l.jpg

Example: Input Domain Partition Analysis

  • Input Domain Analysis:

  • The program takes two integers as input.

  • The branches of the program are controlled by the relative values of the two integers.

  • Input Domain Partitioning:

  • Partition each integer input into three groups: < 0, = 0, and > 0.

  • For this program, < 0 is not allowed. The group that has only one value is called boundary value.

  • Considering the semantics of the program, prime numbers and nonprime numbers play a role in the program. Thus, the positive integers are further divided into prime and nonprime numbers.


Slide16 l.jpg

Input Domain Partition

Based on the analysis, we have a partition of:

i: [0]; [2, 3, 5, 7, 11, …]; [4, 6, 8, 9, …]

j: [0]; [2, 3, 5, 7, 11, …]; [4, 6, 8, 9, …]

The combination of the two inputs generates following cases:

(0, 0): This case is not allowed according to the specification.

(0, 2), (0, 3), (0, 5), (0, 7), (0, 11), …

(0, 4), (0, 6), (0, 8), (0, 9), …

(2, 0), (2 2), (2, 3), (2, 5), (2, 5), …

(2, 4), (2, 6), (2, 8), (2, 9), …

(3, 0), (3, 2), (3, 3), (3, 5), (3, 7), …

. . .

(9, 0), (9, 2), (9, 3), (9, 5), (9, 7), …


Slide17 l.jpg

Coverage Consideration: Branch Coverage

no

n != 0?

yes

no

n != m?

yes

no

n < m?

yes

m = m - n;

n = n - m;

return m;

Exit

(0, 2),(0, 3),(0, 9),(0, 10): 

(2, 2),(3, 3), (9, 9),(10, 10):

(2, 3),(2, 9), (2, 10):...

(3, 2),(9, 2),10, 2),(10, 3),(10, 9): ...

(2, 0),(3, 0), (9, 0),(10, 0):...


Slide18 l.jpg

Testing the Program Using Test Cases

Let's trace the program with (i, j) = (2, 0):

(i, j) =(2, 0)

(2-0, 0)

(2-0, 0)

...

A dead loop occurs – a design fault is found

InputsOutput k

(i, j) = (0, 2)2

(i, j) = (2, 2)2

(i, j) = (2, 3)1

(i, j) = (3, 2)1

(i, j) = (2, 0)?


Apply the partition based input domain reliability model l.jpg

Apply the partition-based input domain reliability model

  • Modify the main program, so that it systematically generate (large number of) input cases from different sub-domain;

  • Test the gcd function and collect data

  • Apply the partition-based input domain model to evaluate the reliability of the program


Slide20 l.jpg

SUMMARY SOFTWARE RELIABILITY MODELS

  • Basic concepts and terminology

  • Faults and failure rates

  • Reliability R(t) and availability A(t)

  • Software reliability models

    • Classifications

    • Time-domain models

    • Fault count model

    • Input-domain models

    • Fault seeding model

  • Sample code of real-time ABS software

    • Sample code of real-time ABS software

    • Partition-based input domain testing


Slide21 l.jpg

Modeling Complex Systems

(Software and Hardware)

  • A large system can be decomposed into smaller components.

  • Evaluate the reliability of the components;

  • Evaluate the reliability of the system based on known component reliabilities

    • Combinatorial Models

    • Markov Models


Slide22 l.jpg

Markov Models

Markov models are more generic than combinatorial models.

They can handle repairs and much more complex situations.

Assumption:

•Any component may in one the two states: working or failed;

•Probability of state transition depends only on the current state.

ß

Failure rates and repair rates are constants.

ß

Transition probability is proportional to the time that the component stays at a state.

ß

Exponential distribution of the reliability/availability


Slide23 l.jpg

ß

Construct differential equations

(2)

ß

Solve the equations to obtain the probability in each state

(3)

(4)

ß

The reliability or availability is the sum of the probabilities of working states.

Steps of Applying Markov Models

A system consists of multiple components

ß

Construct state transition diagram

(1)


Slide24 l.jpg

System

l

0

1

m

A(t) = p0(t)

Step 1: Construct state transition diagram

Example 1: Simplex system with repair


Slide25 l.jpg

Module A

Module B

Voter

Input

Output

Module C

Step 1: Construct state transition diagram

Example 2: Reliability of TMR system with repair


Slide26 l.jpg

b

4a

4b

00

(failed nodes, failed links)

a

10

01

3a+4b

4a+3b

11

20

02

Step 1: Construct state transition diagram

Example 3: A ring system with different node and link failure rates a and b. Assumethat the system fails if any two or more than components failed.

Failed


Slide27 l.jpg

Step 2: Construct differential equations

= –l·p0 (t) + m·p1 (t)

= l·p0 (t)–m·p1 (t)

A(t) = p0(t)

The question is how to obtain the probability of each state.

p0 (t + Dt) = (1 –l · Dt)  · p0 (t) + m · Dt ·p1 (t)

p1 (t + Dt) = l · Dt ·p0 (t) + (1 –m · Dt)  · p1 (t)

Solve the differential equations to obtain (p0 (t), p1 (t)).


Slide28 l.jpg

Step 2: Construct differential equations

= –l·p0 (t) + m·p1 (t)

p0

p1

-lm

l-m

= l·p0 (t)–m·p1 (t)

=

l

0

1

m


Slide29 l.jpg

Step 3: Solve differential equations

p0

p1

-lm

l-m

m

l

-

l

+

m

p0

(

)

t

(

t

)

=

+

e

=

l

+

m

l

+

m

l

m

-

l

+

m

p1

(

)

t

(

t

)

=

+

e

l

+

m

l

+

m

  • There are many different ways to solve differential equations

  • LaPlace Transformation

  • Tools like MatLab or Mathematica


Slide30 l.jpg

m

l

-

l

+

m

(

)

t

A

(

t

)

=

+

e

l

+

m

l

+

m

p0

p0

(

(

t

t

)

)

=

=

Step 4: Find the Probabilities of Working States

m

l

-

l

+

m

p0

(

)

t

(

t

)

=

+

e

l

+

m

l

+

m

l

m

-

l

+

m

p1

(

)

t

(

t

)

=

+

e

l

+

m

l

+

m

If m = 0, the probability at p0 represents the reliability

m

l

-

l

+

m

=

-

l

(

)

t

t

R

(

t

)

=

+

e

e

l

+

m

l

+

m


Slide31 l.jpg

Step 2: Construct differential equations (Find the pattern)

a12

1

2

a21

a13

a23

a31

a32

a14

a41

3

a25

a52

a34

a35

a53

a43

a45

4

5

a54

p1

P2

p3

p4

p5

dp

(

t

)

aij

=

2

dt

dp

(

t

)

5

dt


Slide32 l.jpg

Step 2: Construct differential equations (Find the pattern)

In general, assume a STD has n states and is fully connected. Any state has n incoming and n outgoing transitions:

aij 0 is the transition rate from state i to j.

For i, j = 1, 2, ..., n, and i ≠ j.


Slide33 l.jpg

Step 2: Construct differential equations (Find the pattern)

where

The probability in state j at t + Dt

= the probability in state j at t+ incoming prob – outgoing prob

Math manipulation: Divide Dt on both sides, let Dt 0


Slide34 l.jpg

Let Dt 0

dp

(

t

)

(

)

(

)

(

)

n

n

n

j

=

a

-

×

a

=

a

-

×

b

å

p

(

t

)

p

(

t

)

å

å

p

(

t

)

p

(

t

)

p

(

t

)

i

ij

j

ji

i

ij

j

j

dt

j

1

1

1

=

=

=

i

i

i

¹

¹

¹

i

j

i

j

i

j

(

)

(

)

n

n

a

-

a

D

å

p

(

t

)

å

p

(

t

)

t

+

D

-

D

D

D

p

(

t

t

)

t

t

t

i

ij

j

ji

j

1

1

=

=

i

i

where

¹

¹

i

j

i

j

=

Step 2: Construct differential equations (More detail of the previous slide)


Slide35 l.jpg

p1

p2

p3

pn

b1

a21

a31

a41

an1

a12

b2

a32

a42

an2

dp

(

t

)

=

2

a13

a23

b3

a43

an3

dt

dp

(

t

)

n

bn

a1n

a2n

a3n

a4n

dt

where

Step 2: Construct differential equations (found the pattern)


Slide36 l.jpg

3

l

Example 1: Apply the Pattern

-3

l

m

0

æ

ö

ç

T

=

-(2l+m)

÷

0

ç

÷

è

0

ø

0

2l

R(t) = p1(t) + p2(t)


Slide37 l.jpg

4a

4b

00

10

01

3a+4b

4a+3b

11

-

4

(

a

+

b

)

4

b

4

a

0

0

æ

0

ö

ç

÷

-

(

4

a

+

3

b

)

0

4

a

+

0

3

b

ç

÷

T

=

0

-

(

3

a

+

4

b

)

÷

ç

0

ç

÷

0

0

è

3

a

+

4

b

ø

Example 2

1

2

3

4

R(t) = p1(t) + p2(t) + p3(t)


Summary l.jpg

SUMMARY

  • Basic concepts of reliability and reliability modeling

  • Hardware reliability models

  • Software reliability models

  • System reliability models consisting of multiple components

    • Combinatorial models

    • Markov models


  • Login