Rescaling Reliability Bounds for a New Operational Profile
This presentation is the property of its rightful owner.
Sponsored Links
1 / 29

Rescaling Reliability Bounds for a New Operational Profile Peter G Bishop [email protected] PowerPoint PPT Presentation


  • 63 Views
  • Uploaded on
  • Presentation posted in: General

Rescaling Reliability Bounds for a New Operational Profile Peter G Bishop [email protected] [email protected] Adelard, Drysdale Building, Northampton Square, London EC1V 0HB +44 20 7490 9450 www.adelard.com. Outline of Talk. Original reliability bound theory (same op. profile)

Download Presentation

Rescaling Reliability Bounds for a New Operational Profile Peter G Bishop [email protected]

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Rescaling reliability bounds for a new operational profile peter g bishop pgb adelard

Rescaling Reliability Bounds for a New Operational Profile

Peter G Bishop

[email protected]

[email protected]

Adelard, Drysdale Building, Northampton Square, London EC1V 0HB

+44 20 7490 9450

www.adelard.com


Outline of talk

Outline of Talk

  • Original reliability bound theory (same op. profile)

  • Extended theory (different operational profile)

  • Implications of the theory

  • Experimental evaluation


Original theory

Original Theory

Input Domain

Defect

D

1

Operational

1

Observed

profile (I)

2

D

defect

2

failure

frequency

D

3

3


Theory assumptions

Theory assumptions

  • the operational profile is invariant, i.e.s are constant over time

  • when a failure occurs the associated defect is immediately and perfectly corrected

  • removal of a defect does not affect the s of the remaining defects


Basic idea

Basic idea

Given some test interval t :

  • Defects with large s will be removed already

  • Defects with small s will remain - but have little affect on program reliability

  • So there must be an “worst case”  for a defect that maximises the program failure rate after t


Worst case bound

Worst-case bound

  • Original paper showed that, given the assumptions, max failure /unit time for a defect iis:

    i|t  1/et(where t is the test time)

  • So if there are N faults in the program the failure rate at time tis bounded by:

    |t  N/et


Bound is independent of l

Bound is independent of l

1

l

=0.1

l

=0.01

l

=0.001

0.1

1/et

0.01

Probablity

of failure

0.001

 | t

0.0001

0.00001

1

10

100

1000

10000

t


Refinement for discrete tests

Refinement for discrete tests

  • For for a discrete sequence of T tests the result is:

    |T  N (T/T+1)T/(T+1)

    N/(eT) (conservative approx.)

  • So it is conservative to use original equation.


Limitations

Limitations

  • Assumes operational profile I is constant hence ls are constant

  • But we know that in practice the profile changes.

  • So the reliability bound does not apply if the operational profile changes

    • (e.g. from system test to actual use)

    • but will “settle back” in long term if new profile stable

  • New theory gives a means for “rescaling” the reliability bound for a different profile


Additional assumptions

Additional assumptions

  • Each defect is localised to a single code “block”

  • The operational profile I can be characterised by the distribution of code block executions Q in the program {q(1), q(2), … }

  • The failure rate of defect in block, l(i)  q(i)

  • There is a constant probability of a fault existing in any line of executable code.


Rescaling for known defect

Rescaling for known defect

  • For a defect i in code block j , the re-scaled bound would be:

    where q’(j) is the new execution rate and q(j) is the old execution rate.


Probability of defect in block

Probability of defect in block

  • We do not know which block contains defect i, but we assume that the chance of being in jis:

    L(j)/L

    where L(j) is the length of the code block, and L is the total length of the executable code.


Re scaled bound

¢

q

( j

)

L

(

j

)

å

×

q

(

j

)

L

Re-scaled bound

  • Taking the average over all blocks:

  • So the “scale factor” relative to the original bound is:

  • Also true if there are N faults rather than 1


Theory predictions fair testing

Theory predictions - Fair testing

  • If q  L of blocks “dominated” by decision branch,scale factor unchanged by any other profile

  • Applies to any acyclic graph,

  • And subgraphs with fixed iteration loops

Segment j

L(j)q’(j)

q(j)

L(j).

q’(j)

q(j)

Root 0

10

1

1

10

Branch 1

10

0.1

0.9

90

90

0.9

0.1

10

Branch 2

Sum

110

110

S =Sum/L

1


Unfair testing

Unfair testing

  • Use of “unbalanced” test profile can be very sensitive to changes in profile

  • Factor can be less than 1 if under-tested blocks avoided, e.g. Q’={1,1,0} gives S = 0.19

q’(j)

Segment j

L q’/q

q(j)

L(j)

Root 0

10

1

1

10

Branch 1

10

0.9

0.1

1.1

810

90

0.1

0.9

Branch 2

Sum

110

829

S =Sum/L

7.5


Limits to fair test approach

Limits to fair test approach

  • Fair test apportionment does not work for variable loops, recursion and subroutines

  • Even if we identify a fair test profile, it may be infeasible to execute

Decisions not independent (shared variable)


Maximum scale factor

Maximum scale factor

  • If we know max. possible execution rates for each block, can estimate a “maximum scale factor”:

    ( q(k) max / q(k) ) (L(k) / L)

  • Where k relates to a worst case “thread” through the graph. Hard to identify this thread, but easier to compute a more pessimistic factor:

    ( q(j) max / q(j) ) (L(j) / L)

    where j includes all blocks.

  • No knowledge of the new profile is needed


Including module tests

¢

L

(

j

)

q

(

j

)

å

×

+

L

q

(

j

)

x

(

j

)

/

T

Including module tests

  • Can combine module tests and system tests, composite scale factor is:

    where x(j) are the total executions under module testing

  • Module tests can “fill in” uncovered segments that would make the test profile “unbalanced”


Experimental evaluation

Experimental evaluation

  • Use programs with known set of defects

  • PODS

    • simple reactor trip application (<1000 code lines)

    • simple structure, fixed loops

  • PREPRO

    • ~ 10 000 code lines

    • parses input description file of indefinite length

    • recursive - max execution unknown

  • Similar results - will only discuss PODS here


Pods evaluation

PODS evaluation

  • Measure Q for different test profiles

    • Uniform, Normal, Inverse normal - “bathtub”

  • Measure defect failure rates l(i) under all profiles

  • Predict residual failure rate:l(i) exp(-l(i)T)

  • Compute failure rate for new profile:l’(i) exp(-l(i)T)

  • Compare with scaled bound: (L(j)/L)(q’(j)/q(j))N/eT


Variation in q j

Variation in q(j)


Predicted scale factors

Predicted scale factors

Operational profile

Test profile uniforminv-normal normal

uniform 1 1.20.9

inv-normal 3.2 16.2

normal115 3461

  • Note the predicted reduction in bound


Maximum scale factor1

Maximum scale factor

Test profileMaxscale-up factor

uniform6.6

inv-normal10.0

normal1059

  • 2-5 times worst than bound with a known profile

  • Can be over-pessimistic

  • But could indicate relative sensitivity to change


Unfair normal test profile

1

0.1

0.01

Operation

(uniform)

Mean

0.001

Fails/test

0.0001

(normal)

0.00001

0.000001

10

100

1000

10000

100000

Tests

“Unfair” Normal test profile

Max bound

Scaled bound

N/et bound


Fairer uniform test profile

1

0.1

0.01

Test

Mean

(uniform)

0.001

Fails/test

0.0001

Operation

(normal)

0.00001

0.000001

10

100

1000

10000

100000

Tests

“Fairer” Uniform test profile

Max bound

N/et bound

Scaled bound


Prepro

PREPRO

  • Similar results

    • changes in failure rates are within the scaled bounds

  • But could not compute a maximum bound

    • program is recursive

    • so no upper bound on the execution of program code blocks


Summary

Summary

Theory suggests:

  • Can rescale bound (knowing Q and Q’)

  • Can include module test execution information

  • Can compute max scale up (knowing Q and Qmax)

  • For some program structures can identify a totally "fair" test profile - bound insensitive to change

    The experimental evaluations appear to be consistent with the predictions of the theory


Conclusions

Conclusions

  • Could affect approach to testing:

    • “fairer” test profiles rather than realistic profiles

    • integrated module and system test strategy

  • Could improve reliability bound prediction for new environment

  • Could assess sensitivity to profile change

    • e.g. by computing maximum scale factor

  • But based on quite strong assumptions, need to:

    • validate assumptions

    • assess impact of assumption violation

    • evaluate on more examples


  • Login