slide1
Download
Skip this Video
Download Presentation
Rescaling Reliability Bounds for a New Operational Profile Peter G Bishop [email protected]

Loading in 2 Seconds...

play fullscreen
1 / 29

Rescaling Reliability Bounds for a New Operational Profile Peter G Bishop [email protected] - PowerPoint PPT Presentation


  • 86 Views
  • Uploaded on

Rescaling Reliability Bounds for a New Operational Profile Peter G Bishop [email protected] [email protected] Adelard, Drysdale Building, Northampton Square, London EC1V 0HB +44 20 7490 9450 www.adelard.com. Outline of Talk. Original reliability bound theory (same op. profile)

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Rescaling Reliability Bounds for a New Operational Profile Peter G Bishop [email protected]' - zaina


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Rescaling Reliability Bounds for a New Operational Profile

Peter G Bishop

[email protected]

[email protected]

Adelard, Drysdale Building, Northampton Square, London EC1V 0HB

+44 20 7490 9450

www.adelard.com

outline of talk
Outline of Talk
  • Original reliability bound theory (same op. profile)
  • Extended theory (different operational profile)
  • Implications of the theory
  • Experimental evaluation
original theory
Original Theory

Input Domain

Defect

D

1

Operational

1

Observed

profile (I)

2

D

defect

2

failure

frequency

D

3

3

theory assumptions
Theory assumptions
  • the operational profile is invariant, i.e.s are constant over time
  • when a failure occurs the associated defect is immediately and perfectly corrected
  • removal of a defect does not affect the s of the remaining defects
basic idea
Basic idea

Given some test interval t :

  • Defects with large s will be removed already
  • Defects with small s will remain - but have little affect on program reliability
  • So there must be an “worst case”  for a defect that maximises the program failure rate after t
worst case bound
Worst-case bound
  • Original paper showed that, given the assumptions, max failure /unit time for a defect iis:

i|t  1/et(where t is the test time)

  • So if there are N faults in the program the failure rate at time tis bounded by:

|t  N/et

bound is independent of l
Bound is independent of l

1

l

=0.1

l

=0.01

l

=0.001

0.1

1/et

0.01

Probablity

of failure

0.001

 | t

0.0001

0.00001

1

10

100

1000

10000

t

refinement for discrete tests
Refinement for discrete tests
  • For for a discrete sequence of T tests the result is:

|T  N (T/T+1)T/(T+1)

N/(eT) (conservative approx.)

  • So it is conservative to use original equation.
limitations
Limitations
  • Assumes operational profile I is constant hence ls are constant
  • But we know that in practice the profile changes.
  • So the reliability bound does not apply if the operational profile changes
    • (e.g. from system test to actual use)
    • but will “settle back” in long term if new profile stable
  • New theory gives a means for “rescaling” the reliability bound for a different profile
additional assumptions
Additional assumptions
  • Each defect is localised to a single code “block”
  • The operational profile I can be characterised by the distribution of code block executions Q in the program {q(1), q(2), … }
  • The failure rate of defect in block, l(i)  q(i)
  • There is a constant probability of a fault existing in any line of executable code.
rescaling for known defect
Rescaling for known defect
  • For a defect i in code block j , the re-scaled bound would be:

where q’(j) is the new execution rate and q(j) is the old execution rate.

probability of defect in block
Probability of defect in block
  • We do not know which block contains defect i, but we assume that the chance of being in jis:

L(j)/L

where L(j) is the length of the code block, and L is the total length of the executable code.

re scaled bound

¢

q

( j

)

L

(

j

)

å

×

q

(

j

)

L

Re-scaled bound
  • Taking the average over all blocks:
  • So the “scale factor” relative to the original bound is:
  • Also true if there are N faults rather than 1
theory predictions fair testing
Theory predictions - Fair testing
  • If q  L of blocks “dominated” by decision branch,scale factor unchanged by any other profile
  • Applies to any acyclic graph,
  • And subgraphs with fixed iteration loops

Segment j

L(j)q’(j)

q(j)

L(j).

q’(j)

q(j)

Root 0

10

1

1

10

Branch 1

10

0.1

0.9

90

90

0.9

0.1

10

Branch 2

Sum

110

110

S =Sum/L

1

unfair testing
Unfair testing
  • Use of “unbalanced” test profile can be very sensitive to changes in profile
  • Factor can be less than 1 if under-tested blocks avoided, e.g. Q’={1,1,0} gives S = 0.19

q’(j)

Segment j

L q’/q

q(j)

L(j)

Root 0

10

1

1

10

Branch 1

10

0.9

0.1

1.1

810

90

0.1

0.9

Branch 2

Sum

110

829

S =Sum/L

7.5

limits to fair test approach
Limits to fair test approach
  • Fair test apportionment does not work for variable loops, recursion and subroutines
  • Even if we identify a fair test profile, it may be infeasible to execute

Decisions not independent (shared variable)

maximum scale factor
Maximum scale factor
  • If we know max. possible execution rates for each block, can estimate a “maximum scale factor”:

( q(k) max / q(k) ) (L(k) / L)

  • Where k relates to a worst case “thread” through the graph. Hard to identify this thread, but easier to compute a more pessimistic factor:

( q(j) max / q(j) ) (L(j) / L)

where j includes all blocks.

  • No knowledge of the new profile is needed
including module tests

¢

L

(

j

)

q

(

j

)

å

×

+

L

q

(

j

)

x

(

j

)

/

T

Including module tests
  • Can combine module tests and system tests, composite scale factor is:

where x(j) are the total executions under module testing

  • Module tests can “fill in” uncovered segments that would make the test profile “unbalanced”
experimental evaluation
Experimental evaluation
  • Use programs with known set of defects
  • PODS
    • simple reactor trip application (<1000 code lines)
    • simple structure, fixed loops
  • PREPRO
    • ~ 10 000 code lines
    • parses input description file of indefinite length
    • recursive - max execution unknown
  • Similar results - will only discuss PODS here
pods evaluation
PODS evaluation
  • Measure Q for different test profiles
    • Uniform, Normal, Inverse normal - “bathtub”
  • Measure defect failure rates l(i) under all profiles
  • Predict residual failure rate:l(i) exp(-l(i)T)
  • Compute failure rate for new profile:l’(i) exp(-l(i)T)
  • Compare with scaled bound: (L(j)/L)(q’(j)/q(j))N/eT
predicted scale factors
Predicted scale factors

Operational profile

Test profile uniform inv-normal normal

uniform 1 1.2 0.9

inv-normal 3.2 1 6.2

normal 115 346 1

  • Note the predicted reduction in bound
maximum scale factor1
Maximum scale factor

Test profileMaxscale-up factor

uniform 6.6

inv-normal 10.0

normal 1059

  • 2-5 times worst than bound with a known profile
  • Can be over-pessimistic
  • But could indicate relative sensitivity to change
unfair normal test profile

1

0.1

0.01

Operation

(uniform)

Mean

0.001

Fails/test

0.0001

(normal)

0.00001

0.000001

10

100

1000

10000

100000

Tests

“Unfair” Normal test profile

Max bound

Scaled bound

N/et bound

fairer uniform test profile

1

0.1

0.01

Test

Mean

(uniform)

0.001

Fails/test

0.0001

Operation

(normal)

0.00001

0.000001

10

100

1000

10000

100000

Tests

“Fairer” Uniform test profile

Max bound

N/et bound

Scaled bound

prepro
PREPRO
  • Similar results
    • changes in failure rates are within the scaled bounds
  • But could not compute a maximum bound
    • program is recursive
    • so no upper bound on the execution of program code blocks
summary
Summary

Theory suggests:

  • Can rescale bound (knowing Q and Q’)
  • Can include module test execution information
  • Can compute max scale up (knowing Q and Qmax)
  • For some program structures can identify a totally "fair" test profile - bound insensitive to change

The experimental evaluations appear to be consistent with the predictions of the theory

conclusions
Conclusions
  • Could affect approach to testing:
    • “fairer” test profiles rather than realistic profiles
    • integrated module and system test strategy
  • Could improve reliability bound prediction for new environment
  • Could assess sensitivity to profile change
    • e.g. by computing maximum scale factor
  • But based on quite strong assumptions, need to:
    • validate assumptions
    • assess impact of assumption violation
    • evaluate on more examples
ad