Loading in 5 sec....

Rescaling Reliability Bounds for a New Operational Profile Peter G Bishop [email protected]PowerPoint Presentation

Rescaling Reliability Bounds for a New Operational Profile Peter G Bishop [email protected]

- By
**zaina** - Follow User

- 86 Views
- Uploaded on

Download Presentation
## PowerPoint Slideshow about ' Rescaling Reliability Bounds for a New Operational Profile Peter G Bishop [email protected]' - zaina

**An Image/Link below is provided (as is) to download presentation**

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Rescaling Reliability Bounds for a New Operational Profile

Peter G Bishop

Adelard, Drysdale Building, Northampton Square, London EC1V 0HB

+44 20 7490 9450

www.adelard.com

Outline of Talk

- Original reliability bound theory (same op. profile)
- Extended theory (different operational profile)
- Implications of the theory
- Experimental evaluation

Original Theory

Input Domain

Defect

D

1

Operational

1

Observed

profile (I)

2

D

defect

2

failure

frequency

D

3

3

Theory assumptions

- the operational profile is invariant, i.e.s are constant over time
- when a failure occurs the associated defect is immediately and perfectly corrected
- removal of a defect does not affect the s of the remaining defects

Basic idea

Given some test interval t :

- Defects with large s will be removed already
- Defects with small s will remain - but have little affect on program reliability
- So there must be an “worst case” for a defect that maximises the program failure rate after t

Worst-case bound

- Original paper showed that, given the assumptions, max failure /unit time for a defect iis:
i|t 1/et(where t is the test time)

- So if there are N faults in the program the failure rate at time tis bounded by:
|t N/et

Bound is independent of l

1

l

=0.1

l

=0.01

l

=0.001

0.1

1/et

0.01

Probablity

of failure

0.001

| t

0.0001

0.00001

1

10

100

1000

10000

t

Refinement for discrete tests

- For for a discrete sequence of T tests the result is:
|T N (T/T+1)T/(T+1)

N/(eT) (conservative approx.)

- So it is conservative to use original equation.

Limitations

- Assumes operational profile I is constant hence ls are constant
- But we know that in practice the profile changes.
- So the reliability bound does not apply if the operational profile changes
- (e.g. from system test to actual use)
- but will “settle back” in long term if new profile stable

- New theory gives a means for “rescaling” the reliability bound for a different profile

Additional assumptions

- Each defect is localised to a single code “block”
- The operational profile I can be characterised by the distribution of code block executions Q in the program {q(1), q(2), … }
- The failure rate of defect in block, l(i) q(i)
- There is a constant probability of a fault existing in any line of executable code.

Rescaling for known defect

- For a defect i in code block j , the re-scaled bound would be:
where q’(j) is the new execution rate and q(j) is the old execution rate.

Probability of defect in block

- We do not know which block contains defect i, but we assume that the chance of being in jis:
L(j)/L

where L(j) is the length of the code block, and L is the total length of the executable code.

q

( j

)

L

(

j

)

å

×

q

(

j

)

L

Re-scaled bound- Taking the average over all blocks:
- So the “scale factor” relative to the original bound is:
- Also true if there are N faults rather than 1

Theory predictions - Fair testing

- If q L of blocks “dominated” by decision branch,scale factor unchanged by any other profile
- Applies to any acyclic graph,
- And subgraphs with fixed iteration loops

Segment j

L(j)q’(j)

q(j)

L(j).

q’(j)

q(j)

Root 0

10

1

1

10

Branch 1

10

0.1

0.9

90

90

0.9

0.1

10

Branch 2

Sum

110

110

S =Sum/L

1

Unfair testing

- Use of “unbalanced” test profile can be very sensitive to changes in profile
- Factor can be less than 1 if under-tested blocks avoided, e.g. Q’={1,1,0} gives S = 0.19

q’(j)

Segment j

L q’/q

q(j)

L(j)

Root 0

10

1

1

10

Branch 1

10

0.9

0.1

1.1

810

90

0.1

0.9

Branch 2

Sum

110

829

S =Sum/L

7.5

Limits to fair test approach

- Fair test apportionment does not work for variable loops, recursion and subroutines
- Even if we identify a fair test profile, it may be infeasible to execute

Decisions not independent (shared variable)

Maximum scale factor

- If we know max. possible execution rates for each block, can estimate a “maximum scale factor”:
( q(k) max / q(k) ) (L(k) / L)

- Where k relates to a worst case “thread” through the graph. Hard to identify this thread, but easier to compute a more pessimistic factor:
( q(j) max / q(j) ) (L(j) / L)

where j includes all blocks.

- No knowledge of the new profile is needed

L

(

j

)

q

(

j

)

å

×

+

L

q

(

j

)

x

(

j

)

/

T

Including module tests- Can combine module tests and system tests, composite scale factor is:
where x(j) are the total executions under module testing

- Module tests can “fill in” uncovered segments that would make the test profile “unbalanced”

Experimental evaluation

- Use programs with known set of defects
- PODS
- simple reactor trip application (<1000 code lines)
- simple structure, fixed loops

- PREPRO
- ~ 10 000 code lines
- parses input description file of indefinite length
- recursive - max execution unknown

- Similar results - will only discuss PODS here

PODS evaluation

- Measure Q for different test profiles
- Uniform, Normal, Inverse normal - “bathtub”

- Measure defect failure rates l(i) under all profiles
- Predict residual failure rate:l(i) exp(-l(i)T)
- Compute failure rate for new profile:l’(i) exp(-l(i)T)
- Compare with scaled bound: (L(j)/L)(q’(j)/q(j))N/eT

Variation in q(j)

Predicted scale factors

Operational profile

Test profile uniform inv-normal normal

uniform 1 1.2 0.9

inv-normal 3.2 1 6.2

normal 115 346 1

- Note the predicted reduction in bound

Maximum scale factor

Test profileMaxscale-up factor

uniform 6.6

inv-normal 10.0

normal 1059

- 2-5 times worst than bound with a known profile
- Can be over-pessimistic
- But could indicate relative sensitivity to change

0.1

0.01

Operation

(uniform)

Mean

0.001

Fails/test

0.0001

(normal)

0.00001

0.000001

10

100

1000

10000

100000

Tests

“Unfair” Normal test profileMax bound

Scaled bound

N/et bound

0.1

0.01

Test

Mean

(uniform)

0.001

Fails/test

0.0001

Operation

(normal)

0.00001

0.000001

10

100

1000

10000

100000

Tests

“Fairer” Uniform test profileMax bound

N/et bound

Scaled bound

PREPRO

- Similar results
- changes in failure rates are within the scaled bounds

- But could not compute a maximum bound
- program is recursive
- so no upper bound on the execution of program code blocks

Summary

Theory suggests:

- Can rescale bound (knowing Q and Q’)
- Can include module test execution information
- Can compute max scale up (knowing Q and Qmax)
- For some program structures can identify a totally "fair" test profile - bound insensitive to change
The experimental evaluations appear to be consistent with the predictions of the theory

Conclusions

- Could affect approach to testing:
- “fairer” test profiles rather than realistic profiles
- integrated module and system test strategy

- Could improve reliability bound prediction for new environment
- Could assess sensitivity to profile change
- e.g. by computing maximum scale factor

- But based on quite strong assumptions, need to:
- validate assumptions
- assess impact of assumption violation
- evaluate on more examples

Download Presentation

Connecting to Server..