Rescaling Reliability Bounds for a New Operational Profile Peter G Bishop pgb@adelard

1. Rescaling Reliability Bounds for a New Operational Profile Peter G Bishop pgb@adelard.com pgb@csr.city.ac.uk Adelard, Drysdale Building, Northampton Square, London EC1V 0HB +44 20 7490 9450 www.adelard.com

2. Outline of Talk • Original reliability bound theory (same op. profile) • Extended theory (different operational profile) • Implications of the theory • Experimental evaluation

3. Original Theory Input Domain Defect D 1 Operational 1 Observed profile (I) 2 D defect 2 failure frequency D 3 3

4. Theory assumptions • the operational profile is invariant, i.e.s are constant over time • when a failure occurs the associated defect is immediately and perfectly corrected • removal of a defect does not affect the s of the remaining defects

5. Basic idea Given some test interval t : • Defects with large s will be removed already • Defects with small s will remain - but have little affect on program reliability • So there must be an “worst case”  for a defect that maximises the program failure rate after t

6. Worst-case bound • Original paper showed that, given the assumptions, max failure /unit time for a defect iis: i|t  1/et(where t is the test time) • So if there are N faults in the program the failure rate at time tis bounded by: |t  N/et

7. Bound is independent of l 1 l =0.1 l =0.01 l =0.001 0.1 1/et 0.01 Probablity of failure 0.001  | t 0.0001 0.00001 1 10 100 1000 10000 t

8. Refinement for discrete tests • For for a discrete sequence of T tests the result is: |T  N (T/T+1)T/(T+1) N/(eT) (conservative approx.) • So it is conservative to use original equation.

9. Limitations • Assumes operational profile I is constant hence ls are constant • But we know that in practice the profile changes. • So the reliability bound does not apply if the operational profile changes • (e.g. from system test to actual use) • but will “settle back” in long term if new profile stable • New theory gives a means for “rescaling” the reliability bound for a different profile

10. Additional assumptions • Each defect is localised to a single code “block” • The operational profile I can be characterised by the distribution of code block executions Q in the program {q(1), q(2), … } • The failure rate of defect in block, l(i)  q(i) • There is a constant probability of a fault existing in any line of executable code.

11. Rescaling for known defect • For a defect i in code block j , the re-scaled bound would be: where q’(j) is the new execution rate and q(j) is the old execution rate.

12. Probability of defect in block • We do not know which block contains defect i, but we assume that the chance of being in jis: L(j)/L where L(j) is the length of the code block, and L is the total length of the executable code.

13. ¢ q ( j ) L ( j ) å × q ( j ) L Re-scaled bound • Taking the average over all blocks: • So the “scale factor” relative to the original bound is: • Also true if there are N faults rather than 1

14. Theory predictions - Fair testing • If q  L of blocks “dominated” by decision branch,scale factor unchanged by any other profile • Applies to any acyclic graph, • And subgraphs with fixed iteration loops Segment j L(j)q’(j) q(j) L(j). q’(j) q(j) Root 0 10 1 1 10 Branch 1 10 0.1 0.9 90 90 0.9 0.1 10 Branch 2 Sum 110 110 S =Sum/L 1

15. Unfair testing • Use of “unbalanced” test profile can be very sensitive to changes in profile • Factor can be less than 1 if under-tested blocks avoided, e.g. Q’={1,1,0} gives S = 0.19 q’(j) Segment j L q’/q q(j) L(j) Root 0 10 1 1 10 Branch 1 10 0.9 0.1 1.1 810 90 0.1 0.9 Branch 2 Sum 110 829 S =Sum/L 7.5

16. Limits to fair test approach • Fair test apportionment does not work for variable loops, recursion and subroutines • Even if we identify a fair test profile, it may be infeasible to execute Decisions not independent (shared variable)

17. Maximum scale factor • If we know max. possible execution rates for each block, can estimate a “maximum scale factor”: ( q(k) max / q(k) ) (L(k) / L) • Where k relates to a worst case “thread” through the graph. Hard to identify this thread, but easier to compute a more pessimistic factor: ( q(j) max / q(j) ) (L(j) / L) where j includes all blocks. • No knowledge of the new profile is needed

18. ¢ L ( j ) q ( j ) å × + L q ( j ) x ( j ) / T Including module tests • Can combine module tests and system tests, composite scale factor is: where x(j) are the total executions under module testing • Module tests can “fill in” uncovered segments that would make the test profile “unbalanced”

19. Experimental evaluation • Use programs with known set of defects • PODS • simple reactor trip application (<1000 code lines) • simple structure, fixed loops • PREPRO • ~ 10 000 code lines • parses input description file of indefinite length • recursive - max execution unknown • Similar results - will only discuss PODS here

20. PODS evaluation • Measure Q for different test profiles • Uniform, Normal, Inverse normal - “bathtub” • Measure defect failure rates l(i) under all profiles • Predict residual failure rate:l(i) exp(-l(i)T) • Compute failure rate for new profile:l’(i) exp(-l(i)T) • Compare with scaled bound: (L(j)/L)(q’(j)/q(j))N/eT

21. Variation in q(j)

22. Predicted scale factors Operational profile Test profile uniform inv-normal normal uniform 1 1.2 0.9 inv-normal 3.2 1 6.2 normal 115 346 1 • Note the predicted reduction in bound

23. Maximum scale factor Test profileMaxscale-up factor uniform 6.6 inv-normal 10.0 normal 1059 • 2-5 times worst than bound with a known profile • Can be over-pessimistic • But could indicate relative sensitivity to change

24. 1 0.1 0.01 Operation (uniform) Mean 0.001 Fails/test 0.0001 (normal) 0.00001 0.000001 10 100 1000 10000 100000 Tests “Unfair” Normal test profile Max bound Scaled bound N/et bound

25. 1 0.1 0.01 Test Mean (uniform) 0.001 Fails/test 0.0001 Operation (normal) 0.00001 0.000001 10 100 1000 10000 100000 Tests “Fairer” Uniform test profile Max bound N/et bound Scaled bound

26. PREPRO • Similar results • changes in failure rates are within the scaled bounds • But could not compute a maximum bound • program is recursive • so no upper bound on the execution of program code blocks

27. Summary Theory suggests: • Can rescale bound (knowing Q and Q’) • Can include module test execution information • Can compute max scale up (knowing Q and Qmax) • For some program structures can identify a totally "fair" test profile - bound insensitive to change The experimental evaluations appear to be consistent with the predictions of the theory

28. Conclusions • Could affect approach to testing: • “fairer” test profiles rather than realistic profiles • integrated module and system test strategy • Could improve reliability bound prediction for new environment • Could assess sensitivity to profile change • e.g. by computing maximum scale factor • But based on quite strong assumptions, need to: • validate assumptions • assess impact of assumption violation • evaluate on more examples