How to Suspend Testing and Still Succeed – A True Story

1. How toSuspend Testingand Still Succeed– A True Story Graham Thomas Independent Software Testing Consultant

2. Abstract This is the story of a testing programme that was destined to fail, but ultimately succeeded. We will look at what went wrong, explain why testing had to be suspended, and discuss how with no real hope of recovery the team managed to set and meet their resumption requirements, and ultimately complete their testing on time. We will discuss where things started to go wrong, how this was identified, and what measures were taken to ensure a successful resolution. We will also look at the detail of the challenges that the testing team, and the program, were daily presented with when testing was suspended. And tell how innovation, ingenuity and perseverance, against all the odds, won the day. This is a real ‘war story’, from the testing front line, with valuable hard won experience, and is told in the very real hope that it will benefit all who hear it.

3. Agenda • Background & Set the Scene • Testing Strategy • Initial Progress • Suspension • Resumption • Further Demands • Conclusions

4. Background • The Program was scoped as delivery of the infrastructure and processing of the new Faster Payments solution for one of the member banks • Faster Payments is a new (mid 2008) component of the UK’s Financial Services Infrastructure • First new payments system in UK for 20 years • Centrally Regulated • High Volume and High Availability • Near Real Time financial transactions between member banks for phone, internet and standing order transactions. • Daily (5/7) settlement

5. Background 2 • Arrived as Program Test Manager just as; • Phase 1 development was delivering • The first phase of external Scheme testing commenced (Self Certification) • Development plans for subsequent releases were firming up • The Program Testing Strategy was issued for review • Role was to pull together existing testing threads of: • Unit Testing (offshore, external suppliers, and internal) • Systems Testing • Systems Integration Testing • External Scheme Testing • OAT • UAT • Business Procedure Testing • Providing oversight, control, and stakeholder management for testing

6. Testing Strategy • First task was to review the Draft Testing Strategy • Met with current development and testing approach • Aligned with outsourcing partner • Strengthened testing organisation with representation at Program level • Weak in Systems Integration • Weak test measurement • Weak in presentation • Document too long 50+pp • and not audience friendly, too much text, few diagrams • Outcome of review: • Testing Strategy ‘fit for purpose’ • Systems Integration Risk accepted at Program level • Mitigation too expensive and not viable within timeframe

7. Initial Progress • Systems Integration completed successfully (as scoped) • Initially testing hampered by slippage in development delivery; • Single phase delivery into testing became multiple drops of code • Testing planned on a Just in Time basis • OAT planning delayed and execution moved back • Slow progress in UAT execution due to issues with the test automation infrastructure supplied by the solution vendor • Non-functional proving of infrastructure took 250% longer than planned and had a qualified exit (i.e. not complete)

8. Suspension • Take a view from the room? • How many people have actually experienced testing being suspended? • What if any where the suspension criteria? • What was the impact? • Who got the blame?

9. What the Standard Says • IEEE829 • 9.2.7 (LTP Section 2.7) Suspension criteria and resumption requirementsSpecify the criteria to be used to suspend all or a portion of the testing activity on the test items associated with this plan. Specify the testing activities that must be repeated when testing is resumed. • Two Website References: • Know when to pause in a series of tests. • If the number or type of defects reaches a point where the follow on testing has no value, it makes no sense to continue the test; you are just wasting resources. • Specify what constitutes stoppage for a test or series of tests and what is the acceptable level of defects that will allow the testing to proceed past the defects. • Testing after a truly fatal error will generate conditions that may be identified as defects but are in fact ghost errors caused by the earlier defects that were ignored. • A Document Reference: • Suspension of test execution will occur if the number or severity of incidents raised is so great as to make further execution valueless • Specify the testing activities that must be repeated, when testing is resumed

10. Why We Stopped Testing • We had realised that we did not have enough test resource to complete the task so had brought in external resource to augment UAT • At current rate of progress in UAT 8 weeks into a 12 week schedule it was going to take over a year to complete! • Many of issues (defects) were to do with automation infrastructure and product configuration • The Systems Integration Risk had matured! • Unless we did something radical we were going to have to declare to the Scheme that we were failing • This would mean losing our implementation slot, and cause significant internal cost overruns and knock-on impacts on other programs of work

11. So Now what? • We had to swiftly identify what was wrong • A quick series of workshops with all stakeholders • Developers, Testers, Suppliers, Customers, and Business • Once we had an idea of what was wrong we had to speedily plan a resolution • A targeted Systems Integration Test, carried out in the UAT environments • Our Systems Integration risk had matured • Product configuration • We had a default supplier provided configuration which needed to be revised to reflect the current software and business model • Test infrastructure optimisation • Multi threaded and concurrent testing configuration • We had to execute the resolution • Allocate highest non-production priority • Extended working hours, Provision of support, reservation of key resource to support • Close knit group working between all teams • And the program still had to deliver on time • We had stopped, were going to lose time resolving our problems, but still had to deliver on the same date!

12. Resumption • We also had to determine what would allow us to resume testing • New UAT manager wanted all UAT tests run successfully before UAT could commence! • Requested a set of viable Resumption Requirements that met the reasons why we suspended testing • System Integrated • System Configured • Test Automation met desired throughput rates • That we developed a set of re-usable System Integration Tests that could be re-run • No Short-cuts, position or views – We were not going to kid ourselves • I tried in vain to get the organisation to use the term Resumption Requirements, but everyone kept on sayingResumption Criteria!

13. Re-Planning It wouldn’t be a software development without re-planning! • We re-planned UAT Execution • Combined Future development releases • They weren’t separate business releases • Saved re-testing time lost with resumption activities • Utilised contingency • Accepted a scheme request to move the implementation date back gaining 3 weeks • Moved Business Procedure Testing out of UAT • Implemented predictive test measures

14. Restarting Testing • Amazingly the 4 week suspension was completed on time • We fixed outstanding integration issues • We configured the product • We resolved the problems with test automation • And created a repeatable set of System Integration tests • And in 5 weeks UAT had re-planned from a 3 phase to a 1 phase test cycle. • It took a week to deploy the changes • Don’t forget there may be a lag in resuming execution • And within 3 days of test execution we had already completed 60% of the revised UAT schedule • I don’t want to make it sound that easy, because some tests required multiple cycles to be run, however, suddenly the end was in sight and achievable

15. Further Demands • So all is looking good, testing has just finished a 4 week suspension, and current throughput rates show that UAT will complete on time. • However this is not an easy program • It is the largest, costliest and most complex development that the organisation has undertaken for nearly a decade, and it is in an area of new technologies that the organisation is not familiar with • More testing is identified for UAT • We have 6 times as much to do as they thought! • OAT planning completed and shows that 9 months are required not the 3 that the program has • And External Scheme testing is about to commence, and the timescales have been cut by over 50%!

16. Organise for Success • We had been playing football like 11 year olds • We had all been chasing after the ball – Testing Suspension • We instituted a daily War Room involving • Decision Makers • The doers • Program Office to record and account • We met daily at 13:00, or more frequently if required to manage issues • We made decisions and gave direction

17. The War Room • 1 physical dedicated Meeting Room • Equipped with projector • Reports printed out and placed on the walls • Measures and metrics for all to see • Dedicated conference number • Strict rules as to attendance and protocol • Handled all program issues, not just testing • Worked as a collective, with individual responsibility

18. The End Game • We used all of the tools at our disposal • Took a firm view on prioritisation of defects • Took a clear view of scope for current release • Compromised for success • To fail to deliver would cost the organisation £1m per month, excluding reputational and other damage • Exercised tight control on change • Engaged with Business, IT and Change throughout • Used all contingency even within deployment window • Limited go-live exposure with low initial volumes • Provided full implementation hand-holding

19. Postscript What did Success look like? • The program delivered on-time • There were zero defects in the first 3 months • The development group gained the confidence to know that it could deliver the remaining phases of the program • The bad practice lessons were not repeated for future phases • As a result of the suspension, senior management questioned the original Testing Strategy, • The Testing Strategy was retrospectively re-written • And signed off shortly after the system went live!

20. Summary • Suspension and Resumption is more complicated than the standard leads us to believe • You will probably not be able to precisely specify suspension criteria prior to the suspension of testing. You will only know these at or around the time you suspend testing. (The standard is not too helpful here!) • Resumption Requirements will be directly dependent upon the reasons for suspension and will have to be written and agreed very shortly after suspension. (You are not just repeating previous testing as per the standard.) • Suspending testing does not necessarily buy you more time to test, and in fact may actually give you less • Suspending testing is a very powerful tool. Don’t underestimate the amount of management attention you will receive, but do be aware of the benefits that fully met resumption requirements can bring

21. Contact Details Graham Thomas Independent Software Testing Consultant graham@badgerscroft.com +44 7973 387 853  www.badgerscroft.com