Establishing an effective performance testing environment. Gordon McKeown www.facilita.com TMF 2010

1. Establishing an effective performance testing environment.Gordon McKeown www.facilita.comTMF 2010

2. Agenda • Introductory tour of the issues. • Share our experiences & identify best practice. • In the spirit of the TMF... You will drive the thesession! TMF "Creating an effective performance testing environment"

3. What do we mean by "Environment“ ? • A wide scope. But not too wide! • In • "Systems" • Network • Tool • Load Injectors • System Under Test (SUT) • Software installation & configuration • Test data • Out • People (except for responsibilities / roles ) • Tool selection TMF "Creating an effective performance testing environment"

4. Performance test elements TMF "Creating an effective performance testing environment"

5. Desirable characteristics • Few impediments to tester activities. • Low management & bureaucratic overheads. • “Realistic tests” are possible. • SUT as close as possible to production. • Experimentation is possible. • Isolation • Reproducible tests • Reduced risk of impact on corporate ICT TMF "Creating an effective performance testing environment"

6. Some of the issues • Who controls the test infrastructure? • Security standards. • Test data. • Relationship to "production“. • Resource sharing. • Technical innovations. • Any other major areas? TMF "Creating an effective performance testing environment"

7. Should the test team declare independence? TMF "Creating an effective performance testing environment"

8. Who should control the environment? • "Lack of environmental control by testers is a common, serious impediment to testing efficiency." That is my experience. What is yours? • Testing is an exploratory activity, we often don't know what the next change should be. • Is it easier if load testing is integrated with development? • Are things worse in organisations conducting infrequent tests? E.g. medium sized organisation with limited IT capability who have commissioned a new system. TMF "Creating an effective performance testing environment"

9. Corporate security standards • We have the characteristics of hackers! • Denial of Service Attack. • Recording interactions. • "Man in the middle." • Looking inside systems. • Reverse engineering in extremis. TMF "Creating an effective performance testing environment"

10. Test data • Quantity & realism • Depending on DB design too little data can give unrealistically fast or slow timings. • An "unrealistic" database can distort results • Managing "real" data extracted from a live system. • Legal & ethical issues • Anonymizing TMF "Creating an effective performance testing environment"

11. Test's relationship to “Production" • Naive scaling • But who can afford full duplication? • Are sizing models fit for purpose? • Systems are not linear. • The "unknown unknowns" are a risk. • An example: licensing • One of our client's load tested against a near replica system. We recommended a subset of tests should be run against actual system (with test db) before going live. Discovered default server license for 50 users had not been upgraded to 300. • Significant but unforeseen configuration issues. • Changing configuration settings from scaled down replica to production: should this be tested? TMF "Creating an effective performance testing environment"

12. Resource sharing • Functional testing & UAT sharing resources with performance testing. • Mutual interference & frustration • Reproducible tests may not be possible. • Switching between configurations & databases. • Out of hours testing: a solution or another set of problems? TMF "Creating an effective performance testing environment"

13. Technical innovation • Cloud computing. • Software as service. • Virtualization. • What else might change the game? TMF "Creating an effective performance testing environment"