Think You’re Done Yet?

1. Think You’re Done Yet? Other things to do for your Test Automation implementation Presented by Jim Hazen

2. Introduction • Jim Hazen • Veteran of the Software Testing Trenches • Experience in software testing, both commercial and consulting work. • Working with automation and tools since 1989. • Implemented Automation on DOS, OS/2, Windows and Web platforms. • Experience with various tools; including HP/Mercury, IBM/Rational, and other ‘scripting’ languages.

3. So Now What?

4. Agenda • Test Execution Plan • Grouping tests into suites • Unattended Execution / Distribute Workload • Equipment and test environments • Data and databases • Utility Scripts • Maintenance of framework and scripts • Training of staff • Summary • Q&A

5. Test Execution Plan • What is it? • The game plan that explains the Who, What, When, Where and How for execution of tests. • It can be a detailed document, outline or schedule. • Why create one? • To better organize and manage the execution of tests. • You can’t run everything all at the same time, you need one as a road map. • It provides the structure for grouping tests for execution. • Who uses it? • The whole project team. • It is a needed communication tool. Don’t go ‘dark’.

6. Example Test Execution Plan

7. Grouping Tests into Suites • Why group tests? • To be better able to determine what to run and when. • To better manage the tests and organize them according to need & focus. • Criteria for groups • Type of Test • Smoke, Regression (critical path & in-depth), Business Scenario’s, Fault/Error Handling, and End-to-End • Functionality Coverage • User Interface, Reporting, Database, Middle layer, Back-end layer, and interaction with other systems

8. Grouping Tests into Suites • Criteria for groups cont. • Dependency Levels • None - can be run independently; creates/cleans up data, no dependencies upon prior tests or configuration of system, and no pre-conditions before execution • Data - run only with correct data or data condition present • Environmental – run only with correct environ. or config. • Previous Test - run only if a prior test has been executed • Combination - dependent upon combination of data or environment or prior condition (due to previous test)

9. Grouping Tests into Suites • Effects & Benefits of grouping • Improves the teams ability to classify, organize and execute tests according to focus/purpose of testing effort. • Improves flexibility of test execution. • Improves ability to estimate how long the tests will take to execute/complete. • Makes for more efficient use of the tests and allows for targeted test execution. • Improves ability to understand ‘ripple effect’ of changes/fixes in scripts - Reduce mistakes and rework.

10. Unattended Execution • What is it? • Build it in and methodology to use • SEARCH1,2 • Setup, Execute, Analyze, Report, Cleanup, Help or Home • Base State Restoration – Start and Stop at same place • Evaluating if your scripts are ready • SEARCH capability and Dependencies • Organizing scripts for execution • Grouping and Run Order • Overnight & 24x7 Execution

11. Distribute Workload • Distributing the workload and running in parallel • “Flip the workload stack on its side”

12. Distribute Workload cont. • Leverage “economies of scale”

13. Equipment and Test Environments • Hardware • Multiple workstations vs. Virtual Machines • Cost factors, configuration, and maintenance3 • Environments • Needs to be a duplicate or scaled down version of production • Need for separate automation environment • Reduce various problems due to outside influences • Justifying the costs and getting buy-in

14. Test Lab Example

15. Data and Databases • Test Data • Automation needs the correct types of data and amount of data to be able to run successfully. Without it your scripts are not going to do much. • Possibly pull from Production and ‘Clean’ it for use • The test data needs to be ‘mapped’ so you know what you have and which scripts are dependent upon it. You will have to ‘mine’ for it. This takes time and effort. • If using a shared a database you will need to ‘separate’ your data used by automation from that used by manual testing. • Test Databases • Databases will need to be separate and baselined for automation. This way you can restore as needed to begin a new cycle of testing.

16. Utility Scripts • Setup & Configuration • Scripts that setup and configure the system to some known base state. • Pre-Test conditions, Post Test • Data and Database • Scripts that go and create data or mine for it. • Scripts that go and ‘age’ time sensitive data. • System • Scripts that go and cleanup or restore the test system; for example any temp files created during test run that need to be removed. • Others • FTP of files or data from other systems. Anything else outside of the execution of the tests themselves.

17. Maintenance of Framework & Scripts

18. Maintenance of Framework & Scripts • Maintenance • Time & resources have to be pre-allocated to do this task. • Plan as part of an iteration cycle. If not done it will build up technical debt quickly, you’ll be in catch up mode and put your automation effort in jeopardy. • Explain to management this is a critical task to do. • Framework • Update libraries, components, object definition/repositories and data sets. • Scripts • Update scripts accordingly and retest as part of effort.

19. Training Staff • VS. • Untrained • Trained

20. Training Staff • Automation Team • Need to be trained in the framework and the libraries. • Need in-depth training in the tool(s) being used. • Need training in technologies being tested and automated. • Need training in programming and development methods. • Testers & BA/SME’s • Need to be trained in how to use the scripts and data files to create usable tests. • Need to be trained in how to use SEARCH method effectively. • Other Groups • Need to be educated on what automation can and cannot do, and why. Set expectations properly.

21. Summary • Discussed • Test Execution Plan and why it is needed • Grouping Tests for better execution efficiency • SEARCH method to improve usability of automation • Test Labs and using Virtualization to add efficiency • Test Data, Databases, Utility Scripts and why they are important • Maintenance and how important it is • Training of staff and what needs to be done • Benefits • Simple, it protects the investment in automation and helps to ensure the usefulness, maintainability and longevity of it.

22. Q&A

23. Contact Info • Jim Hazen • Company: (www.connectedtesting.com) • Company email: jim.hazen@connectedtesting.com • Home email: calkelpdiver@gmail.com

24. References • Page 187, “How We Test Software At Microsoft”, Alan Page, Microsoft Press, 2009. • “How to Automate Testing: The Big Picture“, Keith Stobie & Mark Bergman, 1992. • “Virtualization: The Path to Multiple Efficiencies”, Alan Page, http://www.hwtsam.com/star/Virtualization.pdf, STARWest 2009. • “Software Test Automation”, Mark Fewster & Dorothy Graham, Addison-Wesley, 1999. • “Automated Software Testing”, Elfriede Dustin et al., Addison-Wesley, 1999. • “Implementing Automated Software Testing”, Elfriede Dustin et al., Addison-Wesley, 2009. • “Seven Steps to Test Automation Success”, Bret Pettichord, http://www.io.com/~wazmo/papers/seven_steps.html, 2001. • “Success with Test Automation”, Bret Pettichord, http://www.io.com/~wazmo/succpap.htm, 2001.

25. Tool References Test Execution Tools • HP Quality Center • Visual Studio 2010 - http://msdn.microsoft.com/en-us/library/dd286580.aspx • MKS Integrity - http://www.mks.com/platform/integrations/integrationtechnologies/atef • Rational Quality Mgr. - http://www-01.ibm.com/software/awdtools/rqm/standard/ • Selenium Grid – http://selenium-grid.seleniumhq.org/ • Hudson CI Tool and Build Slaves - http://www.testinggeek.com/index.php/testing-tools/test-execution/170-distributed-automated-testing • TestComplete - http://www.automatedqa.com/products/testcomplete/ • PS Exec Remote Execution - http://technet.microsoft.com/en-us/sysinternals/bb897553

26. Thank you for attending this session. Please fill out an evaluation form.