Test Automation – How far should it go?

1. Test Automation – How far should it go? Ian Gilchrist, IPL

2. WeW Contents • Context and Introduction • C++ Example (‘Reverse String’) with script generation, coverage analysis, wrapping • C example (‘Reactor’) with table-driven testing, coverage optimisation, robustness testing 20 minutes 15 minutes 10 minutes

3. Context • C and C++ software • Unit and Integration tests • ‘High integrity’ systems • Up to and including ‘safety-critical’

4. The Back Story • Personal history • I entered the software industry in 1983 as a programmer working mainly in Assembler code • Testing then consisted of ‘suck and see’ • By 1985 we were using test facilities based on use of emulator, but requiring much manual set up, and interpretation of results from hex output dump of memory. • Not pleasant to do! • Not repeatable!

5. System Requirements Architectural Design Detailed Design Code Source code Analysis Unit Design Code Software Development Lifecycle Acceptance Test System Test Integration Test Unit Test Dynamic testing proof Does code do what it should? Is code tested sufficiently? Code Coverage requirements

6. Why Unit Test? Regulated Industries (avionics, medical, etc) • Mandated or highly recommended • Structural code coverage • Variation by SIL (System Integrity Level) Business-critical • Making a profit! • Protecting reputation • Avoidance of litigation

7. Unit/Integration Test - Technical Challenges • Test harness creation • Finding time to create reusable test harnesses to test all functionality • Control of interfaces • Making it simple to test object interactions with the rest of the system • Flexibility of Black-Box/White-box and OO techniques • Having a full tool-kit to test efficiently • Selection/Generation of test data for different purpose • Functional tests – does code do what it should? • Robustness tests – does code survive large data sets? • Baseline tests – we know code’ works’ but want a baseline against which to test changes • Code coverage • Confidence that tests cover the important code • Evidence generated is sufficient to meet standards

8. Automation can do this: • Parse of project code • Generate structured test script • case per function/ method • parameters, call order, global data, returns • test case independence • Incorporate checks according to variable type • Positive and negative checking of global data • Results production Test Harness Creation • Test driver harness for units • Structured auto-repeatable test cases • Checking of Outputs • Resultsproduction

9. Automation can do this • Knowledge of calls made • Automated generation of • Stubs to simulate calls • Wrappers to intercept calls • Programmable instances for Stubs and Wrappers • Checks on parameters based on types • Intercept • Simulate • Intercept Control of Interfaces • Isolation of units from system • Order of calls • Different behavior on calls • Checking/modifyingparameters

10. External Object Stub is a dummy function replacing interface to the External Object Stubs A function/method in test script with programmable instances Replaces call to external software, firmware or hardware Check Parameters Check Call Sequences Stub for External Object Source Code Choose Return Parameter value(s)

11. BEFORE Wrapper Source Code External Object AFTER Wrapper Wrapping A function/method in test script with programmable instances using Before-After or Before-Replace wrapper pairs Intercepts call to external software, firmware or hardware Wrapper for External Object Check Out Parameters Check Call sequences Modify Out Parameters Modify In Parameters and Return Check In Parameters and Return

12. Test Driver Data • Automation can do this: • Automated accessibility instrumentation • Call private methods and static functions • Set/check data which is private, in unnamed namespaces and declared static • Wrapping calls internal to compilation unit • Wrapping OS library calls Private Public • Set + Check  Checks White (clear) Box Testing • Call private methods/static functions • Set/check private/static data • Control of internal calls

13. Test Driver Data • Automation can do this: • Table-driven test generation • Multiple values per parameter • Ranges of values • Functions calculating values • Combinatorial effect calculator • Checks on call sequences and returns • Robustness rule sets for data types Public  Checks Black Box Testing • Large data input sets • Checking large output sets • Robustness tests

14. Automation can do this: • Parallel hierarchy of code and test case re-use • C++ template instantiation and testing • C++ inheritance and factory method testing • Object oriented code coverage Object Oriented Code • Test case re-use aligned with code • Support for Templates • Support for Inheritance

15. Automation can do this: • Coverage Rule Sets • Powerful drill-down views, filters and reports • Coverage Metrics • Entry- Point Coverage • Statement Coverage • Decision Coverage • Call-return Coverage • Condition Coverage (including MC/DC) • Test case coverage optimization Integrated Code Coverage • Set coverage target in tests • Diagnostics over stages / test runs • Sensible coverage metrics • Coverage redundancy

16. Automation can do this • Automatic generation of passing C unit test baseline • Automatic regression suite of makefiles • Knowing testability limitations in code • Dynamically unreachable code • Crash scenarios • Data uninitialised or function static • Implicit function declarations • Compiler type truncation Large Legacy Code Base • Automatic generation of unit tests • Reducing reliance on system regression tests • Identifying testability issues

17. End of Introduction I C++ Example – ‘Reverse String’ C Example – ‘Reactor’ C Example – ‘Airborne’

18. End of Presentation I If you have any questions, please email: Ian.gilchrist@ipl.com THANK YOU FOR ATTENDING!