Software Testing and Reliability Robustness Testing

1. Graduate Assistants: Ramkumar Natarajan Baskar Sridharan Software Testing and ReliabilityRobustness Testing Aditya P. Mathur Purdue University May 19-23, 2003 @ Guidant Corporation Minneapolis/St Paul, MN Last update: April 17, 2003

2. Learning objectives • What is robustness testing? • How to perform robustness testing? (The ballista test methodolgy.) • Results from the Ballista test methodology. Software Testing and Reliability Aditya P. Mathur 2002

3. References • The Ballista Software Robustness Testing Service, John P. DeVale, Philip J. Koopman, and David J. Guttendorf, Testing Computer Software, June 1999. [http://www.ices.cmu.edu/ballista] • Robustness Testing of the Microsoft Win32 API, Charles P. Shelton, Philip J. Koopman, and Koby DeVale, Proceedings of the International Conference on Dependable Systems, June 25-28, 2000, IEEE Press. Software Testing and Reliability Aditya P. Mathur 2002

4. Robustness: Review • Robustness is the degree to which a software component functions correctly in the presence of exceptional inputs or stressful environmental conditions. • Robustness testing • Clues come from requirements. The goal is to test a program under scenarios not stipulated in the requirements. Software Testing and Reliability Aditya P. Mathur 2002

5. Failures due to Lack of Robustness • Apollo 11: Lunar Module experienced three computer crashes and reboots during powered lunar descent due to exceptional sensor and equipment settings. • Ariane 5 rocket: Flight had to be aborted shortly after liftoff due to an unhandled exceptional error condition. • Others: We observe program crashes almost daily! Often these are due to erroneous or non-handling of exceptional conditions. Software Testing and Reliability Aditya P. Mathur 2002

6. Programming error or Requirements error? • Often, program crashes, or hangs, occur due to the incorrect handling of an exceptional condition. • The input that causes the mishandled exception condition to arise might arise due to an error in the requirements, or in the design, or in the code. However, for a user it is simply a defect in the application that leads to a failure. • The impact of an application crash could be inconvenience caused to its user or as extreme as to a cause a catastrophe. Software Testing and Reliability Aditya P. Mathur 2002

7. Why lack of robustness ? • Specifications are often incomplete. This may lead to an implementation that does not handle cases left unspecified. • Resource constraints often leave code segments untested for extreme or unexpected values. • Example: atoi(NULL) ---> segmentation fault? • Why would any one invoke atoi() with a NULL parameter? What form of black- and white- box testing are needed to detect hidden sources of exceptional conditions? Software Testing and Reliability Aditya P. Mathur 2002

8. Robustness, functionality, and operational profiles • Most development organizations plan for functional testing. Often, various types of black-box testing techniques are used. • Justification of functional testing is relatively straightforward by invoking the “unhappy customer” scenario. • But functional testing is unlikely to lead to the test of exceptional conditions. • Tests based on well-derived operational profiles are also unlikely to test for exceptional conditions for the obvious reason that the operational profile merely prioritizes functionality. Software Testing and Reliability Aditya P. Mathur 2002

9. Off-the-shelf components and robustness • Often one composes an application with one or more components acquired from a different vendor and integrating these with code developed in-house. • The code and other related documents for off-the-shelf component might not be available thus making code-based testing impossible. • Hence the need for a methodology for robustness testing that does not require access to the source code of the components under test. Software Testing and Reliability Aditya P. Mathur 2002

10. The Ballista Approach to Robustness Testing • Often one composes an application with one or more components acquired from a different vendor and integrating these with code developed in-house. • The code and other related documents for off-the-shelf component might not be available thus making code-based testing impossible. • Hence the need for a methodology for robustness testing that does not require access to the source code of the components under test. Software Testing and Reliability Aditya P. Mathur 2002

11. The Ballista Testing Methodology • Based on combinational generation of tests that consist of valid and invalid values of parameters. • For each test case a module under test(MuT) is called once to check if it is robust to that input. • Parameter values are drawn from a pool of valid and invalid values for a set of data types. • Functionality of MuT is not a concern. • Of concern is whether or not MuT crashes or hangs for the test case. Software Testing and Reliability Aditya P. Mathur 2002

12. Test Cases from Ballista • Each test case is of the form {MuT, param_value1, param_value2, …param_valueN} and corresponds to the procedure call MuT (param_value1, param_value2, …param_valueN) Software Testing and Reliability Aditya P. Mathur 2002

13. 8 ZERO MAXINT ZERO E E ONE MININT 16 ONE DBLMAX DBLMAX ZERO NEGONE 32 NEGONE DBLMIN DBLMIN TWO TWO 64 ONE SMALLNOTZERO SMALLNOTZERO PI PI NEGONE 1K NEGSMALLNOTZERO NEGSMALLNOTZERO PIBY2 64K PIBY2 2 inttrap( 64K ) ONE, DBLMAX, TWOPI 4 TWOPI Test Cases from Ballista: Example inttrap(double a, double b, intN) All combinations are generated. Software Testing and Reliability Aditya P. Mathur 2002

14. Scalability of Ballista Tests • Scalability: Growth in the number of test cases is linear, or sub-linear, in the number of modules tested. • Scalability achieved by… • constraining specifications to “don’t crash” and “don’t hang,” • base the test cases on the exceptional values of data types, and • reuse of test cases amongst modules that have interfaces with matching parameters. • Example: 233 functions and system calls in Posix API with real-time extensions required defining only 20 data types. Software Testing and Reliability Aditya P. Mathur 2002

15. 12/1/1899 Date String 1/1/1900 Generic String BIGSTRING 1/1/2047 GenericPointer STRINGLEN1 NULL 8/31/1992 ALLASCII DELETED NONPRINTABLE 1K PAGESIZE MAXSIZE INVALID Organization of Test Values • Each data type is treated as a module and fits into a hierarchical object oriented class structure. Thus, all values (test cases) of a parent type are inherited by the child. Software Testing and Reliability Aditya P. Mathur 2002

16. Generation of Test Values • Each test value is associated with up to three code segments. • The first code segment creates an instance of a data type with specific properties. • The optional second code segment modifies the effects of the constructor such nas deleting a file while returning a file handle! • The optional third code segment deletes or frees system resources that might be reserved by other constructors such as deleting a file or freeing system memory. Software Testing and Reliability Aditya P. Mathur 2002

17. Phantom Parameters • How to handle modules that have no parameters and might input from file(s) and/or system state?. • Set up a dummy module that sets up system state and/or files to be used by the MuT and then invoke the MuT. • Ballista allows the specification of “phantom parameters” during parameter set up. Ballista invokes constructor/destructor pairs for these parameters but does not actually pass them to the MuT. • For example, for a file input, a phantom parameter is defined to create a file. If this file is accessed via, say, a central registry, then phantom parameter can appropriately register the file object. Software Testing and Reliability Aditya P. Mathur 2002

18. Failure Types • Catastrophic failure corrupts the state of the OS and generally leads to a restart of the machine. • Restart failure causes a process to “hang” requiring a kill signal for its termination after an application-dependent time interval. • Abort failure causes abnormal termination of the process such as a “core dump.” • Silent failure causes return of a value with no indication of an error when an exceptional value is input, such as returning a value for log (0). [Might require manual analysis.] • Hindering failure is characterized by the MuT returning an error code that incorrectly describes the exception that led to the error. [Might require manual analysis.] Software Testing and Reliability Aditya P. Mathur 2002

19. Test Results [Sample] • For 233 function calls across 15 different POSIX compliant OS, normalized failure rates were found to be 9.99% to 22.69%. • Code that produces catastrophic failure on Win 95, 98, and CE: GetThreadContext(GetCurrentThread(),NULL); • Overall robustness failure rates=Number of failed tests/total tests Sample: Windows NT: Varies from 6.2%-51% (No catastrophic failures) Windows 2000: Varies from 7.4%-48.9% (No catastrophic failures) Windows CE: Varies from 6.7-33.4% (excluding catastrophic failures) Software Testing and Reliability Aditya P. Mathur 2002

20. User machine Interface Specification Capture Hardening Wrapper Creation Result Pattern Discovery Ballista Test Client Module Under Test WWW and RPC Test Selection Test Reporting Balista server Ballista WEB Server Software Testing and Reliability Aditya P. Mathur 2002

21. Summary • What is robustness? • A robustnes testing methodology • Sample robustness failure rates of a few operating systems Software Testing and Reliability Aditya P. Mathur 2002