Software Fault Injection

1. Software Fault Injection Kalynnda Berens Science Applications International Corporation NASA Glenn Research Center

2. What is Software Fault Injection? • A testing technique that aids in understanding how software behaves when stressed in unusual ways. • A product-based assurance technique. • Variations in the technique allow it to be applied to many types of software and for different purposes. Software Fault Injection NASA Glenn Research Center

3. How does SFI work? • Legal permutations or faults are input at interfaces (external and/or internal). • Outputs show whether the injected fault propagates through the software. • Requires instrumentation (software code) to observe the propagation process. Software Fault Injection NASA Glenn Research Center

4. Uses for Software Fault Injection • Finding defects in software • Robustness Testing • COTS Validation/Determining failure modes • Safety Verification • Security Assessment • Software Testability Analysis Software Fault Injection NASA Glenn Research Center

5. SFI Examples • Operating System Validation • Ballista (CM) – Linux and VxWorks robustness • WindowsNT • Network Security • NCSA httpd server • Safety • Advanced Automatic Train Control system • Magneto Stereoaxis System SFI can be used with or without source code Software Fault Injection NASA Glenn Research Center

6. SFI without Source Code • Create software wrapper for COTS functions and other interfaces • “Trick” OS to call wrapper functions first • Software under test usually run in debug mode • Wrapper can be used • Pass through for baselining response • Call alternative function • Call original function but change result Software Fault Injection NASA Glenn Research Center

7. SFI wrapper operations Pass through wrapper: OS or Hardware Application Wrapper Call alternative function: New Function Wrapper Application Call original function but change result: Wrapper Application OS or Hardware Software Fault Injection NASA Glenn Research Center

8. Center Initiative on SFI • Can SFI be used by an IndependentV&V engineer? • Is SFI a useful and cost-effective technique for NASA? • Are the errors and problems found of sufficient severity or abundance? • Are the costs of applying the technique reasonable for the number/severity of errors found? • Is SFI a good tool for safer software? Software Fault Injection NASA Glenn Research Center

9. Methodology • Determine scope • Select projects • Metrics • Perform SFI on projects • Create Test Plan (prototype due 1st quarter, FY02) • Lessons Learned Software Fault Injection NASA Glenn Research Center

10. Determine Scope • Why narrow the scope? • SFI is a collection of related techniques • Comparison across projects requires using one technique for all • Why no source/interfaces technique chosen • IV&V perspective (cost effective) • “Outside” events or system limitations trigger many errors • Interfaces selected • COTS software • Hardware • User input • Communications medium Software Fault Injection NASA Glenn Research Center

11. Project Selection • Potential Projects • CM-2 • Tempest Web Server (VxWorks and Java) • MDCA, FPP, SAMS, others • Selection Criteria • Selection difficulties • Project support not free • Contracted software not accessible • Final Choice Software Fault Injection NASA Glenn Research Center

12. Metrics • Time spent per task • Familiarization, researching errors, instrumenting software, testing • Subjective “effort” scale per task • Software project metrics • SLOC, #classes/modules, complexity, interface information • Fault Injection metrics • #faults, #failures, #faults no effect/correctly handled Software Fault Injection NASA Glenn Research Center

13. SFI Process • Obtain Tempest software (completed) • Obtain access to VxWorks (completed) • ***Overcome compatibility problems • Determine all interfaces to test • Select errors to inject • Create necessary wrappers for SFI • Record test procedure and results Software Fault Injection NASA Glenn Research Center

14. Tempest Interfaces • VxWorks OS • Task creation and control functions • C/C++ language functions • File system functions • Networking functions • Outside world • Requests from external sources • Standard HTML, built-in functions • Tempest (VxWorks version) can execute OS functions Software Fault Injection NASA Glenn Research Center

15. Example Injection Errors • OS errors • Memory allocation failures • File errors (corrupted, not found) • Single task abort, hang • External World errors • Invalid request • Too many requests • Requests too frequent Software Fault Injection NASA Glenn Research Center

16. Test Plan • How to perform software fault injection on “generic” software • Steps prior to actual testing • Method of determining errors to inject • Procedure for performing the test • Appendices of lessons learned, example faults, other guidance Software Fault Injection NASA Glenn Research Center

17. Difficulties Encountered • Tempest documentation limited • VxWorks simulator does not support networking • Cost of hardware and full VxWorks not within the budget • Attempt to “fake” networking unsuccessful Software Fault Injection NASA Glenn Research Center

18. Status and Future Work • VxWorks incompatibilities not easily overcome • Shift to Java version of Tempest for now • Test VxWorks version of Tempest on actual hardware (if possible) or alternate operating system (Linux, uClinux, eCos) • If funding continues, test on actual flight experiment (CM-2). Software Fault Injection NASA Glenn Research Center