Maintaining System & SW Verification Test Consistency

1. IV&V Facility Maintaining System & SWVerification Test Consistency Testing Flight SW Response to Enhanced Failure Modes OSMA Software Assurance Research Program FY03: Initiative 583 FY04: CSIP2004-77 By Ted Bennett & Paul Wennberg 2004 Software Assurance Symposium TRIAKIS Corporation July 2004

2. Analyze/Test/V&V Analyze/Test/Verify Requirements Build Model,Simulate,Prototype,ES, etc. SYSTEM SW Interpretation Integration Testing Design/Debug Design/Debug Problem Most embedded SW faults found at integ. test traceable to Rqmts. & interface misunderstanding Disconnect exists between System and software development loops

3. Analyze/Test/V&V Analyze/Test/Verify Requirements Build ES-BasedFull System Simulation (VSIL) SYSTEM SW Integration Testing • Simulate Embedded Controller HW Design/Debug Design/Debug • Replace ES Controller Part • Verify SW Using Unmodified System Tests ROM CPU I/O • Load Object Software RAM Approach • Test Results

4. Accomplishments • Unmodified ES verification tests give same results when testing object SW running on simulated PowerPC-based SRMS • 131 system-level verification tests written • ES- and DE-based system simulations pass 129 tests • 2 failed tests due to simulator bug

5. Importance/Benefits • Early discovery of SW faults prior to HW integration testing Unmodified system-level tests can be used to verify embedded object SW Promotes SW assurance through close coupling of system and SW V&V Lower development $$ by finding more faults early, w/o lab or integration HW

6. Relevance to NASA • Potential to reduce project costs • Improve project-level assurance • Improve IV&V through SW testing • Verify executable SW • Manually generated software • Auto-generated software • Reused/modified software • Real-time operating systems

7. Problem: FMEA Limitations Expensive & time-consuming List of possible failure modes extensive Focuses on prioritized subset of failure modes Approach: Test SW w/sim’d Failures Create pure virtual simulation of Mini-AERCam HW & flight environment running on PC Induce realistic component/subsystem failures Observe flight SW response to induced failures IV&V Facility Mini-AERCam Empirical Assurance of Embedded SWUsing Realistic Simulated Failure Modes • Can we improve coverage by testing SW resp. to sim’d failures? • Compare results with project-sponsored FMEA: #Failure modes evaluated? #Issues uncovered? Effort involved?

8. Mini-AERCam Project Importance/Benefits • Virtual environment testing requires no HW • HW integration lab expensive, scarce • Simulate more failure modes than analysis permits • Beyond practical capability of HW integration labs • More failure modes tested  more bugs found • SMA use of simulator for orthogonal testing • System & SW implementation tested – not just design

9. Mini-AERCam Project Relevance to NASA • Addresses failure mode growth due to increasing system complexity & autonomy • Mini-AERCam TBU for spacecraft inspections • Space shuttle, ISS, CEV, et al • Applicable to all NASA embedded systems • Multiple uses for simulator • Systems & SW Development • Project-level Safety & Mission Assurance • Independent Verification & Validation • Post-launch support

10. Mini-AERCam Project Accomplishments • Project begun in earnest April ’04 • Well into simulator development Next Steps • Complete Mini-AERCam simulator • Write failure mode tests • Test SW response to enhanced failures • Compare empirical results with FMEA results