Contingency Software in Autonomous Systems

1. Terminate Flight SAFE Contingency Software in Autonomous Systems OSMA Software Assurance Symposium July 20-July 22, 2004 Robyn Lutz, JPL/Caltech & ISU Stacy Nelson, Nelson Consulting/QSS This research was carried out at the Jet Propulsion Laboratory, California Institute of Technology, and at NASA Ames Research Center, under a contract with the National Aeronautics and Space Administration. The work was sponsored by the NASA Office of Safety and Mission Assurance under the Software Assurance Research Program led by the NASA Software IV&V Facility. This activity is managed locally at JPL through the Assurance and Technology Program Office

2. Contingency Software in Autonomous Systems Topics • Overview • Goals • Technology Readiness Level • Availability of Data • Approach • Preliminary Results • Work-in-progress • Benefits • Potential Applications • Barriers to Research or Application • Future Work

3. Virtual rotorcraft following APEX plan (green bar) Apex plan Video from Camcorder Video from tracking camera on trailer Video from Color Camera DART DEMO

4. Contingency Software in Autonomous Systems Unique Research Relevant to NASA • Adding intelligent diagnostic capabilities by supporting incremental autonomy • Responding to anomalous situations currently beyond the scope of the nominal fault protection • Contingency planning using the SAFE (Software Adjusts Failed Equipment) approach

5. Contingency Software in Autonomous Systems Overview • Mitigate failures via software contingencies resulting in safer, more reliable autonomous vehicles in space and in FAA national airspace • Enhance diagnostic techniques to identify failures • Provide software contingencies to mitigate failures • Perform tool-based verification of contingency software • Apply results to ARP (Years 1 & 2) and MSL (Years 2 & 3) • Status: Year 1 of planned 3-year study (1/04 start) Current Practice SW Contingency Planning Full Autonomy

6. Contingency Software in Autonomous Systems Technology Readiness Level • Current technology readiness level = 2+ • 2: “Technology concept and/or application formulated” – completed 6/04 • 3: “Analytical and experimental critical function and/or characteristic proof-of-concept” – in-progress (12/04 completion) • Current penetration factor = 8 • Data passed back to project

7. Contingency Software in Autonomous Systems Availability of Data: High

8. Failure WHAT FAILED? Contingency Software in Autonomous Systems Problem Autonomous vehicles have limited capacity to identify/mitigate failures

9. Failure Contingency Software in Autonomous Systems 3 Failure Diagnosis Approach 1 2 SAFE Vehicle (Software Adjusts Failed Equipment) • Enhance diagnostic techniques to identify failures • Provide software contingencies to mitigate failures • Perform tool-based verification of contingency software and • Apply results to ARP (and MSL) to pave the way to more resilient, adaptive unmanned systems Flight Critical Parameters

10. ARP Functional Requirements: Current Planned Contingency Analysis: SFMECA SFTA Contingency Planning: Available indicators Contingency triggers Contingency responses 2-Level (recover/predict) CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Contingency Process Overview Customized the IEEE/EIA 12207.2 Annex I Evolutionary/Spiral Methodology 1. Brainstorm with UAV team to uncover candidates for software contingencies Review UAV literature and project reports Lead brainstorming sessions with domain experts Work with team to identify and prioritize high-concern candidates Select top priority candidates 2. Model unit of interest (i.e. cameras, communications systems…) Model system including: Architecture & State diagram Verify models with UAV team 3. Contingency requirements verification Perform SFMECA 4. Analyze testability Identify how each contingency can be detected Perform SFTA Experiment with assignment of measure of uncertainty 5. Develop recovery strategy Determine candidate strategies for contingency responses (prevent/respond/safe) Determine availability of data needed to determine/execute appropriate contingency 6. Prototype contingency in progressively higher fidelity testbeds 7. Monitor contingency performance

11. CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Related WLAN Work • RF signal strength measurements can be normalized to theoretical values and used to predict range • ( Good correlation and repeatability of signal strength measurements using different antenna configurations and test distances) • Network throughput is reasonably predictable for single hop links at short distances (WLAN link runs under nominal conditions with no packet loss) • However, network throughput is not predictable for complex WLANs consisting of multiple repeater hops or long distances. WLAN links run under conditions of varying packet loss. Packet loss significantly reduces data pipelining by introducing highly variable packet transfer latencies due to packet re-transmission • Packet loss due to multi-path, low signal strength, interference significantly disrupt the timing of packet transfers due to packet re-transmission. • MAC layer uses packets for many purposes such as node authentication, data flow management and data transfer. Packet loss can affect any of these functions resulting in a wide variety of failures. Design of Hybrid Mobile Communication Networks for Planetary Exploration Richard Alena, John Ossenfort, Charles Lee, Edward Walker, Thom Stone

12. CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Perception (Cameras) Perception • Perception is a critical function in systems requiring obstacle avoidance, threat detection, science missions and “opportunistic” discovery. • Optical flow systems use contrasts in the surrounding imagery to determine position. If a vehicle using optical flow flies, for instance, over a very regular terrain such as a grassy field or an empty parking lot, it may crash.

13. Rotorcraft 802.11b PCMCIA card Radio Modem CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS GPS Autonomous flight (Nominal Case) (RC pilot standing by in case of emergency) Onboard Antenna Equipment Rotorcraft Control Center (“Trailer”) Comm. Range (varies) Not to Scale New: Critical communications over radio modem and other communications via WiFi. Reason: Security and bandwidth

14. CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Partial Onboard Architecture Yamaha System APEX Reactive Planner * CLAW Flight Control Laws DOMS Distributed Messaging System *domsD – DOMS transport daemon Telemetry GPS

15. CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Perception (Cameras) Perception • Perception is a critical function in systems requiring obstacle avoidance, threat detection, science missions and “opportunistic” discovery. • Optical flow systems use contrasts in the surrounding imagery to determine position. If a vehicle using optical flow flies, for instance, over a very regular terrain such as a grassy field or an empty parking lot, it may crash.

16. CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Cameras Onboard Rotorcraft Right Wing Gray scale wing tip (stereo vision) Color Camera for situational awareness Color Camcorder Left Wing Firewire Firewire Hub Image Processing System

17. GPS Cameras Scanning Laser Range Finder (SICK) (coming soon) Sonar Range Finder Laser Range Finder (coming soon) CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Other Perception Components Onboard Rotorcraft • SIC (K) – Fast & accurate scanning laser • Laser range finder – returns single point used for precision autonomous landing if GPS signal is lost • Sonar (or Ultrasonic) range finder to determine distance to ground

18. CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Camera Criticality Cases in which the cameras are a critical system: • Cameras assigned responsibility during nominal ops • No line of sight -> Camera provides position info • Cameras are backup when other subsystems fail • Failed/degraded GPS -> Camera provides position info • Failed/degraded ARP -> Camera provides landing-site data • Images as mission objective (surveillance) • Failure of cameras can jeopardize success

19. CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Preliminary Results • Collaborating with Autonomous Rotorcraft Project to experimentally apply approach • Project provides feedback on our models, guidance on future plans • Feasibility check • Reviewed ARP architecture including communications & perception • Proposed initial SW contingencies for communication and perception failures • ARP team including us in team meetings • PM has agreed to try contingencies appearing viable • Finalized SW contingencies for communications & perception with ARP team • ARP team considers further investigation & simulation high priority for 4 identified SW Contingencies

20. CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Preliminary Results • Loss of Communication: • Detect loss of communication revise mission plan: • Reroute • Fly to rally point • Interference with Communication: • WiFi Security • Throttle back communication • Loss of Perception: • Detect camera failure and reconfigure to use another camera • If color camera used for situational awareness fails, then switch to one of the gray scale cameras. • If left wing camera fails then reconfigure to use left wing color camera for stereo vision. • Degradation of Perception: • Change image-acquisition configuration or parameters • If need to lower resource usage, reduce image size • Change image-transmission configuration or parameters • If need to lower bandwidth, drop color, drop frame rate, compress image more (trade off with CPU cycles)

21. CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Benefits • Paves the way to more resilient, adaptive unmanned systems • Supports spectrum of project adoption of autonomy • Flexible: project determines how much autonomy • Incremental requirements (evolutionary process model) • Considers contingencies beyond failures: • Environmental changes that threaten mission (e.g., surveillance) • Changes in resource needs vs. availability that impact mission success (e.g., will need high-bandwidth) • Mobility capabilities that create tradeoffs with communication, imaging optimizations • NASA Experience: Will demonstrate on NASA projects • Anticipated cost savings for projects with evolving autonomy needs • Equips us with a methodology to continue to move toward autonomy

22. CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Towards MSL Risk Assessment for SW Contingencies Example Using DDP tool (fault tree Approach) to assess risk of SW Contingency Plans (collaboration between CSAS & Dr. Martin Feather) Note: example risk numbers relative not absolute – more work required

23. CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Potential Applications • Autonomous Rotorcraft Project (ARC) • Mars Science Laboratory (JPL) • Other autonomous vehicles • Other mobile imaging systems

24. CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Barriers to Research or Application • Challenge 1: ARP is moving target (rapid evolution) Approach: Track planned & unplanned changes via weekly telecons • Challenge 2: Planning for MSL application Approach: Demo benefits on ARP first; select ARP functionalities also important to MSL (communication, perception) • Challenge 3: Tech transfer will depend on ease of reuse Approach: Provide results both in terms of (1) improved verification techniques for contingencies and (2) reusable designs for common contingency applications

25. CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Future Work • Tool-based verification on NASA project • Advance NASA’s information about communications and perception systems for autonomous vehicles