Flight Software Through the Looking Glass

1. Flight SoftwareThrough the Looking Glass Scott Burleigh Jet Propulsion Laboratory California Institute of Technology 12December 2013 This research was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration. (c) 2013 California Institute of Technology. Government sponsorship acknowledged.

2. Flight Software is TWO systems We all know that real-time flight software is indispensable to spacecraft operations. It drives the hardware of a robotic vehicle and its instruments, and it has been present since deployment of the first flight computer. As missions become more capable, a second system emerges. It’s a non-real-time system that manages data.

3. The second system The software we usually think of when we say “flight software” is the real-time, safety-critical, interrupt-driven foreground system. The other part of “flight software” – which we typically don’t recognize as being different in nature – is the non-real-time, discretionary, time-sharing background system.

4. Where it lives Scheduling theory tells us that the real-time tasks’ time allocations should add up to no more than about 60% of CPU time, to ensure they never miss deadlines. The remaining 40% of CPU – “idle time” – isn’t necessarily idle. It’s time that is available for interruptible tasks, the second system. That second system should include everything that is not truly real-time.

5. Second system overview • All tasks run at the same priority, the lowest priority supported by the O/S. • Possibly multiple background subsystems. • To the real-time FSW they all look like “idle”. • Tasks have no deadlines. • Locking one another out for prolonged periods is okay, so long as everybody gets a chance to run eventually. • Each task must release the CPU on finishing a unit of work, usually blocks on something.

6. A Practical Example The Flight Software architecture of the Curiosity rover is clearly an excellent approach to the design of a FSW real-time system (but other good approaches are possible). Postulate: the Interplanetary Overlay Network (ION) architecture, an implementation of Delay-Tolerant Networking, is a good approach to the design of a FSW non-real-time system (but again other good approaches are possible).

7. ION Design Overview Application issue deliver bundles for delivery bundles to be forwarded Routing Object database Sender Receiver transmit receive dispatch

8. ION Design Principles • Use shared memory. • Often there’s no protected memory, so we have no option. • But this can be turned to advantage: shared memory is a highly efficient way to pass data between flight software tasks. give take sender S receiver enqueue dequeue linked list • Zero-copy procedures: leverage shared memory to minimize processing overhead. • Encapsulation in layers of protocol overhead (headers and trailers) can be done by reference rather than by copy. • The same data object can be shared by multiple tasks, provided reference counting prevents premature deletion. • Portability: this is an unfamiliar programming model, so we must make it easy to develop in an environment with good programming support (e.g., Linux) and then deploy – without change – in the target RTOS environment.

9. Through the looking glass The foreground and background systems of a spacecraft’s FSW share many qualities that distinguish them – both – from workstation or PC software. But in some ways the character of the background system is the exact inverse of the character of the real-time FSW. And the two can coexist in perfect compatibility. It’s been demonstrated.

10. What they have in common… • …is most of the flight project cultural values: • Project schedule can’t be jeopardized. • Module coupling must be minimized. • No dynamic system memory allocation. • Make maximum use of available resources. • Contain and, as possible, tolerate faults. • Test as you’ll fly, fly as you tested. • Formal, controlled development process.

11. Where they differ • What is optimized • Determinism • Portability • Mutual exclusion • Data sharing • Message passing • Memory management

12. What is optimized • The foreground system must be reliable, else you lose the spacecraft. • Fixed scope enables minimal, mission-specific design which enables comprehensive testing. • The background system must be efficient, else not enough work gets done. • Minimize wasted space and cycles. • This adds complexity, so it’s important to support portability: reliability comes from extensive multi-mission testing history.

13. Mutual exclusion • Real-time FSW can’t tolerate lengthy mutual exclusions: tasks miss deadlines. • In the background system, no problem. You can serialize an entire subsystem on a single mutex, because there are no deadlines. • Only one task runs at a time anyway. (For now; multi-core flight computers are on the horizon.) • Loops and function calls in critical sections are okay, so long as they eventually terminate and let other tasks run. • Minimize cycles spent on task switching.

14. Data sharing Data sharing requires mutual exclusion, so it’s no good in the foreground system. In the background system, lengthy mutual exclusion is okay – so shared access to data is okay. Which is good, because shared access is also the fastest way multiple tasks can operate on the same data.

15. Message passing • Because shared access is excluded, the real-time system uses message passing to enable multiple tasks to operate on common data. • But because shared access is okay in the background system, message passing is not needed. • No cycles wasted in copying anything. • Signal “data ready” by giving a semaphore.

16. Memory management • Because mission scope is fixed and is known at launch time, all foreground FSW memory can be in fixed-length arrays, each with margin. • But the background system gains reliability from being multi-mission, hence portable and evolvable. • Management of private common heap: pooled resource, pooled margin, efficient use of space. • Automatically adapts to mission scope change.

17. How they collaborate • The foreground system never calls the background system’s library functions – never blocks, ignores background tasks. • Background tasks are interrupted whenever foreground tasks need to run. • For communication between the two: operating system (e.g., VxWorks) message queues. • Non-blocking at the foreground end. • Blocking at the background end.

18. How we know this works libfdput_px.so PxDb PxVdb PX pxout S S LTP segs Out FPDU Outbound FPDUs pxlso To telecom system and earth cfdp_manager LTP pxin S S In FPDU Inbound FPDUs pxlsi • The Deep Impact Network (DINET) project validated ION in flight, on the EPOXI (formerly Deep Impact flyby) spacecraft in October-November 2008. The ION non-real-time data management system was integrated with DI’s real-time flight system, written by Ball Aerospace. No modification to either the Ball flight software or the ION multi-mission software was required; a small “link service adapter” system, named “PX”, provided the interface between the foreground and background systems. • The ION software enabled EPOXI to function as a DTN router in deep space, about 10 million miles from Earth, for four weeks while the spacecraft was in cruise to comet Hartley 2. About 14.5 MB of data (292 images) were routed through the spacecraft over 8 low-rate DSN tracking passes, without data loss and without software failure.

19. Thanks!