Dave Stewart, PhD Director of Research and Systems Integration InHand Electronics

1. Unique Software Considerations in Military and Avionic Embedded SystemsEmbedded Systems Conference Boston 2011ESC-402 Dave Stewart, PhD Director of Research and Systems Integration InHand Electronics dstewart@inhand.com www.inhand.com Embedded Systems Conference Boston, Sept 2011

2. Overview • Most software presentations on Military and Avionic Systems focus on two key items: • Reliability and Fault Tolerance • Security • This presentation focuses on many other aspects of software that are somewhat unique to military and avionic systems, yet are rarely discussed Embedded Systems Conference Boston, Sept 2011

3. Outline • Is that REALLY the Requirement? • Self Destruct! • Development Environment != Deployed System • Sorry, no debug I/O allowed, but debug it quick! • Error detection: Why bother? • It Uses Too Much Power! Embedded Systems Conference Boston, Sept 2011

4. Is that REALLY the Requirement?Understanding the Real Requirements • Hundreds of pages of requirements • This is typical for military and avionic applications • If only tens of pages, first step is likely a need to refine the requirements! • Separate between “wish-list” and “really really required” • Many requirement documents are the culmination of the wish-lists of everyone involved in prior projects • Given fixed cost contracts, understand which ones have flexibility, and which do not • The systems engineer should look closely for any conflicting requirements; address those first! Embedded Systems Conference Boston, Sept 2011

5. Is that REALLY the Requirement?The Impossible Requirements • It is not unusual to find requirements that seem like they will be very difficult to meet • When this happens, investigate the requirement carefully, and make sure what the customer wants is properly captured • Understand every Use-Case Scenario. Often the documented requirement does not actually meet the needs of the use cases • Yet, a simpler requirement might actually meet the needs of all use cases • Review how that requirement was implemented in previous or competitor projects • Some requirements are designed to capture what was implemented on a prior system, whether in-house or by a competitor • Review those other implementations closely, as more often than not, the customer really wants the same thing, not something new that is substantially more complex Embedded Systems Conference Boston, Sept 2011

6. Self Destruct!Zeroization • Many requirements call for the zeroization, but the actual meaning is left to interpretation. • The interpretation is often “erase flash” upon either user activation or detection of tamper • This is often overkill. The real requirement imposed by military and security agencies is usually that an enemy who gains hold of the device is unable to access any classified data • Therefore unencrypted classified code and data must be erased quickly and reliably • Ask the customer • Do you really need to erase an operating system that is readily available in the public domain? Embedded Systems Conference Boston, Sept 2011

7. Self Destruct!Understand the Use Cases • If the customer insists they mean erase all flash, challenge their understanding of the implications • This is often not what the end user really wants, even if they are asking for it • During requirements reviews, understand ALL the use-cases. • For example, what if there is a false alarm? Erasing all flash will erase the OS and Bootloader, meaning the device becomes useless and must be returned to manufacturer. • This simply can’t happen if system is deployed in the war zone. When such use-cases are discussed with the customer, better (aka easier to implement) solutions are usually found. Embedded Systems Conference Boston, Sept 2011

8. Self Destruct!Erasing the Code and Data • Unencrypted classified code and data must be erased before an enemy could get to it • Erase Times: • NOR Flash: Minutes, needs power • NAND Flash: Tens of seconds, needs power • DRAM: Under 5 seconds, does not need power • Crowbar DRAM: Milliseconds, does not need power • Simplest solution to meet requirements: put unencrypted classified code and data in crowbar DRAM only! • Only store strongly encrypted data in flash • Decrypt data into DRAM only • Which encryption to use: a topic beyond the scope of this talk. • This is one of the topics that is covered often by others Embedded Systems Conference Boston, Sept 2011

9. Self Destruct!Crowbar DRAM • Cheapest, fastest, and most reliable means of erasing • Even if power is removed from system, method works reliably • Simple Solution: DRAM hardware is modified so that a switch turns off Vcc to DRAM, and at the same time shorts Vcc to ground. • In cases of “false alarms” does not render the device into a brick Embedded Systems Conference Boston, Sept 2011

10. Self DestructErasing Flash • A software-only solution will not work. • All it takes is removal of power to stop the erasing, at which time other techniques can be used, such as removing flash from PCB then reading it independently. • Some secure flash drives and memories will latch internal controller • Once erasing begins, it cannot be stopped. Removing power only pauses the erase. • Any attempt to power the chip to read will only resume the erase operations. • Caveats: • These are expensive flash memory options! • Understand the implications: what happens to device if feature is triggered? Embedded Systems Conference Boston, Sept 2011

11. Development Environment != Deployed SystemWhat do you need for software development? • It is typical for hardware requirements to drive the selection, design, and fabrication of hardware • From the first revision, hardware will incorporate all the required features needed in the end product, such as no user-accessible debug ports, no simple unsecure high-speed access, no keyboard or mouse, etc. • How is software developed for these systems? Embedded Systems Conference Boston, Sept 2011

12. Development Environment != Deployed System Software Team: Be part of Hardware Development! • The software team must be part of the requirement reviews and early design meetings. • Assume that every required feature will have a bug and needs to be debugged and fixed. • Ensure that common tools used to troubleshoot such errors are available, otherwise software costs will go through the roof, and high levels of reliability will be unachievable • Coordinate with hardware designers a plan for scaling back development features as the product matures and gets closer to production Embedded Systems Conference Boston, Sept 2011

13. Development Environment != Deployed System “Back Door” Mechanisms • Software engineers like to put in “back doors” to allow easy access to troubleshooting and debug utilities. • Such features, however, are frowned upon in military and avionic systems, because they are prime entry points for hacking, security breaches, and often contain unreliable code. • Presence of these mechanisms will usually cause a system to not meet certification requirements. • But absense of the mechanisms can greatly increase software maintenance costs. Embedded Systems Conference Boston, Sept 2011

14. Development Environment != Deployed System “Front Door” Mechanisms • Instead, consider “front door” mechanisms. • Don’t try to hide these entry points, where they tend to be unmanaged and not well secured. • Rather, design them into the system from the start, to undergo the same review scrutiny as any other feature • Share the planned usage and lockouts with the customer and end user. Often, they will want to use the features too during early development. • Include the necessary features to disable them when necessary, and securely re-enable them when they are needed. • Include a means to componentize the front door mechanisms, so that it can also be easily removed in final production versions. Embedded Systems Conference Boston, Sept 2011

15. Development Environment != Deployed System Debug Boards • Work with hardware designers to bring all the desired interfaces that customer does not want to a high-density debug connector • e.g. debug serial ports, USB ports, JTAG, spare GPIOs, LEDs, inter-chip communication signals, or anything else that would be useful to the software developer • Create a separate “debug board” that expands the signals into standard connectors • Hardware team might balk at cost of such a board, which is probably in the $5000 range • However, existence of such a board can save 10 to 100 times that cost in reduced software development effort • Standardize the high-density debug connector, thus making debug board reusable across multiple projects Embedded Systems Conference Boston, Sept 2011

16. Sorry, No Debug Allowed!Once unit is packaged, say bye to the debug board • The debug board helps reduce initial development, and can be used for troubleshooting any issues that can be replicated on an open system • Once the system is packaged, that key development interface is no longer available. Alternate mechanisms are needed to troubleshoot issues. Embedded Systems Conference Boston, Sept 2011

17. Sorry, No Debug Allowed!Log Files are the key! • Continuously collect data on every aspect of the system • Budget the amount of space needed to store log files early during system design • Store data on flash, for retrieval either by opening unit, or through an existing customer communications link • Do not store any classified data in the log files! Embedded Systems Conference Boston, Sept 2011

18. Sorry, No Debug Allowed!Not all log files are equal • Quality of log files is of extreme importance! • It can eliminate the reliance on customer descriptions of the problems. • Often, their observations are inaccurate, or they provide their interpretation of the problem. • With just the right information in a log file, many issues occurring in the field can be identified • This is absolutely essential, especially in avionics • Avoids the need to have to replicate the issue to debug it • Carefully review what goes into the log file • A log file contains poor information if every time a problem occurs, the log file only provides minimal clues to the actual problem Embedded Systems Conference Boston, Sept 2011

19. Sorry, No Debug Allowed!Concise Log Files • Memory reserved for log files can fill up quick • Optimize memory usage as part of the design. • Be cryptic, that’s fine as long as there is easy way to decipher the log file • “Average of velocity for past 30 seconds = 3.49231” • uses 40 bytes of memory • “av30=3.49231” • uses 12 bytes of memory • Both the above provide equal amount of information, but second one is more concise, and log is searchable by unique acronym “av30” • Store in binary: 0xaa30 0x405F8202 • Uses 6 bytes of memory • 16-bit number as a key allows for 65536 separate types of log entries • 32-bit floating point number is only 4 bytes of data • Create scripts to quickly search or filter desired data • If this is not easy, then the uniqueness of keys is not adequate Embedded Systems Conference Boston, Sept 2011

20. Sorry, No Debug Allowed!What information should go into Log Files? • The more information stored, the more likely a single capture of a log file is sufficient to pinpoint a problem that occurred in field • Timestamp EVERYTHING! • Microsecond resolution is desirable. • Millisecond resolution is essential. • Relative time, stored as a 16-bit or 32-bit number is usually acceptable. Don’t use more than 4 bytes per timestamp, or memory will fill up too quickly. Embedded Systems Conference Boston, Sept 2011

21. Sorry, No Debug Allowed!What information should go into Log Files? • Periodically log the “state” of the system, including the following: • Any code that is built as a finite state machine, include the ‘mode’ or ‘state’ • The current value and setting of every GPIO pin • Current value of every key register • Current value of any key global variables • Current value of any sensor reading • Current value of any controller output Embedded Systems Conference Boston, Sept 2011

22. Sorry, No Debug Allowed!What information should go into Log Files? • Summary of every communication message that is sent or received: • Includes Ethernet, WiFi, USB, SPI, etc. • Save key information, such as sender or receiver, size of message, command or message type. • For short messages (i.e. a few bytes) include entire message. • For longer messages, include first few data bytes. Embedded Systems Conference Boston, Sept 2011

23. Sorry, No Debug Allowed!What information should go into Log Files? • Every Output I/O Device Driver Call: • E.g. Instrument the stream interface, so that every Write operation is captured • Save timestamp, device, port, and value written • Don’t need to log inputs or reads unless it is a message (see previous slide). For sensors, the state data logs the values periodically, instead of every time the device driver is called. Embedded Systems Conference Boston, Sept 2011

24. Sorry, No Debug Allowed!What information should go into Log Files? • Every Interrupt • Timestamp of entry into and exit from ISR • IRQ number • Running counter of how many interrupts of this type received • If this is a shared IRQ, which device was interrupting Embedded Systems Conference Boston, Sept 2011

25. Sorry, No Debug Allowed!What information should go into Log Files? • Every Context Switch • Timestamp of entry into each thread • Thread ID • Timestamp of any entry into idle state or thread • The line of code to log this event needs to be in the operating system scheduler, right before the context switch occurs Embedded Systems Conference Boston, Sept 2011

26. Sorry, No Debug Allowed!How can we guarantee all software logs what it needs? • Log files are most effective when there are no “gaps” in information, resulting from one or more modules not logging like they should, or disabling the logs if they have the ability • When possible, put code to log information into kernel code, such as drivers or operating system functions, where they cannot be touched or changed by application programmers. • Logging is then enabled always for everything, with no ability to disable it. Embedded Systems Conference Boston, Sept 2011

27. Sorry, No Debug Allowed!Wow, isn’t that too much to log! • Estimate log file size based on typical usage.Here is an example • State of system, 400 bytes, once per second • Messages, 16 bytes per message, 500 messages per second • I/O Driver Writes, 12 bytes per call, 200 times per second • Interrupts, 12 bytes per interrupt, about 800 times per second • Context Switches, 8 bytes per switch, 200 times per second • Above scenario = 400+12*800+12*200+8*200+16*500 = 22 kBytes per second= 1.3 Mbytes per minute= 80 Mbytes per hour= 2 GBytes per day Embedded Systems Conference Boston, Sept 2011

28. Sorry, No Debug Allowed!Wow, isn’t that too much to log! • In past systems, this was a problem. In today’s system, this is easy. • E.g. add a 2 GB micro-SD card solely for logging, and the full log for any 24-hour period is possible. • This is worst case • might only need to log for 8-hour days, and only a fraction of the above, as system might only really be in use 10% of the time during the day and idle or in a sleep mode rest of the day, thus bringing actual log requirements down to under 100 Mbytes per day, or 20 days of storage with the same 2GB card. • Use a circular buffer • discard oldest data when maximum data usage has been reached. • Store multiple small files instead of one big file. • E.g. start a new file every 5 minutes. • Makes it easier to retrieve only partial logs within specific time periods. Embedded Systems Conference Boston, Sept 2011

29. Sorry, No Debug Allowed!Can’t afford the performance loss! • With proper optimization of log code, the logged data should not use more than 2% of the processing power. • E.g. Can write SD at 1MByte/sec, only need to write 22KByte/sec worst case, thus SD write is active less than 2% of the time. • Use a DRAM FIFO • most logging is done in kernel space, where file access is not available • Write to DRAM FIFO, overhead is negligible. • Once per second, have a high-priority system-level thread read FIFO and write to log file. • Only systems that are 98% loaded or more won’t be able to support this added overhead. If the system is that loaded, use a faster processor, or optimize one thread to reduce CPU usage by 2% • Consider the cost of NOT having this data when a problem occurs in a field unit that needs to be fixed. The 2% overhead is well justified! Embedded Systems Conference Boston, Sept 2011

30. Error Detection, Why Bother?What is right amount of error detection and handling? • What percentage of your code is error detection and handling? • Do you think that is too little? Too much? Just right? • How is more or less error detection and handling code justified? Embedded Systems Conference Boston, Sept 2011

31. Error Detection, Why Bother?Do I need to detect all errors? • Don’t waste time detecting errors for which the error handling is “blue screen of death” • More often than not, this elevates simple errors into fatal errors • Only detect errors when it is possible to create a handler that recovers gracefully from the error • Otherwise, let the logging mechanism simply log the information as part of its normal duties • Error handling code must be part of the design documents • Adding ad-hoc error handling often causes more harm than good, because the code is rarely tested adequately, and only gets used when there are already problems in the system. Embedded Systems Conference Boston, Sept 2011

32. It Uses Too Much Power!Battery and Power Management • Mobile military applications and avionic applications have one thing in common: • Power consumption is ALWAYS too much, so reduce it more! • Aircraft need low power to reduce energy needs, and often just as important to reduce heat production within the closed body • Military applications need devices to run on battery for entire missions, as the ability to recharge is often at best a few hours each night Embedded Systems Conference Boston, Sept 2011

33. It Uses Too Much Power!Battery Life • Capacity is specified in mAh • mAh = milli-Amp-Hours • E.g. 1400 mAh means the battery can drain 100mA for 14 continuous hours • Or it can draw 1.4A for one hour • By measuring current when battery is discharging, can compute power consumption • Power = Battery_Voltage * Discharge_Current Embedded Systems Conference Boston, Sept 2011

34. It Uses Too Much Power!Battery Life • Battery capacity only guaranteed for “new” batteries • After a battery has been recharged a few dozen times, it starts to age • As it ages, capacity decreases. • The faster you discharge the batteries, typically the shorter its lifespan • Check battery datasheet. Some batteries may have capacity reduced by 50% in just a few hundred recharge cycles. • This means less than a year if recharged daily. Embedded Systems Conference Boston, Sept 2011

35. It Uses Too Much Power!Battery Level • For Alkaline batteries, voltage is a reasonable gauge of remaining capacity • For Lithium type batteries, voltage level of battery decreases relatively slowly • thus voltage is not indicative of remaining life Embedded Systems Conference Boston, Sept 2011

36. It Uses Too Much Power!Battery Level • Li-Ion and Li-Polymer batteries typically use a “fuel gauge” to measure battery level • The gauge counts coulombs • When charging, the count increases • When discharging, the count decreases • Battery level is only accurate when the capacity is “calibrated” • Once a battery starts aging, a battery level algorithm may require the battery to be fully discharged, then reset the algorithm to indicate a new battery (even though it is old), then fully charge. This allows the new capacity to be re-learned. • Some drivers auto-calibrate anytime a full charge occurs • Actual algorithm is hardware dependent • If battery level does not seem to be accurate, verify that drivers are properly reading the fuel gauge, and that there is some form of recalibration for aging batteries Embedded Systems Conference Boston, Sept 2011

37. It Uses Too Much Power!Reduce Power: Go For Idle • Idle systems can offer best power savings • Sometimes, running FASTER can reduce power, by getting back to idle quicker • E.g. SPI driver with busy-wait loops. Run SPI clock as fast as possible, to get back to idle more quickly • For processor, running faster usually means more power • Slow down periodic threads • Read an A/D 50 times per second instead of 100 times per second, if application allows it • Eliminate unnecessary services and interrupts • If system is not being used for 1 minute, capture every interrupt that occurs or every service that executes. • Ask WHY they execute, if system is idle? • Coming out of idle often is costly to battery life. Embedded Systems Conference Boston, Sept 2011

38. It Uses Too Much Power!Reduce Power: Go For Idle • Idle Implementations • Idle Thread • Lowest-priority thread has an empty while(1) loop. • Anything that becomes active preempts this thread. • Hardware-independent, thus easily portable to any processor • Power-hungry; processor is still running at full tilt • CPU Idle Mode • Most modern processors have an Idle Mode • By installing hardware-dependent support to use this mode, significant power savings can be achieved when idle • Deep Idle • Turn off the processor oscillator when idle • May take 100 usec extra to wakeup • But the power savings can be significant • Extra care needed if DMA running in background, e.g. for Display • If turning off oscillator not an option because it breaks some devices, consider just slowing down clock when idle Embedded Systems Conference Boston, Sept 2011

39. It Uses Too Much PowerSuspend and Shelf Modes • Suspend (aka Sleep) • Turn off processor and most peripherals • Keep DRAM powered • State of each I/O pin controlled by software • Enables resuming within a second • Shelf (aka Deep Sleep) • Turn off processor and all peripherals • Turn off DRAM • State of each I/O pin is undefined • Resuming requires rebooting • Used to maintain enough power for a “soft” power-switch, and perhaps maintain an RTC • Hibernate is a form of shelf mode, except that the contents of DRAM are first saved to the file system Embedded Systems Conference Boston, Sept 2011

40. It Uses Too Much PowerSuspend and Shelf Modes • Suspend Mode Power • Software has significant impact on power consumption during this mode, even though CPU not running • Devices not properly turned off or put into low power modes may continue to draw power • I/O pins not placed in the right state may cause power drain • Shelf Mode Power • Software has no control of this. • If this power consumption is too high, hardware engineers need to investigate Embedded Systems Conference Boston, Sept 2011

41. It Uses Too Much PowerSuspend and Shelf Modes • Manage entry into suspend mode • Use the xxx_power_off() of the driver • Start by saving the context • In addition to saving context: • if the device has a low-power mode, enter it; or • if the device has a power-enable, disable it • if turning off power, hardware engineers should verify there is no backfeed, as that will unnecessarily consume power during the sleep mode • Ensure that any output signal to that device is placed into its inactive state • Note that for active-low signals that typically have a pull-up on the line, this means set the output HI • When exiting suspend mode • Do opposite steps in xxx_power_on(), in reverse order • Exit low-power mode or re-enable power • End by restoring the context • If device was powered off instead of a low-power mode • It might be necessary to re-initialize the device, as any internal state would have been lost. Embedded Systems Conference Boston, Sept 2011

42. It Uses Too Much PowerDevice Low-Power Modes • When a device is not needed, it should be placed into a low-power mode • In theory, the xxx_power_off() function of the driver is what should be called to turn off that device • However, that only gets called during a suspend operation • For high-powered drivers, create an IOCTL to execute the commands of the power-off function under user control • Create a second IOCTL to re-enable the power Embedded Systems Conference Boston, Sept 2011

43. It Uses Too Much PowerDuty Cycling • Duty cycling high-powered items can lower average power consumption • A 50% Duty Cycle of a device can reduce consumption in half for that device • This means ON half the time, then either OFF or in low-power mode the other half • Look at each individual hardware item to determine if there is an opportunity. • LEDs • Analog Inputs • Communications Embedded Systems Conference Boston, Sept 2011

44. Summary • Is that REALLY the Requirement? • Review requirements early! • Unrealistic requirements can often be negotiated • Self Destruct! • Don’t erase everything, that is overkill! • Put unencrypted classified data into DRAM • Development Environment != Deployed System • Just because end requirements means no debug does not mean prototypes can’t have it • Create a debug board with favorite signals going to a high-density connector Embedded Systems Conference Boston, Sept 2011

45. Summary • Sorry, no debug I/O allowed, but debug it quick! • Continuously generate log files • Design the content to manage memory needs • Error detection: Why bother? • Only detect errors that can be handled gracefully • No blue screens of death! • It Uses Too Much Power! • Understand how battery power really works • Leverage idle, sleep modes, and duty cycling Embedded Systems Conference Boston, Sept 2011