fsfa

1. Embedded Systems and Software Safety, Ethics, Professionalism Titan IV Rocket Explosion Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-1

2. Safety, Ethics, Professionalism • Embedded systems are being increasingly used in critical applications – Safety-critical potential for death or injury loss or damage to property – Societal-critical potential for major disruption to everyday life – Business-critical potential for large economic loss to company/business exposure to litigation Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-2

3. Role/Responsibility of Embedded System Engineer • Act professionally and ethically • Understand and appreciate risk factors • Insure that proper attention is paid to these factors in system design and implementation – Interlocks – Cross-checks – Independent monitoring of critical functions – etc. • Insure that thorough analysis and testing is done – Long-term – Anomalous situations Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-3

4. Ethical Responsibilities • Errors of Commission – Falsifying test results – Covering up known or suspected problems – Knowingly leaving potentially serious bugs in a product – Knowingly violating safety/reliability standards • Errors of Omission – Failure to adequately analyze potential failure modes and consequences – Failure to adequately test a system – Failure to design and implement appropriate safety/robustness features – Failure to react aggressively enough to reports/warnings of possible problems – Others??? Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-4

5. Three Quick Case Studies • Therac-25 Radiation Therapy Accidents • Ariane 5 Rocket Failure • Patriot Missile System Failure Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-5

6. The Therac-25 Accidents • Computerized Radiation Therapy Machine • Designed in early 1980s • Resulted in Six Major Accidents between June 1985 and January 1987 – Patients received ~100 times the intended radiation dose – At least two deaths – Several serious injuries • These accidents highlighted the dangers of software control of safety-critical systems, and they have become a standard case study in health informatics and software engineering. Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-6

7. Therac-25 Overview • Linear Particle Accelerator • Replaced earlier version • Utilized much more computerized control • In particular, more software responsibility for safety maintenance • Reused some software from earlier versions • Fault analysis considered only computer hardware failures Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-7

8. Therac-25 Accident History • First accidental overdoses reported in 1985 • Manufacturer could not reproduce accident scenarios • Suspected hardware (microswitch) problems and did redesign • Did not include independent interlock to prevent overdose Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-8

9. Therac-25 Accident History • Accidents continued in 1986 and 87 • Traced to operator behavior (keyboard entry) – Timing related, so-called “race condition” • Several different software problems eventually implicated – Related to concurrency – Lack of locking/atomic operations for access to shared variables Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-9

10. Race Condition, Fault, Hazard A race condition or race hazard is a flaw in an electronic system or process whereby the output and/or result of the process is unexpectedly and critically dependent on the sequence or timing of other events. The term originates with the idea of two signals racing each other to influence the output first. Race conditions can occur in electronics systems, especially logic circuits, and in computer software, especially multithreaded or distributed programs. Consider a two input AND gate fed with a logic signal A on one input and its negation, NOT A, on another input. In theory, the output (A AND NOT A) should never be high. However, if changes in the value of A takes longer to propagate to the second input than the first when A changes from false to true, a brief period will ensue during which both inputs are true, and so the gate's output will also be true. Reference: en.wikipedia.org/wiki/Race_condition Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-10

11. Therac-25 Retrospective • Embedded software was designed and implemented by one engineer • Fairly stringent real-time constraints – Preemptive schedule – 100 ms scheduling granularity • No real synchronization of access to shared variables • Many potential race conditions, but with relatively low probability of occurrence • Most accidents ultimately traced to synchronization problems (race conditions) in processing operator input from the keyboard – Resulted in improper settings – No independent feedback to warn operator Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-11

12. Therac-25 Retrospective • Ethical Issues. The vendor appears, at the very least, to be guilty of: – Inadequate software-engineering practices – Inadequate/flawed fault analysis – Failure to implement needed cross-check and/or interlock features to prevent accidental overdose – Failure to react aggressively to initial reports of problems Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-12

13. Ariane 5 Rocket Failure • First test launch of French Ariane 5 rocket • June, 1996 • Self-destructed due within 40 seconds after lift-off due to software anomaly • YouTube: http://www.youtube.com/watch?v=z-r9cYp3tTE • The rocket was on its first voyage, after a decade of development costing $7 billion. • The destroyed rocket and its cargo were valued at $500 million. • Titan IV Rocket – YouTube: http://www.youtube.com/watch?v=ZFeZkrRE9wI Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-13

14. Ariane 5 Rocket Failure—Cause • Floating point exception generated after lift-off by a software module that was concerned only with missile/launchpad alignment prior to launch • Alignment module was reused from earlier guidance system • Module remained operational for 50 seconds after launch – This had been a requirement in the earlier application • Testing procedures had not considered the behavior of the alignment module after lift-off • Interestingly, the rocket guidance system was completely replicated to protect against hardware failures, but this was of no use for this failure condition Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-14

15. Patriot Missile System Failure Scud Patriot Missile • Patriot Missile Defense System used for first time in first Gulf War (1991) • Failed to intercept incoming Scud missile • Missile struck U.S. Army Barracks, killing 28 soldiers and wounding over 100 • GAO investigation ultimately implicated software Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-15

16. The Patriot Missile Software Problem • System’s internal clock measured time in 100 ms units • Multiplied by 1/10 to convert to seconds • Calculation performed using a 24 bit register. Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-16

17. The Problem • 1/10 = 1/24 + 1/25 + 1/ 28 + 1/29 + 1/212 + 1/213 + … • Truncated to 24 bits to fit into register • Resulted in cumulative timing error • For each calculation this error was very small: – Approximately 0.000000095 • However, after 100 hours of operation, this error was approximately 0.34 seconds • A Scud travels more than 1 kilometer in this time • Error resulted in Scud being outside of the Patriot’s “range gate” • Interestingly, this error had been corrected in some parts of the Patriot’s software but not in others. This is what ultimately caused the failure Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-17

18. What Can We Learn From These Case Studies? • Note that the specific causes were quite different in each failure situation • All systems underwent stringent design review and testing • So, what went wrong? Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-18

19. Ethical Responsibilities • Errors of Commission – Falsifying test results – Covering up known or suspected problems – Knowingly leaving potentially serious bugs in a product – Knowingly violating safety/reliability standards • Errors of Omission – Failure to adequately analyze potential failure modes and consequences – Failure to adequately test a system – Failure to design and implement appropriate safety/robustness features – Failure to react aggressively enough to reports/warnings of possible problems – Others??? Hopefully not this… Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-19

20. Ethical Responsibilities • Errors of Commission – Falsifying test results – Covering up known or suspected problems – Knowingly leaving potentially serious bugs in a product – Knowingly violating safety/reliability standards • Errors of Omission – Failure to adequately analyze potential failure modes and consequences – Failure to adequately test a system – Failure to design and implement appropriate safety/robustness features – Failure to react aggressively enough to reports/warnings of possible problems – Others??? Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-20

21. What Does All Of This Have To Do with Ethics and Professionalism? • Ethics-Personal code of behavior – Responsibility – Accountability – Determination of right vs. wrong • Engineers are often in the best position to determine the ethical consequences of their actions (or inactions) or those of others with whom they interact. Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-21

22. Examples of Ethical Issues • Have sufficient safety/robustness features been designed into this system? • Has it been adequately validated and tested? • Have corners been cut? • Has engineering integrity been compromised by cost or marketing considerations? • Have problems been covered up? Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-22

23. Special Ethical Issues for Safety Critical Systems • How much is a life worth? • What is the proper balance between product safety and cost? – Would you allow yourself to be pressured into removing a safety feature if you were convinced that this could eventually result in death or injury? • Are any systems really non-safety critical? • What about other forms of criticality: societal, economic? Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-23

24. Professional Ethics • Professional societies have ethics codes – IEEE – ACM – ASCE – etc. Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-24

25. Embedded Systems and Software, 55:036. The University of Iowa, 2013 Ethics-25