1 / 47

High-Pressure Steam Engines and Computer Software

High-Pressure Steam Engines and Computer Software. Nancy G. Leveson International Conference on Software Engineering, Melbourne. Introduction. Computers are increasingly being used in safety-critical systems What is the possible contribution of software in such accidents?

Samuel
Download Presentation

High-Pressure Steam Engines and Computer Software

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. High-Pressure Steam Engines and Computer Software Nancy G. Leveson International Conference on Software Engineering, Melbourne CS 851 Forensic Software Engineering

  2. Introduction • Computers are increasingly being used in safety-critical systems • What is the possible contribution of software in such accidents? • GOAL: Need to ensure that computers are introduced in safety-critical systems responsibly. • CONSTRAINT: Risk always exists with technological innovation. CS 851 Forensic Software Engineering

  3. Introduction (continued) • We should learn from our past mistakes. • Theme of this paper: Parallelism exists between early development of high-pressure steam engines and software engines. CS 851 Forensic Software Engineering

  4. Exploding Boilers: History • Hero of Alexandria (60 AD) did some first research into steam-power. • Problem of pumping water out of mines – search for steam power became a necessity around 16th-17th century. • Cylinder and piston engine (Newcomen, 1700) CS 851 Forensic Software Engineering

  5. The Steam Engine • James Watt (1786) worked on improvement of the Newcomen engine. • Teamed up with Matthew Boulton. • Came up with a design that transformed the industry. • Introduced revolutionary changes in transportation. CS 851 Forensic Software Engineering

  6. The Steam Engine James Watt CS 851 Forensic Software Engineering

  7. The Steam Engine CS 851 Forensic Software Engineering

  8. The Steam Engine • The Watt engines used low-pressure steam (5 -15 psi). • Higher pressure could yield more poweful and economic engines. • However, it increased the danger of explosion – Watt was opposed to this idea. • Their patent expired in 1800. CS 851 Forensic Software Engineering

  9. The Steam Engine • Evans (US) and Trevithick (England) designed engines that dispensed condensers and used steam directly to a push a piston. • High-pressure engines required greater than atmospheric pressure to work. • Applied on steamboats – resulted in frequent and disastrous explosions. • Accidents also occurred in industrial systems. CS 851 Forensic Software Engineering

  10. The Steam Engine • The forces of economy demanded more and more steam power. • The trend towards greater efficiency and power also increased the risk of explosion. • Watt campaigned against the use of high-pressure steam engines. • However, his competitors did not share his viewpoint. CS 851 Forensic Software Engineering

  11. Failure Causes • The explosion of boilers was the real cause of failure. • Lag between the technological development of boilers and the rapid improvement of engines. • Specifically, engineers knew about: thermodyanmics, action of steam in the cylinder, strength of materials (in the engine). CS 851 Forensic Software Engineering

  12. Failure Causes • Little scientific knowledge about: buildup of steam pressure in the boiler, effect of corrosion/decay, and the causes of boiler explosions. • Excessive strain on the boilers due to high steam pressure: the existing design was definitely obsolete. CS 851 Forensic Software Engineering

  13. Safety features introduced • Safety valves to reduce steam pressure. • Fusible lead plugs that were supposed to absorb heat from the system. • These fixes did not work. WHY? • Analogy to software: A patch introduces more bugs. CS 851 Forensic Software Engineering

  14. Reasons for failures • Dynamics of steam generation not well understood. • Working environment of the steam engines not considered. • Skill of the operators and maintainers miscalculated. • The designs assumed rational behavior on part of the operators and the owners. • Economic incentives were given to override the safety devices. CS 851 Forensic Software Engineering

  15. Exploding Boilers • There was opposition to using high-pressure steam. • However, legislation to curtail the use of this new technology did not materialize for several years. • A Select Committee was created in England to report on the dangers of high-pressure steam (1817). (Steam Engine invented in 1786) CS 851 Forensic Software Engineering

  16. Exploding Boilers • 2,562 people were killed and 2,097 people were injured between 1816-1848 in a total of 233 steamboat explosions in the U.S. • Property losses of about $3 M. • The Franklin Institute (Philadelphia) began a six-year study of boiler explosions in 1824. • Several other reports came out that looked at the causes of explosions. CS 851 Forensic Software Engineering

  17. Exploding Boilers • The bias against government regulation changed – laws were passed for compensation of people killed in accidents. • Still, no inspection criteria or qualifications for engineers existed. • Finally, in 1852 Congress passed a law that corrected the problems for steamboat boilers. • Problems still with locomotive/stationary boilers. CS 851 Forensic Software Engineering

  18. Exploding Software? • We are again faced with a new technology. • Great economic incentive to use computers for controlling dangerous systems. • Let us examine the parallels from steam engine development. CS 851 Forensic Software Engineering

  19. Comparison I: Tech Lag • “Boiler technology lagged behind improvement in steam engines themselves.” • Lag behind the development of hardware and software. • Two solutions to this problem: 1. Keeps things Simple (Increase the complexity slowly as we learn from our experiences). 2. Dampen our enthusiasm in computers. (Why?) CS 851 Forensic Software Engineering

  20. Comparison I: Tech Lag • Keeping things simple • Example: Ontario Hydro – The first license in Canada for a completely computerized nuclear plant shutdown. • How they did it? 6000 lines of code in the software. Uses the simplest and most straightforward coding methods. CS 851 Forensic Software Engineering

  21. Comparison I: Tech Lag • Hardware fail-safe devices are present to deal with software errors. • Formal and Informal verification of the code is easy due to its simple design. CS 851 Forensic Software Engineering

  22. Comparison I: Tech Lag • CONTRAST: • A similar system in England has a 100,000 lines of code, involves 300-400 microprocessors and contains both control and shutdown functions. • It is difficult to do a verification for this piece of code. CS 851 Forensic Software Engineering

  23. Comparison I: Tech Lag • Point 2: Confidence in computer systems? • Hardware safety devices/mechanisms are being increasingly eliminated and being replaced with software here. • This violates the principle that a single-point failure should not be able to bring down the system. • Example: Therac-25 CS 851 Forensic Software Engineering

  24. Comparison I: Tech Lag • Does the class have any other examples in this context? • The Right Approach: Watt was against high-pressure steam. Edison was against high-voltage electricity. But Thompson suggested a fix by suggesting safety devices to reduce accidents. CS 851 Forensic Software Engineering

  25. Comparison II: Scientific Cause • “There was little scientific understanding of the causes of the boiler explosions.” • Scientific foundations of our field are still being developed. • We are building tools without rigorous proof or scientific basis. • We need to validate and assess our hypotheses using scientific principles. CS 851 Forensic Software Engineering

  26. Comparison II: Scientific Cause • Accumulating engineering knowledge through trail-and-error is OK, but a slow process. • We should try to build up on the vast accumulated knowledge that exists. • Two stages in the early stages of a new technology. - Invention - Evaluation CS 851 Forensic Software Engineering

  27. Comparison II: Scientific Cause • We have not concentrated on evaluation as much as we should have. • Low-pressure steam engines vs High-pressure steam engines. • Need to develop a good theoretical foundation for software engineering. • Need to develop criteria for designing specification languages. • Understand conflicts and trade-offs – examples ? CS 851 Forensic Software Engineering

  28. Comparison II: Scientific Cause • Theoretical foundations can provide: • Criteria for evaluation • Means of comparison • Theoretical limits and capabilities • Means of prediction • Underlying rules, principle and structure. CS 851 Forensic Software Engineering

  29. Comparison II: Scientific Cause • How do we build this? • Building mathematical models and theories ? • Carefully-designed experiments – example? • Need to also look at software as both a mathematical and a human object. • Example – Formal Methods CS 851 Forensic Software Engineering

  30. Comparison III: Safety devices • “The safety features designed for the boilers did not as well as predicted because they were not based on scientific understanding of the causes of accidents.” • The methods used to deal with errors are based on erroneous assumptions. • Example: NVP CS 851 Forensic Software Engineering

  31. N-version programming • Separate teams write multiple versions of the program. • The majority answer is selected. • Based on the N-modular redundancy technique in fault tolerance. • This was not developed for design errors. • Assumption of independence of failures. CS 851 Forensic Software Engineering

  32. “Software Diversity” • Diversity in hardware just does not happen, you have to design it in. • Assumption: Components have different failure modes. (in electronics or hydraulics) • Not satisfied by multiple software versions. CS 851 Forensic Software Engineering

  33. Labeling and definition • There is a tendency to label a technique with the property we hope to achieve with it. Example: “Production-Rule System” • Proof by definition should be avoided: Example: Defining fault tolerance by redundancy. This limits our solution space for the problem. CS 851 Forensic Software Engineering

  34. Comparison IV: Environment • “The introduction of safety devices for steam engines was inhibited not only by the lack of underlying scientific knowledge about boilers, but also by a narrow view of attempting to design a technological solution without looking at the social and organizational factors involved and the environment in which the device is used.” CS 851 Forensic Software Engineering

  35. Situational Awareness • Example: Expert system introduced to aid the maintenance staff of a major airline – the quality of maintenance fell. • Too much dependence on commuters leads to decreased situational awareness. • Thus, it is essential to take human factors into account. CS 851 Forensic Software Engineering

  36. Comparison IV: Environment • Example: Eliminating humans from safety-critical loops is not good. examples? • Also need to pay attention to organizational and managerial aspects. • Examples: TMI, Challenger, Chernobyl…… CS 851 Forensic Software Engineering

  37. Comparison V: Human Error • “The operators of steam engines received most of the blame for the accidents, not the designers or the technology.” • Operator blame was very common a century ago. • The situation has not changed. • Software engineers designing interfaces without adequate knowledge of human factors. CS 851 Forensic Software Engineering

  38. Comparison V: Human Error • An Air Force study showed that out of 681 in-flight emergencies, only 10 were due to pilot errors. • TMI • Software examples: Therac-25 Chemical plant in Britain CS 851 Forensic Software Engineering

  39. USS Vincennes incident CS 851 Forensic Software Engineering

  40. Comparison V: Human Error • So, a basic understanding of human psychology is a prerequisite for user interface design – this is missing from current software engineering education. • Software Engineers must take human factors into account. CS 851 Forensic Software Engineering

  41. Comparison VI: Safety Standards • “The early steam engines had low standards of workmanship, and engineers lacked proper training and skills.” • Building safety – critical software requires special skills and knowledge on the part of both developers and management. • Education in software engineering is behind the state-of-art (1992). CS 851 Forensic Software Engineering

  42. Standards • Government standards in the U.S. require at least licensed Professional Engineer (P.E.) on their staff. • System safety projects have additional licensing requirements. • However, no such requirements for software projects. CS 851 Forensic Software Engineering

  43. Standards • Edison warned against the problems of poor workmanship and ignorance on part of the electrical contractors. • Similarly, Watt wanted the engineers to be responsible in case of accidents. • In software engineering, the public expects that the dangerous systems are built using the safest technology available. CS 851 Forensic Software Engineering

  44. Standards • How strict should standards be? • Example: The extensive regulation of high-voltage electricity in Great Britain. • Poorly written standards can inhibit the development of computer technology (comparison). • Standards can shift the blame from manufacturers and developers to government agencies. CS 851 Forensic Software Engineering

  45. Comparison VI: Safety Standards • Examples of accidents in software due to standards? Buffer overflow? • Current approaches to formulate safety standards for critical systems equate safety and reliability. • Or, define reliability as the reliability of safety protection devices. (Software = Hardware) CS 851 Forensic Software Engineering

  46. Comparison VI: Safety Standards • Two approaches in safety engineering: • Upstream approach • Downstream approach • Example: fire • There is a trade-off in both cases. CS 851 Forensic Software Engineering

  47. Conclusions • Does the paper have any ? • I think this is true for any new technology. • Mostly the paper covers points that have been made in the presentations so far. Anything new ? • What about the internet as a new technology ? Standards: http://www.ietf.org Internet Engineering Task Force CS 851 Forensic Software Engineering

More Related