Risk Aversion and Other Obstacles to Mission Success A presentation to SPIN March 2, 2007

1. Risk Aversion and Other Obstacles to Mission SuccessA presentation to SPINMarch 2, 2007 Scott Jackson jackessone@cox.net – (949) 854-0519 • Systems Architecture and Engineering Program: • A Node of the Resilience Engineering Network

2. This presentation is based in part on a paper titled “The Science of Organizational Psychology Applied to Mission Assurance” presented to the Conference on Systems Engineering Research, Los Angeles, 7-8, April 2006. Co-authors were Katherine Erlick, PhD, and Joann Gutierrez both of whom have degrees in organizational psychology. Acknowledgments

3. System Resilience – the ability of organizational, hardware and software systems to mitigate the severity and likelihood of failures or losses, to adapt to changing conditions, and to respond appropriately after the fact. Culture is a key element in System Resilience Thesis: The traditional methods of executive mandate and extensive training are not sufficient to achieve a System Resilience culture. The science of organizational psychology promises to show us a better way. Common paradigms, especially with respect to risk, can be obstacles to a System Resilience System Resilience, Culture and Paradigms

4. The System Resilience Architecture Taking risk seriously is here Tools and processes, e.g., risk, are here

5. Root Causes Lack of Rigorous System Safety Lack of Information Management Culture Lack of Risk Management Regulatory Faults Lack of Review and Oversight Incomplete Verification Conflicting Priorities Poor Schedule Management Lack of Expertise Organizational Barriers Maintenance Cost Management Incomplete Requirements Faulty Decision Making Emergence Common Root Causes Suggest Key Capabilities Capabilities Cultural Initiatives System Resilience Oversight System Resilience Infrastructure Risk Management Schedule Management Cost Management Requirements Management Technology Management Verification System Safety Configuration Management Expertise Software Manufacturing Operations Work Environment Information Management Regulatory Environment Maintenance Reliability Supplier Management Adaptability Capabilities require more than system safety and reliability

6. American Flight 191 – Reason Apollo 13 – Leveson, Reason, Chiles Bhopal – Leveson, Reason, Chiles Challenger – Vaughn, Leveson, Reason, Chiles, Hollnagel Chernobyl – Leveson, Reason, Chiles Clapham Junction – Reason Columbia – Columbia Investigatory Committee, Chiles, Hollnagel The Fishing Industry - Gaël Flixborough – Leveson, Reason Hospital Emergency Wards – Woods and Mears Japan Airlines 123 – Reason Katrina - Westrum King’s Cross Underground – Leveson, Reason Mars Lander - Leveson Nagoya Airbus 300 – Dijkstra New York Electric Power Recovery on 9/11 - Mendoça Philips 66 Company – Reason, Chiles Piper Alpha – Reason, Chiles, Paté-Cornell, Hollnagel Seveso – Leveson, Reason Texas City – Hughes, Chiles Three Mile Island – Leveson, Reason, Chiles TWA 800 – National Transportation Safety Board (NTSB) Windscale – Leveson Case Studies Covered Many Domains Risk Emphasis

7. “A safety culture is a learning culture.” James Reason, Managing the Risks of Organizational Accidents “The severity with which a system fails is directly proportional to the intensity of the designer's belief that it cannot.”(The Titanic Effect) Nancy Leveson, Safeware: System Safety and Computers “Focus on problems.” Weick and Sutcliffe, Managing Uncertainty “One of our largest problems was success.” Cor Horkströter, Royal Dutch/Shell Some Quotes

8. “[Feynman’s] failure estimate for the shuttle system was 1 in 25…” “[NASA’s] estimate [of failure] range from 1 in 100 [by working engineers] to 1 in 100,000 [by management]” Diane Vaughn, The Challenger Launch Decision The Feynman Observation

9. Priority Number 3 – Do you have a good risk tool? Priorities: A personal view(page 1)

10. Priority Number 2 – Do you have a good risk process? Priority Number 3 – Do you have a good risk tool? Priorities: A personal view(page 2)

11. Priority Number 1 – Do you take risk seriously? Priority Number 2 – Do you have a good risk process? Priority Number 3 – Do you have a good risk tool? Priorities: A personal view(page 3)

12. - Paradigm No. 1 – The belief that even having risks is a sign of bad management - Paradigm No. 2 – Risk as a “normative” condition Diane Vaughn: “NASA’s ‘can do’ attitude created a … risk taking culture that forced them to push ahead no matter what..” “…flying with acceptable risks was normative in NASA culture.” Two important risk paradigms Definition: Mind-set, perception, way of thinking, cultural belief

13. Definition: Mind-set, perception, way of thinking, cultural belief Some Paradigms • Don’t bother me with small problems. • Our system (airplane, etc) is safe. It has never had a major accident. • We can’t afford to verify everything. • My job is to assure a safe design. • If I am ethical, I have nothing else to worry about.

14. More Paradigms (p. 2) • If I get too close to safety issues, then I may be liable. • Safety is the responsibility of individuals, not organizations • Our customer pays us to design systems, not organizations • Human error has already been taken into account in safety analyses • Accidents are inevitable; there is nothing you can do to prevent them • Organizational issues are the purview of program management

15. Still More Paradigms (p. 3) • I am hampered by scope, schedule and cost constraints • Our contracts (with the customer and suppliers) do not allow us to consider aspects outside of design • Human errors are random and uncontrollable • You can’t predict serious accidents • To change paradigms all we need is a good executive and lots of training

16. Wreathall says we must consider “meta-risks,” that is, risks that we all know are there and do not consider Epstein says that the important risks are in the lower right hand corner of the risk matrix: Some Thoughts on Risk(from the Second Symposium on Resilience Engineering, Juan-les-Pins, France, November 2006) • Low probability • High consequence • Lots of them • Examine by simulation

17. The Genesis of Paradigms Our paradigms Cultural Beliefs Pressures (cost, schedule, etc.)

18. The Old Model Start Here Executive Management The Vision Employees Attend Training

19. The New Model (Simplified) Start Here Endorse Self-Discovery Executive Management New Paradigms Employees Establish Communities of Practice Learn

20. Training The Hero-Executive Socratic teaching Coaching Self-discovery through communities of practice Independent reviews Cost and schedule margins Standard processes Teams Rewards and incentives Management selection Some Approaches Community of Practice Core Group

21. Bottom-up Informal Core Group Dialogue Respect Inclusive Self-Discovery Through Communities of Practice

22. Progress is both top-down and bottom-up Organizational psychology is a necessary discipline for mission assurance and, hence, also for systems engineering Training and top-down mandates have limited effectiveness Self-discovery is the preferred path. No one can teach you the right paradigm; you have to learn it yourself. Conclusions

23. Vaughn, Diane, The Challenger Launch Decision: Risky Technology, Culture and Deviance at NASA, University of Chicago Press, 1996 Reason, James, Managing the Risks of Organizational Accidents, Ashgate, 1997 Leveson, Nancy, Safeware: System Safety and Computers, Addison Wesley, 1995 Weick, Karl E. and Sutcliffe, Kathleen M., Managing the Unexpected, Jossey-Bass, 2001 Senge, Peter, et al, The Dance of Change, Doubleday, 1999 Wegner, Etienne, et al, Communities of Practice: Learning, Meaning and Identity, University of Cambridge, 1998 Some Recommended Reading