Logic vs. Magic in Critical Systems

Logic vs. Magic in Critical Systems or: why wizards are not enough Ian O’Neill

Agenda • Introduction: a historical magic • The software process • What are “Critical Systems”? • The need to “reason” • Some obstacles • Advantages of logic — project experience • Summary

A Historical Magic • Go to statement considered harmful • Structured programming • “De-spag” tools

A Specification Add up the first N (positive) values of the array traverse the array until N positive values have been found and sum them or traverse N elements of the array and sum any positive ones (ignoring any negative ones) ?

function Sum(A : Atype; Count : Aindex) return Integer is Accumulator, Value : Integer; Index : Aindex; begin Accumulator := 0; Index := 1; <<Label1>> Value := A(Index); if Value <= 0 then goto Label2; endif; Accumulator := Accumulator + Value; <<Label2>> if Index = Count then goto Label3; endif; Index := Index + 1; goto Label1; <<Label3>> return Accumulator; end Sum;

function Sum(A : Atype; Count : Aindex) return Integer is IC : Counter; Accumulator, Value : Integer; Index : Aindex; begin IC := 1; Accumulator := 0; Index := 1; while IC < 6 loop case IC is when 1 => Value := A(Index); IC := 2; when 2 => if Value <= 0 then IC := 4; else IC := 3; endif; when 3 => Accumulator := Accumulator + Value; IC := 4; when 4 => if Index = Count then IC := 6; else IC := 5; endif; when 5 => Index := Index + 1; IC := 1; when 6 => Null; endcase; endloop; return Accumulator; end Sum;

Engineering: The use of science and mathematics to solve practical problems

The software process • Progressive refinement: • a need • requirements • specification • design • source code • object code

A Specification Add up the first N (positive) values of the array

Language Insecuritya tiny example procedure Init2(X, Y : out Integer) is begin X := 1; Y := 2; end Init2; What is the meaning of: Init2(A, A);

Formality at last! 10100110100100001000100010010001 10101000111101010100001010101010 10101010100101001010101010000101 01010010010101001101010101010101 00000101000101010010101001010010 01010101010101010101011110100101 0010101010101010010.......

A typical development process Informal, opaque black box Precise object code

What are critical systems? • Some important objectives: • efficiency • cost • time to market • functionality

What are critical systems? • Systems where reliability is more important than: • efficiency • cost • time to market • functionality Robert Dewar - Ada Core Technologies

Contributors to Lifecycle Costs Cost of error correction Time spent: <50% <20% >50% >80%

Producing Safety-Critical Software • Not just a question of “being more careful” • The need to be able to show, before there is any service experience, that a system will be safe enough requires a qualitatively different approach • We can only achieve this by logical reasoning.

Foundations of Reasoning • We reason about information • But what about “information hiding”? • We need to hide detail not information • We do this through abstraction • We can’t just magic complexity away Example: Program “State”

Swap Algorithm Temp : T; procedure Swap(X, Y : inout T) is begin Temp := X; X := Y; Y := Temp; end Swap;

Swap Algorithm procedure Swap(X, Y : inout T) is begin Temp := X; X := Y; Y := Temp; end Swap; Temp : T;

A Store Object package Store is procedure Put(X : in T); function Get return T; end Store;

Swap Algorithm procedure Swap(X, Y : inout T) is begin Store.Put(X); X := Y; Y := Store.Get; end Swap;

Heap

The location of state is the single biggest influence on coupling and cohesion. • It is probably the most important design decision we must make yet • OOP regards it as an “implementation detail”; and • UML does not even have a notation to express it!

OOP - The Biggest Magic of them All • We hide all state • We disguise the hierarchical relationships between objects • We hide the control flow by use of messages and dynamic dispatch • In extreme cases, we even deny there is a software design process at all (Shlaer Mellor) Of course all this hiding makes things easier - at least until all the hidden complexity emerges during integration testing!

An OOP Array Sum

“There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.” Professor C.A.R. Hoare The 1980 Turing award lecture

Some obstacles • Fashion • Low expectations • Poor contract writing • The difficulty of saying “no” • Gurus • Belief that: • Software engineering is the same as knowing tool or product X • Knowing tool or product X is more important than domain or engineering knowledge

Fashion • We must use X, everyone else is • The staff want to use X, we won’t be able to recruit or keep them if we don’t • All the tool vendors are supporting X • Y, which we know is better, “is dead” • Progress is the same as the number of lines of code written • The most important thing is to get to test quickly

Low Expectations • Software is always buggy • Even Microsoft “the best software company in the world” produces buggy software • Vendors never offer warranties • Since software is always a load of rubbish we might as well buy the cheapest load of rubbish

Poor Contract Writing • Fixed-price contracting • sets producer and customer in opposition • even mutually beneficial changes are difficult • Time and material contracting • removes any incentive to do a more efficient job

The Difficulty of Saying No • Engineers have a duty to say “No” where safety is being compromised; however, • There is often someone else who can be persuaded to say “Yes” (and charge you less!) • The Chinook FADEC saga: • the safety assessors have been cast as the villains unreasonably holding up entry to service • the software is being treated as safe because it is so badly written it is not possible to prove that it’s dangerous

Gurus • Close associates of wizards and magicians • Know all 123 operator precedence levels of C • Don’t make the kinds of mistakes that mere mortals make so don’t see the benefit of techniques that help avoid them • Have a strong incentive to maintain the status quo because they fear losing their guru status • Don’t often make good engineers

Misunderstanding the Nature of Knowledge • C programmers are easy to hire so we should use C • riveters are easy to hire but it hasn’t stopped aircraft companies adopting composites • Learning a new tool or programming language is hard but learning a new problem domain is easy • so people experienced in banking systems should write flight control systems • We should use techniques that are widely known rather than ones that are appropriate • like driving Ford Mondeos to the moon!

Contributors to Lifecycle Costs Cost of error correction Time spent: <50% >50%

The Need to Reason • To bring error detection forward we must be able to reason about source code • We can’t reason about source code unless it has a precise meaning • Too often the meaning of code is defined by its test results!

Logic vs. Magic in Critical Systems