Information value: the value of evidence

2. Information value:the value of evidence Dr David J Marsay, C.Math FIMA A presentation to 19 ISMOR 29.08.2002

3. Contents 1 Introduction 2 Examples 3 Theory 4 World-view 5 Implications 6 Conclusions

4. Introduction Section 1

5. 1 Introduction • Information is the life-blood of military C4ISR. • Any time we prefer one set of information to another we implicitly ‘value’ it. • We think we could do better: • lessons identified. • studies. • Specifically needed to support UK MOD’s ARP 14 ‘Battlefield picture compilation’.

6. 1 Introduction • We use P(A|B) to denote ‘the probability of A given B’ • P(A) is used to denote the [prior] probability. • For hypotheses {H} and evidence E: • Shannon’s ‘entropy’ is calculated from the ‘final’ probabilities, {P(H|E)}. • Jack Good’s ‘weight of evidence’ is calculated from the likelihoods, {P(E|H)}. • According to Bayes’ rule, the probabilities can be calculated from the likelihoods and ‘the priors’.

7. Examples Section 2

8. 2 Examples Control: Suppose that a source sends accurate data to a deterministic machine. • Shannon’s concept does not apply. Nor does the notion of ‘priors’. • The value of the data can be determined by valuing the function of the machine - no fancy method needed. • The likelihoods make sense. They are 0 or 1.

9. 2 Examples Command - ‘soft’ aspects: • For an information artefact (e.g., an INTSUM) to represent the same information implies that all recipients had the same priors. Thus everyone receives everything in the same order. • Is this realistic? • Alternatively, one could define some privileged ‘central’ viewpoint for which the information is defined. • Does this fit doctrine? • Is it helpful?

10. 2 Examples Command - ‘soft’ aspects: • The likelihoods {P(E|H)} are a rating of the source of E. They are thus relatively ‘objective’, ‘knowable’ and ‘shareable’. • Likelihoods relate to current practice (reliability, accuracy).

11. 2 Examples Compilation: • The work being reported on has looked at the relatively ‘hard’ problem of compilation, particularly ‘Battlefield picture compilation’ under ARP 14. • Weights of evidence can be used. (See accompanying paper.) • When is this reliable?

12. Theory Section 3

13. 3 Theory Jack Good’s evidence: • Likelihoods are often straightforward.E.g., P(‘Heads’|‘Fair Coin’) = 0.5 by definition. • Lab and field testing traditionally establish, in effect, likelihoods. • Surprise = -log(likelihood). • Weight of evidence (woe) is surprise, normalised by the prior expected surprise for the same evidence. (So that only ‘relevant detail’ counts.)

14. 3 Theory Evidence is more fundamental than Shannon’s information • Shannon’s entropy is expected surprise. • The more useful cross-entropy is likely surprise. • Woe supports alternative decision methods, such as sequential testing, hypothesis testing.

15. 3 Some questionable assumptions • Shannon assumes that systems of interest are ‘Markov’. • Shannon noted that ‘state-determined systems’ are ‘Markov’ with probability 1. • But Smuts (e.g.) noted that evolution drives dynamical systems to adopt synergistic ‘emergent’ structures. • These had a priori probability 0. • So for social systems, international relations, military conflict ... we cannot rely on Shannon’s ‘information’.

16. 3 Some questionable assumptions • But can likelihoods be used? • If we abandon Markov models, how are we to judge if a given algebra of likelihoods is valid? • We need a ‘world meta-model’ to replace Markov.

17. World-view Section 4

18. 4 SMUTS(synthetic modelling of uncertain temporal systems) Delayed Double Viper BUBs2D5.5 • Allows one to investigate emergence within complex systems. • Evidence of piece-wise Markov behaviour. • Developed under the MOD CRP TGs 0,5,10. t = 0.0502 t = 0.2005

19. 4 Alternative ideas • I postulate a model in which systems of interest to the military are Markov in space-time ‘zones’, with more interesting transitions at their boundaries. • Thus Markov locally, but not globally. • In essence emergence only happens when an over-adaptation is exploited. (E.g. Ashby, Piaget.) • Thus, as long as we can learn at least as quickly, we should be able to recognise these situations too.

20. 4 Supporting evidence Applications to, and experiences of: • warfare • economics • international relations. (My subjective view)

21. 4 Reuters data for the Balkans, the 90s Evidence of locally Markov behaviour

22. Implications Section 5

23. 5 Implications for ‘information’ Technical differences: • The difference between the expected weight of evidence (woe) and Shannon’s entropy is not a constant. • Systems of interest tend to have ‘long range’ sources of uncertainty, in addition to the ‘local’ entropy. • We need to allow for this and ‘expect the unexpected’ to achieve robustness.

24. 5 Implications for ‘information’ Some cases where Shannon might not be appropriate • Poor ‘local’ information. • The ‘situation’ cannot necessarily be recognised. • The ‘target’ is adaptable (particularly if adapting against us).

25. 5 Implications for ‘information’ Typical symptoms that Shannon is inadequate: • Mistakes often reflect a need to validate assumptions. • Ossification, atrophy and vulnerability (Ashby / Piaget)

26. 5 Implications for ‘information’ Notes: • We can’t expect to have considered all possible hypotheses in advance. • However, we do know when the truth is ‘something else’ because the weights of evidence are poor for the assumed hypotheses. • Thus we can detect deception and ‘fixation’ (a form of self-deception).

27. Conclusions Section 6

28. 6 Conclusions • The common concepts of information assume that systems are globally ‘simple’. • Our systems of interest are not simple, but may be piece-wise ‘simple’. • Jack Good’s ‘weight of evidence’ can be used to ‘bridge’ ‘islands of simplicity’. • Using ‘weight of evidence’ gives significant ‘added value’ to using just Shannon information.