Perspectives on Sensemaking

1. Perspectives on Sensemaking Simon Attfield Visual Analytics Summer School, 2010

2. Picture by Ed Sanders http://news.bbc.co.uk/1/hi/england/london/3732169.stm http://drlillianglassbodylanguageblog.wordpress.com

3. Sensemaking is... • The integration of information and knowledge to form an interpretation (Attfield, last weekend)

5. Cognitive dissonance 'Oh, you aren't even ripe yet! I don't need any sour grapes.' The fox and the grapes ‘adaptive preference formation’ Picture credit - Milo Winter

6. A search for coherence knowledge expectation experience values aspirations information data

7. Key questions • Cognition- How do people reason with information? How do they find meaning, reconstruct a domain, reach conclusions, and derive new puzzles and questions? • Social interaction– How do people interact around information? How do they share, convince, question, agree, cajole and debate? • Behaviour - How do people manage information? How do they adapt and shape their environments in order to achieve cognitive and social processes? • Tools- What can all of this say about the resources and functionality that can provide the leverage to do these things more naturally, invisibly and effectively?

8. Expectation and language • Battered Child Syndrome • Up to 1940’s no concept of battered child syndrome. • Late 1940’s - a surprise - a paediatric radiologist noticed a few cases of child injuries not adequately accounted for by parents medical ‘histories’. • Discrepancy explained away. • More cases noticed. • In 1953 a comprehensive report identified 749 cases. • Within a few years laws introduced mandating reporting. Weick, 1995

9. How do we makes sense of the world with very little information? – schemata • Data structures for representing generic concepts stored in memory • Introduced into psychology by Bartlett • ‘The building blocks of cognition’ (Rummelhart, 1980) • Variables that can be bound to an instantiation – e.g. Buying • A schema is a ‘theory’ – goes beyond our perception • Comprehension is a process of hypothesis testing http://venturebeat.com/2010/04/21/zipcar-buys-into-second-european-car-sharing-company/

10. Eh? The procedure is actually quite simple.  First you arrange things into different groups depending on their makeup.  Of course, one pile may be sufficient depending on how much there is to do.  If you have to go somewhere else due to lack of facilities that is the next step, otherwise you are pretty well set.  It is important not to overdo any particular endeavor.  That is, it is better to do too few things at once than too many.  In the short run this may not seem important, but complications from doing too many can easily arise.  A mistake can be expensive as well.  The manipulation of the appropriate mechanisms should be self-explanatory, and we need not dwell on it here.  At first the whole procedure will seem complicated.  Soon, however, it will become just another facet of life.  It is difficult to foresee any end to the necessity for this task in the immediate future, but then one never can tell. From Bransford and Johnson, 1973

11. The procedure is actually quite simple.  First you arrange things into different groups depending on their makeup.  Of course, one pile may be sufficient depending on how much there is to do.  If you have to go somewhere else due to lack of facilities that is the next step, otherwise you are pretty well set.  It is important not to overdo any particular endeavor.  That is, it is better to do too few things at once than too many.  In the short run this may not seem important, but complications from doing too many can easily arise.  A mistake can be expensive as well.  The manipulation of the appropriate mechanisms should be self-explanatory, and we need not dwell on it here.  At first the whole procedure will seem complicated.  Soon, however, it will become just another facet of life.  It is difficult to foresee any end to the necessity for this task in the immediate future, but then one never can tell. From Bransford and Johnson, 1973

12. The data-frame theory of sensemaking Klein, Phillips, Rall, Peluso (2007) • Frame • An explanatory structure that links elements e.g. a story, a map, a script, a plan. • Accounts for data and guides the search for more data. • Inferred from a few key anchors (3 or 4 at most) • Functions as a hypothesis • Data • Information from the world. • The interpreted signals of events. • Process proceeds through fitting data to the frame while testing and improving the frame.

13. Klein, Phillips, Rall, Peluso (2007)

14. Depends on data, goals, repertoire and stance. Pattern matching. Klein, Phillips, Rall, Peluso (2007)

15. Depends on data, goals, repertoire and stance. Pattern matching. Add details. Klein, Phillips, Rall, Peluso (2007)

16. Depends on data, goals, repertoire and stance. Pattern matching. Add details. Fundamental surprise (Lanir, 1991). Navigator’s frame- breaker. Emotional response Experts have more differentiated frames (Feltovich, 84) Klein, Phillips, Rall, Peluso (2007)

17. Depends on data, goals, repertoire and stance. Pattern matching. Explain away data. Knowledge shields. Fixation error produced by inaccurate anchor early on. Add details. Fundamental surprise (Lanir, 1991). Navigator’s frame- breaker. Emotional response Experts have more differentiated frames (Feltovich, 84) Klein, Phillips, Rall, Peluso (2007)

18. Depends on data, goals, repertoire and stance. Pattern matching. Explain away data. Knowledge shields. Fixation error produced by inaccurate anchor early on. Add details. Fundamental surprise (Lanir, 1991). Navigator’s frame- breaker. Emotional response Experts have more differentiated frames (Feltovich, 84) Nurses in ICU maintaining multiple frames. Tracking up to three frames. Logical competitor set. Klein, Phillips, Rall, Peluso (2007)

19. Depends on data, goals, repertoire and stance. Pattern matching. Explain away data. Knowledge shields. Fixation error produced by inaccurate anchor early on. Add details. Fundamental surprise (Lanir, 1991). Navigator’s frame- breaker. Emotional response Experts have more differentiated frames (Feltovich, 84) Nurses in ICU maintaining multiple frames. Tracking up to three frames. Logical competitor set. Duncker’s radiation problem. Klein, Phillips, Rall, Peluso (2007)

20. Depends on data, goals, repertoire and stance. Pattern matching. Explain away data. Knowledge shields. Fixation error produced by inaccurate anchor early on. Add details. Fundamental surprise (Lanir, 1991). Navigator’s frame- breaker. Emotional response Experts have more differentiated frames (Feltovich, 84) Explicit search. Nurses in ICU maintaining multiple frames. Tracking up to three frames. Logical competitor set. Duncker’s radiation problem. Klein, Phillips, Rall, Peluso (2007)

21. Reasoning in sensemaking Klein, Phillips, Rall, Peluso (2007) • Experts and novices reason in the same way, except novices find it difficult to distinguish signal from noise and are more reluctant to speculate. • Experts have a richer repertoire of frames – greater variety, more coverage, finer differentiation. More precise expectations. • Abductive reasoning predominates in sensemaking – ‘reasoning to the best explanation’ – based on plausibility.

22. A symmetry of inference • If it rains in the night the lawn will be wet in the morning. • It rained last night. • The lawn is wet (said in the morning). 1 1 2 1 2 2 Deduction Induction Abduction

23. A symmetry of inference • When someone with drugs in their pocket sees a policeman they put their hand in their pocket. • A man with drugs in his pocket saw a policeman • The man put his hand in his pocket 1 1 2 1 2 2 Deduction Induction Abduction

24. http://www.guardian.co.uk/politics/2008/sep/03/london.justicehttp://www.guardian.co.uk/politics/2008/sep/03/london.justice

27. Will you ever look at things the same again? http://www.guardian.co.uk/politics/2008/sep/03/london.justice

28. The rain example If it rains in the night, the lawn will be wet in the morning. If the children get up in the night and play with the hose, the lawn will be wet in the morning. If a bush-fire fighting helicopter on a training mission in the middle of the night accidently drops its load of water over our garden, the lawn will be wet in the morning.

29. What rule am I thinking of? • I am thinking of a rule the describes the relationship between three numbers. • The sequence 2-4-6 satisfies the rule. • Your job is to discover the rule by suggesting other number sets - I will tell you whether each one satisfies the rule. • When you think you know the rule, tell me.

30. Confirmation bias Peter Wason • Challenged subjects to identify a rule applying to triplets of numbers • e.g. 2-4-6 • Subjects seemed to test only positive examples, e.g. each number is two greater than its predecessor (11, 13, 15)

31. Confirmation bias Karl Popper • Marxism and psychoanalysis explain everything? • Worked with Alfred Adler, co-founder of psychoanalysis • Alder analysed child in terms of theory of inferiority without seeing the child. Picture credit: http://www.nndb.com/people/164/000087900/ Popper, 1963

32. Hang on... how does this relate to visual analytics?Pirolli and Card’s model of intelligence analysis Pirolli and Card (2005)

33. Attfield and Blandford’s model of legal investigations • Externalises evidence and reasoning • Structures thinking • Reveals gaps • Can enforce rigor Attfield and Blandford (in press)

34. Representing the story with chronologies P1 I think it’s a very natural way for us to think here, we always use chronologies, our great organising basis. […] I had a team of five or six people and I allocated responsibilities to each of these people saying “Right you’re going to become the master of [issue a], I’m going to do [issue b], [issue c], [issue d]. Someone else is going to do [issue e]”

35. Representing the story with chronologies Evidence link ? A met B at <location > on <date> … B flew to <location > on <date> … B met C at <location > on <date> …

36. Support recursive decomposition Theories Questions Info. seeking strategies Evidence Line of enquiry Knowledge reps Investigators Lines of enquiry

37. Support recursive decomposition Theories Theories Questions Questions Info. seeking strategies Info. seeking strategies Evidence Line of enquiry Evidence Knowledge reps Knowledge reps Investigators Investigators Lines of enquiry Lines of enquiry

38. Support recursive decomposition Theories Theories Theories Questions Questions Questions Info. seeking strategies Info. seeking strategies Info. seeking strategies Evidence Line of enquiry Evidence Evidence Knowledge reps Knowledge reps Knowledge reps Investigators Investigators Investigators Lines of enquiry Lines of enquiry Lines of enquiry

39. Support recursive decomposition Theories Theories Theories Theories Questions Questions Questions Questions Info. seeking strategies Info. seeking strategies Info. seeking strategies Info. seeking strategies Evidence Line of enquiry Evidence Evidence Evidence Knowledge reps Knowledge reps Knowledge reps Knowledge reps Investigators Investigators Investigators Investigators Lines of enquiry Lines of enquiry Lines of enquiry Lines of enquiry

40. Analysis of Competing Hypotheses

41. Analysis of Competing Hypotheses

42. Analysis of Competing Hypotheses • Starts with a set of possibilities rather than a most likely alternative for which the analyst seeks confirmation (avoids satisficing) • Identifies the few items of evidence or assumptions that have greatest diagnostic value. • Seeks evidence to refute hypotheses. Most probable hypothesis is one with least evidence against it (generally).

43. Analysis of Competing Hypotheses (ACH) 1. Identify hypotheses that merit consideration Heuer’s example... Q: Will Iraq retaliate for US bombing of its intelligence headquarters? H1 - Iraq will not retaliate. H2 - It will sponsor some minor terrorist actions. H3 - Iraq is planning a major terrorist attack. • If you don’t generate the right hypothesis you wont get the right answer • Use a group of analysts

44. Analysis of Competing Hypotheses (ACH) 2. List significant evidence for and against Heuer’s example... E1. Saddam public statement not to retaliate. E2. Absence of terrorist offensive during 1991 gulf war E3. Assumption that Iraq would not want to provoke another US attack E4. Increase in frequency/length of monitored Iraqi agent radio broadcasts E5. Iraqi embassies instructed to take increased precautions. E6. Assumption that failure to retaliate would be unacceptable loss of face for Saddam • Include assumptions • Ask what should I expect to see or not see? • Note the absence of evidence as well as its presence.

45. Analysis of Competing Hypotheses (ACH) 3. Prepare a matrix with hypotheses across the top and evidence down the side. Analyse how they relate. Consider diagnosticity

46. Analysis of Competing Hypotheses (ACH) 4. Refine the matrix. Reconsider hypotheses and delete evidence and arguments with no diagnostic value. More hypotheses needed? Is your thinking influenced by evidence or assumptions not included?

47. Analysis of Competing Hypotheses (ACH) 5. Draw tentative conclusions about likelihood of hypotheses. Work down the matrix. Let them compete for favour. Hypothesis with fewest minuses is probably best, BUT You can never prove a hypothesis true.

48. Analysis of Competing Hypotheses (ACH) 6. Consider how sensitive your analysis is to a few items of evidence. What if they were mistaken? Question the linchpin assumptions – a common source of error. What about deception?

49. Analysis of Competing Hypotheses (ACH) 7. Report conclusions – discuss relative likelihood of all hypotheses. There is always the possibility of being wrong. Decision makers need to evaluate.

50. Analysis of Competing Hypotheses (ACH) 8. Identify future observations that may indicate events are taking a different course than expected. Analytic conclusions should always be considered as tentative.