1 / 52

Methods and Metrics for Analysis of Sensemaking

Methods and Metrics for Analysis of Sensemaking. Dr Karen Carr & Mr Barry McGuinness BAE SYSTEMS Advanced Technology Centre. Our objectives. For this meeting: Contribute what makes sense to us, in our given context and with our goals In our work:

marius
Download Presentation

Methods and Metrics for Analysis of Sensemaking

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Methods and Metrics for Analysis of Sensemaking Dr Karen Carr & Mr Barry McGuinness BAE SYSTEMS Advanced Technology Centre

  2. Our objectives • For this meeting: • Contribute what makes sense to us, in our given context and with our goals • In our work: • Develop the ability to supply C3I ‘capabilities’ (in partnership) • Systems Engineering of socio-technical systems • Driven by need to deliver usable and demonstrable results • Science as well as engineering and domain expertise NB We want to ensure that human issues drive the developments - but we don’t want to forget that we have to inform technology (as well as organization, process)

  3. What we mean by sensemaking Why we want to use this concept to try and answer the questions we need to answer • Our question: “How can we develop technology, design and manage systems which support/ enhance the human roles in defence operations?” • Significant human role is ability to adapt, respond to unexpected, creativity, play mind games, etc. Need to preserve & enhance that - not interfere. • Support is needed for dealing with the unexpected, the unknown, as well as recognisable situations • Include broad System of Systems issues, developers, rapid change Sensemaking (& Situational Awareness) is a working concept to enable us to start manipulating, analysing, and measuring context, goals and human performance

  4. Why we want methods and metrics for studying sensemaking • Need to attribute effects - predict - in order to provide support. • Move from concepts to metrics to analysis to (testable) models. • Reduce subjective bias (influence of our own sensemaking, interpretation) • No existing clear metrics we can use - no absolutes 1. Understand how human performs, and what conditions facilitate ‘good’ performance (what hinders) 2. Identify the properties of organisation, process, technology, training, etc which are important for success 3. Develop design and management methods and tools to enable implementation • NB not necessarily numbers - could be properties

  5. Range of methods • observation (non intrusive) • subjective investigation (e.g. ethnographic, knowledge elicitation) • storytelling/anecdotes (knowledge building) • metaphor (pattern matching) • scientific method (empirical hypothesis testing) • mathematical analysis (baseline)

  6. Methods and Metrics • Concepts • Metrics • Some analyses • Implications for sensemaking

  7. Concepts • Orientation • complex, uncertain situations • SA determines capacity to decide and act • sensemaking determines SA • cognitive processes are intrinsically goal-directed • people form nested hierarchy of processes & outcomes • Objectives • Understand SA and sensemaking • Feed into design & development of information systems and human systems • Applied research -- theory into practice

  8. What is a Situation? • A situation is a pattern in state space, especially one which appears to deviate from a “normal” intended or expected pattern. • Example:- aircraft fuel x time into flight Normal takeoff Aircraft fuel level Unexpected rate-- we have a situation! Normal cruise Time into Flight

  9. Unrecognized Patterns • An unrecognized pattern demands attention. Attention! perception comprehension ??? Unknown pattern? Known patterns? Perceived pattern

  10. Definitions Knowledge: = capacity for “action” Situational Awareness: = dynamic “situated” knowledge, i.e. capacity to act effectively here & now in a given specific situation Sensemaking: = process of creating effective SA in situations of uncertainty • doing • saying • thinking “Knowing what’s going on so you can figure out what to do.”

  11. Situational awareness Dynamic mental representation of the current and future state of one’s domain of action • includes awareness of • environment • entities • events • processes • actions • others’ perceptions & intentions • insofar as these are relevant to • performing an action, or • choosing a course of action Through a continuous process of situation assessment

  12. Situational awareness SA is based on ... • prior KNOWLEDGE • SCHEMAS: generalized patterns representing typical situations • based on experience, training, culture • recent INFORMATION • direct perception of the environment • perception of instruments and displays • received communications KNOWLEDGE SA INFORMATION instruments communications PHYSICAL DOMAIN

  13. Central role of SA • SA both informs and is informed by • sense-making • decision-making Sense- making Decision- making SA COGNITIVE DOMAIN Information acquisition Action performance PHYSICAL DOMAIN

  14. Inside SA: Cutting up the cake Observed Implied Models situational schemas e.g., “Fuel leak?” “Faulty sensor?” Projections mental simulations e.g., “Risk of not reaching destination” Abstract (generalized patterns) Concrete (situation-specific) Information specific propositions e.g., “rate of fuel loss is high” Intentions selected actions afforded by situation e.g., “Contact ATC and inform”

  15. Processes involved in SA PERCEPTIONAcquisition of information about the given situation COMPREHENSIONDiagnosticinterpretation of the given situation PROJECTIONPrognostic simulation of future situations and their possible outcomes RESOLUTIONSelection of actions to direct the given situation towards the desired outcome … All serving to support dynamically effective action information models projections intentions

  16. Sensemaking and SA PROJECTION COMPREHENSION Models Projections Decision making Sense making Information Intentions PERCEPTION RESOLUTION Sensing Acting Sense-making: when comprehension is uncertain Decision-making: when resolution is uncertain

  17. Metacognition • Defined as: • “Thinking about thinking” or “knowledge about knowledge” • i.e. “Awareness of your own SA” • noticing uncertainties, gaps, conflicts in your mental reps • identifying information needs • employing strategies for sensemaking & decision-making ? SA “It’s like looking over your own shoulder.” Gives a subjective sense of SA

  18. SA and metacognition • Four possible states: Actual awareness: True SA False SA Inappropriate Confidence (danger state) Appropriate Confidence (ideal state) Confident in SA Subjective attitude Inappropriate Sensemaking Appropriate Sensemaking Not confident in SA Need for sensemaking

  19. Team SA and shared SA • Not the same thing • Team SA = sum of current knowledge held across a team, irrespective of who has it • Shared SA = those parts of the team SA that are common between team members Team SA Shared SA Personal SA

  20. What to share, with whom? • The nature of SA in groups is dictated by goals • Goals are hierarchic • Top-level goals are shared by all members • therefore need shared SA with respect to that objective • Lower-level goals are specific to individuals • therefore need personal SA with respect to own task • Sharing one’s SA is necessary only to the extent that the knowledge has bearing on the goals of others

  21. Team SA and shared SA • Shared SA elements can be differentially allocated resolution comprehension projection comprehension projection perception perception resolution perception perception perception resolution perception resolution

  22. Distributed SA in the C2 HQ Metacognition? u z Models (COMPREHENSION) Intel m Projections (PROJECTION) Ops m m M m m m Intentions (RESOLUTION) Information (PERCEPTION) Commander Signal

  23. So... • Explicit sensemaking processes are needed when comprehension cannot easily occur • Sensemaking requires metacognitive awareness of own knowledge -- uncertainties, gaps • Metacognitive assessments can be wrong and lead to inappropriate subjective attitude -- and inappropriate behaviour

  24. Measuring SA • COGNITIVE approach • queries about the situation • Reveals mental reps • Multiple choice (SAGAT) • True/False (QUASA) • Sit Reps • SUBJECTIVE approach • self-ratings of SA • Reveals metacognitive state • Unidimensional (SARS) • Multidimensional (SART) • Multidimensional and intelligible! (CARS) • OBJECTIVE approach • behavioural & physiological correlates • Reveals changes in metacognitive state • EEG, fMRI • Eye pointing As a rule, take both cognitive & subjective measures together.

  25. CARS • Crew Awareness Rating Scale • a subjective tool to elicit operator’s subjective sense of SA • multi-dimensional • generic, adaptable, easy to use

  26. Dimensions KnowledgeProcessing Perception Comprehension Projection Resolution

  27. Eight CARS questions knowledge 1. the most recent information 2. what is really going on here 3. what could happen 4. what actions should be taken Would you say you have a good sense of … processing Would you say it is easy for you to … 1. monitor the flow of information 2. understand the big picture 3. predict how it is likely to evolve 4. decide what actions to take

  28. Six possible responses For sure? CertainUncertain Do I ? YES NO Think so Definitely Definitely not Think not Don’t know Don’t need it

  29. CARS results Def Prob Prob not Def not DK NA • CONTENT • Perception ||| ||| | | • Comprehension | |||| || | • Projection || |||| | | • Resolution | ||| ||| | • PROCESSING • Perception |||||| || • Comprehension ||| ||| || • Projection || || ||| | • Resolution || |||| ||

  30. 1. Definitely 2. Probably 3. Probably NOT 4. Definitely NOT CARS results Comprehension knowledge over time 100 80 60 % of ratings 40 20 0 Run 1 Run 2 Run 3 Run 4

  31. QUASA • Quantitative Assessment of Situational Awareness • a probe tool to elicit operator’s actual SA • mathematical : based on SDT • still under development, but promising

  32. QUASA Square? • Signal Detection Theory YES! perception discrimination • Targets vs non-targets • Hits, False Alarms, Good Misses, False Rejections • Also applies to internal (mental) representations

  33. QUASA • “Is this item true?” • Confidence in perceived truth value of items varies Number of items Max SENSITIVITY = ideal SA FALSE items TRUE items Weak Strong Confidence in truth value of items

  34. Number of items Confidence in truth of items QUASA No sensitivity, poor SA Literally can’t tell the difference between true & false items… They have similar-strength levels of confidence Weak Strong

  35. QUASA Deception Number of items Max NEGATIVE sensitivity = worst case SA TRUE items FALSE items Weak Strong Confidence in truth value of items

  36. SA’ Number of items IB’’ Confidence in truth value of items QUASA Some positive sensitivity Low positive bias (acceptance threshold) Good rejections Good acceptances Bad rejections Bad acceptances Weak Strong

  37. QUASA Example probe: “ The tanks adjacent to bridge are enemy ” Response: YES (accept as true) or NO (reject as false) Evaluate: Sensitivity (discrimination of true/false situations) = SA’ Bias (probability of item acceptance/rejection) = IB’’

  38. QUASA +100 0 -100 Maximum negative sensitivity: the wrong situation! Maximum negative bias: too cautious Maximum positive sensitivity: ideal SA Maximum positive bias: too rash TYPICAL

  39. QUASA +100 0 -100 Comprehension: model of situation Projection: Future developments Resolution: CoA intention Perception: information

  40. QUASA • Mathematical assessment of SA • Needs the truth! • SA, bias, components, temporal • ? Team & shared SA

  41. Behavioural correlate of SA Tracking eye-point-of-gaze (EPOG) Do EPOG patterns correlate with SA?

  42. EPOG research ‘Entropy’ = known loss of SA

  43. Radio “party line” Collision avoidance system Heathrow control this is Speedbird five five, descending now to flight level one four. Speedbird five five, Heathrow control, roger, descending to flight level one four. Heathrown control this is Delta four zero four, flight level two zero, request descent clearance. Delta four zero four, Heathrow control, what is your present altitude? ... +01 SA and flightdeck automation

  44. With automation Conventional 50% Reported aircraft 25% 0% Detectable aircraft Non-Detectable aircraft Traffic Situation Reporting SA and flightdeck automation

  45. SA and C2 digitization - ISTAR Own force positions Enemy positions BGHQ crewstation Common Operational Picture

  46. SA and C2 digitization - ISTAR Battlespace digitization demonstrator Synthetic environment

  47. SA and C2 digitization - ISTAR • 2-hr ISTAR recce operation • Performed with voice AND digital C2 systems Measures taken of mental workload & situational awareness

  48. SA and C2 digitization - ISTAR voice digital DEF PROB rating PROB NOT DEF NOT PERCEPTION COMPREHENSION PROJECTION RESOLUTION aspects of SA (knowledge of enemy)

  49. Some implications Both actual SA and subjective sense of SA affect decision-making & performance Technology can affect SA for better or worse Analyses with metrics provide specific insights

More Related