Umbrella Presentation Theme C: Cognitive Science of Cyber SA

1. Umbrella PresentationTheme C: Cognitive Science of Cyber SA ASU (Cooke) Cyber Security as a Complex Cognitive System PSU (McNeese & Hall) Computer-aided Computer-Aided Human Centric Cyber Situation Awareness

2. Cognitive Models & Decision Aids • Instance Based Learning Models • Simulation • Measures of SA & Shared SA • Software • Sensors, probes • Hyper Sentry • Cruiser • Information Aggregation & Fusion • Transaction Graph methods • Damage assessment • Automated • Reasoning • Tools • R-CAST • Plan-based narratives • Graphical models • Uncertainty analysis Data Conditioning Association & Correlation Multi-Sensory Human Computer Interaction Computer network • Enterprise Model • Activity Logs • IDS reports • Vulnerabilities Real World Computer network System Analysts Test-bed

3. Situation Awareness Endsley’s Definition: the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future

4. Cyber Situation Awareness is Inherently Human SA is not in the technology (e.g., visualization); it is in the interface between humans and technology

5. Team Situation Awareness A team’s coordinated perception and action in response to a change in the environment Contrary to view that all team members need to “be on the same page”

6. Cyber SA is Distributed and Emergent • Detector • Responder • Threat Analyst • Perception • Comprehension • Projection

7. Cyber SA is Distributed and Emergent • Detector • Responder • Threat Analyst • Perception • Comprehension • Projection

8. Cyber Security as a Complex Cognitive System N.Cooke, P. Rajivan, M. Champion, G. Dube, V. Buchanan, S. Jariwala Top-down Cognitive Science Theoretical Foundations Cyber Defense Interactive Team Cognition/ Sociotechnical Systems Theory Distributed Research Simulations CyberCog & DEXTAR Observe Observation Fields of Practice Metrics & Measures Tools & Methods Communication & Coordination Team Situation Awareness Agent-Based & EAST Modeling Cognitive Systems Engineering Bottom-Up

9. Cyber Security as a Complex Cognitive System N.Cooke, P. Rajivan, M. Champion, G. Dube, V. Buchanan, S. Jariwala Theoretical Foundations Human-Centered Distributed Research Simulations CyberCog & DEXTAR Interactive Team Cognition/Sociotechnical Systems Theory Workload Specialization Actual Experimental Studies Conducted Teams vs Groups Team and Organization Models

10. Computer-Aided Human Centric Cyber Situation Awareness M. McNeese, D. Hall, N. Giacobe, V. Mancuso, D. Minotra, and E. McMillan Top-down Cognitive Science Theoretical Foundations Cyber Defense Situated Cognition Distributed Research Simulations teamNETS Observe Observation Fields of Practice Metrics & Measures Tools & Methods Visual Analytics TestbenchComplex Event Processing Cognitive Systems Engineering Bottom-Up

11. Computer-Aided Human Centric Cyber Situation Awareness M. McNeese, D. Hall, N. Giacobe, V. Mancuso, D. Minotra, and E. McMillan Theoretical Foundations Human-Centered Distributed Research Simulations teamNETS Situated Cognition Attention/Disruption Memory / Access Awareness Actual Experimental Studies Conducted Team Cognition Embedded Model of the Threat

12. ASU/PSU Objectives PSU Objectives ASU Objectives To develop theory of team-based SA to inform assessment metrics and improve interventions (training and decision aids) Iterative Refinement of Cyber Testbedsbased on cognitive analysis of the domain Cybercog DEXTAR Conduct experiments on Cyber TSA in the testbed to develop theory and metrics Extend empirical data through modeling • To understand Individual and Team cognition of Situation Awareness in Cyber-Security domains • Refine and implement evaluation environment to support evaluation of new analysis models, cognitive tools, and adversarial team cognition via hidden knowledge profiles • Develop new tools for practice based on field- and laboratory-based findings

13. Cyber Security as a Complex Cognitive System Nancy J. Cooke, PhD Prashanth Rajivan, MS Michael Champion, MS Shree Jariwala Geneviève Dubé, Université Laval, Québec Verica Buchanan Arizona State University October 29, 2013 This work has been supported by the Army Research Office under MURI Grant W911NF-09-1-0525.

14. Outline Overview of Project Definitions and Theoretical Drivers Empirical Study on Teams vs. Groups Agent-Based Modeling Two Case Studies and EAST Models Next Steps

15. Overview of Project

16. ASU Project Overview • Objectives: • Understand and Improve Team Cyber Situation Awareness via • Understanding cognitive /teamwork elements of situation awareness in cyber-security domains • Implementing a synthetic task environment to support team in the loop experiments for evaluation of new algorithms, tools and cognitive models • Developing new theories, metrics, and models to extend our understanding of cyber situation awareness • Department of Defense Benefit: • Metrics, models, & testbedsfor assessing human effectiveness and team situation awareness (TSA) in cyber domain • Testbed for training cyber analysts and testing (V&V) algorithms and tools for improving cyber TSA • Scientific/Technical Approach - Year 4 • Explore the role of teamwork in cyber defense through: • Empirical work in CyberCogtestbed • Agent-Based Modeling • Case Studies and EAST Modeling • Further refine team metrics and testbeds • Year 4 Accomplishments • Found an empirical benefit of cyber teaming • Replicated this benefit in an agent-based model • Compared two cyber defense organizations • Refined team metrics and cybercogtestbed • Challenge • Struggle to maintain realism in testbed scenarios while allowing for novice participation and team interaction – now addressing with CyberCog and Dextar

17. Summary of FY 13 ASU Accomplishments PUBLICATIONS Cooke, N. J., Champion, M., Rajivan, P., & Jariwala, S. (2013). Cyber Situation Awareness and Teamwork. EAI Endorsed Transactions on Security and Safety. Special Section on: The Cognitive Science of Cyber Defense, 13. Cooke, N. J. & McNeese, M. (2013). Preface to special issue on the cognitive science of cyber defence analysis. EAI Endorsed Transactions on Security and Safety. Special Section on: The Cognitive Science of Cyber Defense, 13 Rajivan, P., Champion, M., Cooke, N. J., Jariwala, S., Dube, G., & Buchanan, V. (2013). Effects of teamwork versus group work on signal detection in cyber defense teams. In D. D. Schmorrow and C.M. Fidopiastis (Eds.), AC/HCII, LNAI 8027, pp. 172-180., Berlin: Springer-Verlag. Rajivan, P., Janssen, M. A., & Cooke, N. J., (2013). Agent-based model of a cyber security defense analyst team. Proceedings of the 57th Annual Conference of the Human Factors and Ergonomics Society, Santa Monica, CA: Human Factors and Ergonomics Society. Champion, M., Rajivan, R., Jariwala, S., Cooke, N. J., & Buchanan, V. Understanding the cyber security task. Poster presented at ASU's Sixth Annual Workshop on Information Assurance, May 1, 2013, Tempe, AZ. • STUDENTS SUPPORTED • PrashanthRajivan (PhD) • Verica Buchanan (UG) • PROJECTS SUPPORTED FY 13 • CyberCog and metrics development • CyberCog study • Agent-based models of cyber teaming • Agent-based models of cyber warfare • Case Studies and EAST models • COLLABORATION • Coty Gonzalez – IBLT and Agent-Based Modeling • SushilJajodia– DEXTAR • Several MURI partners on an ARL proposal • TECH TRANSFER • Working with Charles River Analytics and AFRL on team measures of cyber defense • Working with SA Technologies on cyber visualization • Presentation to ASU Information Assurance • Presentation to General Dynamics – The Edge AWARD PrashanthRajivanwins HFES 2013 Alphonse Chapanis Award for best student paper!!!

18. Definitions and Theoretical Drivers

19. Theoretical Drivers • Interactive Team Cognition • Sociotechnical Systems Theory/ Human Systems Integration

20. Interactive Team Cognition Team is unit of analysis = Heterogeneous and interdependent group of individuals (human or synthetic) who plan, decide, perceive, design, solve problems, and act as an integrated system. Cognitive activity at the team level= Team Cognition Improved team cognition  Improved team/system effectiveness Heterogeneous = differing backgrounds, differing perspectives on situation (surgery, basketball)

21. Interactive Team Cognition Team interactions often in the form of explicit communications are the foundation of team cognition • ASSUMPTIONS • Team cognition is an activity; not a property or product • Team cognition is inextricably tied to context • Team cognition is best measured and studied when the team is the unit of analysis

22. Implications of Interactive Team Cognition • Focus cognitive task analysis on team interactions • Focus metrics on team interactions (team SA) • Intervene to affect team interactions

23. Cyber Defense as a Sociotechnical System • Cyber defense functions involve cognitive processes allocated to • Human Operators • Tools/Algorithms • Human Operators • Different roles and levels in hierarchy • Heterogeneity (Information, skills and knowledge) • Tools • For different kinds of data analysis and visualization • For different levels of decision making • Together, human operators and tools are a sociotechnical system • Human System Integration is required

24. Scaling Up Complexity

25. Findings: Cyber Security Defense Analyst Teaming • Cyber analysts work as a group – Not as a team • Collaboration among cyber operators is minimal • Little role differentiation • Bottom-up information flow • Possible Reasons • Cognitive overload • Organizational reward structures • “Knowledge is Power” • Lack of effective collaboration tools

26. Empirical Study on Teams vs. Groups

27. Hypotheses • Reward structures conducive to team work in cyber defense analyst groups performing triage level analysis will lead to higher signal detection performance. • Improving interactions between analysts (micro level) can improve overall cyber defense performance (macro level emergence)

28. CyberCog -Synthetic Task Environment • Task: team based triage analysis using the CyberCog simulation. • Synthetic Task Environment • Simulation environment • Recreate team and cognitive aspects of the task

29. CyberCog STE

30. The Experiment • 3-person teams/groupsin which each individual is trained to specialize in types of alerts • 2 conditions: • Team Work (Primed & Rewarded for team work) • Group Work (Primed & Rewarded for group work) • 6 individuals at a time • Team Work - Competition between the 2 teams • Group Work - Competition between the 6 individuals • Experimental scenarios: • 225 alerts • Feedback on number of alerts correctly classified - constantly displayed on big screen along with other team or individual scores • Simulates knowledge is power for individuals group condition • Measures Signal Detection Analysis of Alert Processing Amount of Communication Team situation awareness TransactiveMemory NASA TLX – workload measure

31. Results

32. Cyber Teaming is Beneficial for Analyzing Novel and Difficult Alerts • Working as team helps when alerts are novel and involves multi step analysis, not otherwise. • Signal Detection Measure: A' as performance measure • A' ranges from values 0.5 and 1 with 0.5 indicating lowest performance possible and 1 indicating highest performance possible.

33. Cyber Teaming Helps When the Going Gets Rough Sensitivity to true alerts F(1,18) = 5.662, p = .029** (Significant effect of condition)

34. Groups that Share Less Information Perceive More Temporal Demands than High Sharers • NASA TLX Workload Measure: Temporal Demand • Measures perception of time pressure • Higher the value higher the task demand Statistically significant across scenarios and conditions (p-value = 0.020)

35. Groups that Share Less Information Perceive Work to be More Difficult than High Sharers • NASA TLX Workload Measure: Mental Effort • Measures perception of mental effort • Higher the value, more mental effort required Statistically significant across scenarios and conditions (p-value = 0.013)

36. Conclusion • Break the “Silos” • Use the power of human teams to tackle information overload problems in cyber defense. • Simply encouraging and training analysts to work as teams and providing team level rewards can lead to better triage performance • Need collaboration tools and group decision making systems.

37. Agent-Based Modeling

38. Introduction • Human-in-loop experiment • Traditional method to study team cognition • Agent based model • Macro emergence • A complimentary approach • Modeling computational agents with • Individual behavioral characteristics • Team interaction patterns • Extend Lab Based Experiments

39. Model Description • Agents: Triage analysts • Task: Classify alerts • Rewards for classification • Cognitive characteristics: • Knowledge and Expertise • Working memory limit • Memory Decay

40. Model Description • Learning Process: Simplified – Probability based – 75% chance to learn • Cost: 200 points • Payoff: 100 points • Collaboration: Two strategies to identify partners • Conservative or Progressive • Cost: 100 points for each • Payoff: 50 points for each • Attrition

41. Model Process Team? Recruit if needed Assign alerts Yes No Adjust Expertise And Remove Analysts No Learn? Know? No Collaborate with Agents Yes Yes Add Knowledge Get Rewards

42. Model in Netlogo Software

43. Agents in the Progressive/Teamwork Condition Classified More Alerts(replicates experiment) p<0.001

44. Agents in Team of Six Classified More Alerts p = 0.004

45. Irrespective of Team Size Agents in Progressive Condition Classified More Alerts

46. Agents in Progressive Condition Accrued Least Rewards p<0.001

47. Agents in Small Teams Accrued Most Rewards p<0.001

48. Agents in Large Progressive Teams Accrued Least Rewards

49. Conclusion • Large progressive teams classified most alerts • Large progressive teams accrued least rewards • Big progressive teams • Lot of collaboration • Less learning • Constant knowledge swapping • More net rewards of 50 points • However small progressive teams accrued rewards on-par

50. Conclusions • Small heterogeneous teams of triage analysts could be beneficial. • Agent based modeling • Can extend lab based experiments • Can be used to ask more questions quickly • Can raise new questions and identify gaps