Objectives : Understand cognitive/contextual elements of situation awareness in cyber-security domains

1. Computer-Aided Human Centric Cyber Situation Awareness M. McNeese, D. Hall, N. Giacobe, V. Mancuso, D. Minotra, and E. McMillan • Objectives: • Understand cognitive/contextual elements of situation awareness in cyber-security domains • Implement a systems perspective to research linking real-world analysts with theory and human in the loop experiments • Utilize multi-modal research methodology • Focus on the humanand team elements within real context applications • Accomplishments • Developed framework /process for studying SA in cyber security via a Living Laboratory framework • Collected interview/survey data from practicing analysts • Implemented a simulation toolset for cyber SA to support human in loop experiments • Conducted experiments in transactive memory, dynamic task prioritization and visualization aids • Developed new SA evaluation metrics framework • Challenges • Rapid evolution of cyber threats and threat environments • Access to domain experts and state of the art practice • Modeling adversarial threats – integration in simulation • Scientific/Technical Approach • Living laboratory framework involving; • Ethnographic studies • Knowledge elicitation of domain experts • Development of cognitive and process frameworks and theories • Implementation of a scaled world prototype • Conduct of human in the loop experiments • Analysis and transition to real-world environments

2. Main Scientific/Technical Accomplishments Motivation • Improvement in Cyber SA requires focus on the ultimate limited resource: the human cyber analyst. This in turn requires understanding of the cognitive processes, the context, limitations and issues associated with perception, cognition and decision making for cyber SA Summary of Accomplishments • We conducted three experiments on; i) transactive memory, ii) task prioritization and iii) visual analytics in cyber-security and completed analysis efforts • We completed three PhD dissertations and one Master’s thesis • Developed new concepts for computer assisted SA using Complex Event Processing and Coherence Net Processing • Designed a new, general visual analytics workbench for cyber SA • We disseminated our findings via peer-reviewed journal articles, conference papers, edited book chapters, and presentations.

3. Task Statistics and Summary Students supported: • Four graduates/undergraduate students: Nicklaus Giacobe (50%), Vincent Mancuso (5 %), Dev Minotra ( 5 %), Eric McMillan (5 %), Tristan Endsley (10 %) and Erin Johnson ( 50 %) • Two faculty (D. Hall, M. McNeese) – Note: funding for all faculty provided by Penn State • Degrees awarded: (MS, PhD): E. McMillan (M.S.), V. Mancuso (PhD), D. Minotra (PhD) N. Giacobe (PhD) • Degrees in progress: E. Johnson (M.S.), Tristan Endsley (PhD) Publications: • Refereed journal papers - 2 • Conference papers – 3 • Conference presentations - 3 • Dissertations and Theses – 4 Technology Transitions: • Interactions with industry • Ethnographic studies/knowledge elicitation with network analysts working in education, military, government, and industry domains. • Briefings provided to several companies including: Deloitte, Lockheed Martin, Raytheon Corporation, GE, MITRE, Computer Sciences Corporation, and MIT Lincoln Laboratory • Interactions with other government agencies • Briefings presented to representatives from the National Security Agency (NSA), Defense Threat Reduction Agency (DTRA), Office of Naval Research (ONR), Department of Homeland Security (DHS), Department of Defense Intelligence Information Systems (DoDIIS), and Air Force Research Laboratory – AFRL – 711 Performance Wing 3

4. Publications, Honors and AwardsYear 4 • Peer-Reviewed Journals • Tyworth, M., Giacobe, N.A., Mancuso, V.F., McNeese, M.D. and Hall, D.L. (2013). A Human-in-the-loop Approach to Understanding Situation Awareness in Cyber Defense Analysis. ICST Transactions, 3 May 2013. • Cooke, N. and M. McNeese (2013), preface to special issue on the cognitive science of cyber defense analysis, editorial in EAI endorsed Transactions on Security and Safety, 13 (2), May 2013 • Tyworth, M., Giacobe, N.A., Mancuso, V.F., McNeese, M.D. and Hall, D.L. (2013). “A Human-in-the-loop Approach to Understanding Situation Awareness in Cyber Defense Analysis”, research article in EAI Endorsed Transactions on Security and Safety. 13 (2) May 2013 • Refereed Conference Proceedings • Mancuso, V., McNeese, M., “Effects of Integrated and Differentiated Knowledge Structures on Distributed Team Cognition”. (2012), Proceedings of the 56th annual Meeting of Human Factors and Ergonomics Society Annual Meeting, Boston, 2012 • Giacobe, Nicklaus A.; McNeese, Michael D.; Mancuso, Vincent F.; Minotra, Dev, "Capturing Human Cognition in Cyber-Security Simulations with NETS," Intelligence and Security Informatics (ISI), 2013 IEEE International Conference on , vol., no., pp.284,288, 4-7 June 2013 • Giacobe, N.A., “A Picture is Worth A Thousand Alerts”,Proceedings of the 57th annual Meeting of Human Factors and Ergonomics Society Annual Meeting, San Diego, 2013 • Presentations • Mancuso, V., McNeese, M., Effects of Integrated and Differentiated Knowledge Structures on Distributed Team Cognition. (2012), Proceedings of the 56th Annual Meeting of Human Factors and Ergonomics Society Annual Meeting, Boston, 2012 • Giacobe, Nicklaus A.; McNeese, Michael D.; Mancuso, Vincent F.; Minotra, Dev, "Capturing Human Cognition in Cyber-Security Simulations with NETS," Intelligence and Security Informatics (ISI), 2013 IEEE International Conference on Intelligence and Security Informatics, Seattle, WA, pp 284-288, 4-7 June, 2013 • Mancuso, V. and M. McNeese (2013), “TeamNETS: Scaled World Simulation for Distributed Cyber Teams”, poster session at the International Conference on Human Computing Interaction, HCII 2013, July 21 – 26, 2013 • McNeese, M. , Reset – Alternative Visions for Cyber Worlds, “Cognition and Cyber-Security” panel presentation, 57th annual Meeting of Human Factors and Ergonomics Society Annual Meeting, San Diego, 2013

5. Publications, Honors and AwardsYear 4 • Dissertations and Theses • N. A. Giacobe (2013), Measuring the Effectiveness of Visual Analytics and Data Fusion Techniques on Situation Awareness in Cyber-security, Ph.D. dissertation for the Pennsylvania State University, College of Information Sciences and Technology, May, 2013 • V. Mancuso (2012), An Interdisciplinary Evaluation of Transactive Memory in Distributed Cyber Teams, Ph.D. dissertation for the Pennsylvania State University, College of Information Sciences and Technology, August, 2012 • D. Minotra (2012), The Effect of a Workload-Preview on Task-Prioritization and Task Performance, Ph.D. dissertation for the Pennsylvania State University, College of Information Sciences and Technology, August, 2012 • E. McMillan (2012), Promoting the Use of Intelligence and Intelligence Analysis as Complementary Components to Enhance Situation Awareness in Cyber Security: A Qualitative Research Study of the Use of Analytical Techniques and Structured Methodologies by Cyber Security Experts, M.S. thesis for the Pennsylvania State University, College of Information Sciences and Technology, August, 2012

6. 4th Year Focus and Contributions • New theoretical frameworks for transactive memory, dynamic task prioritization & visual analytics • Completed 3 prototypes for experimentation & conducted human in the loop experiments • Conduct of 3 human-in-loop experiments • Developed new concepts for computer aids for cyber SA and supporting visual analytics workbench • Continued maturation and evolution of CyberCities/teamNETS/NETS-DARTDevelopment of synthetic test data • Evaluated a general framework and metrics for evaluation of SA

7. Test and Evaluation Environment • Two Laboratories in the Penn State College of Information Sciences and Technology enable human in the loop experiments with cyber situational awareness in individual and team environments • The Multidisciplinary Initiatives in Naturalistic Decision Systems (MINDS) Laboratory • The Extreme Events Laboratory

8. Promoting the Use of Intelligence and Intelligence Analysis as Complementary Components to Enhance Situation Awareness in Cyber Security E. McMillan, M.S. thesis • Objectives: • Understand cognitive/contextual elements of situation awareness in cyber-security domains • Understand how Intelligence and Intelligence Analysis methods can be utilized by cyber security experts in structuring their analysis and assisting in gaining control of situations they encounter • Focus on the human element within real context applications • Accomplishments • Conducted extensive literature review in cyber security, intelligence analysis, situation awareness and cyber situation awareness • Enhanced the Data, Information, Knowledge and Wisdom (DIKW) framework to include Intelligence analysis and created a taxonomy for cyber security • Collected interview/survey data from practicing cyber analysts from a Security Operations Center in a large healthcare organization in the western U. S. on their utilization of analytical and structured tools compared with Intelligence Analysts • Scientific/Technical Approach • Living laboratory framework involving; • Conduct of Ethnographic studies • Knowledge elicitation of domain experts, using NVivo tool for content analysis • Development of cognitive and process frameworks and theories

9. An Interdisciplinary Evaluation of Transactive Memory in Distributed Cyber Teams V. Mancuso PhD Dissertation • Objectives: • To understand how distributed cyber teams form, maintain and utilize transactive memory systems • To understand the behavioral, social and organizational outcomes of transactive memory systems in distributed collaborations • To determine how to design collaborative interfaces to better support transactive memory formulation, utilization and maintenance in distributed cognition • Accomplishments • Conducted literature search related to team-based cognition and transactive memory in distributed teams • Designed a scaled-world simulation called teamNETSfor a distributed cyber situation awareness team • Conducted a human-in-the-loop experiment with 66, 3-person teams to evaluate collaboration and transactive memory formulation and use • Conducted a quantitative assessment of transactive memory perceptions, utilization and content, situation awareness and team perception • Developed recommendations for transactive memory, collaboration tools for real-world environments • Scientific/Technical Approach • Living laboratory framework involving; • Ethnographic studies • Knowledge elicitation of domain experts • Development of cognitive and process frameworks and theories • Implementation of a scaled world prototype • Conduct of human in the loop experiments • Analysis and transition to real-world environments

10. The Effect of a Workload Preview on Task-Prioritization and Task Performance D. Minotra PhD Dissertation • Objectives: • To understand the impact of high mental workload on cyber-security analyst performance • To determine the utility of task-prioritization cognitive aids for improved focus of attention and facilitation of performance under high time-pressure • To evaluate the effectiveness of a workload preview aid for improved performance • Accomplishments • Conducted a literature review of attention-guidance, task-management, interruptions and workload previews • Created a scaled-world simulation (NETS-DART) to emulate cyber-security monitoring and decision-making • Conducted a human in the loop experiment involving 77 participants under different task-load conditions to determine the effectiveness of a workload preview aid on task performance • Developed recommendations on factors that influence the effectiveness of cognitive-aids aimed at guiding attention and improving task-performance • Scientific/Technical Approach • Living laboratory framework involving; • Ethnographic studies • Knowledge elicitation of domain experts • Development of cognitive and process frameworks and theories • Implementation of a scaled world prototype • Conduct of human in the loop experiments • Analysis and transition to real-world environments

11. Measuring the Effectiveness of Visual Analytics and Data Fusion Techniques on Situation Awareness in Cyber Security N. Giacobe PhD Dissertation • Objectives: • To determine effective methods for measuring the impact of interface design on situation assessment for the cyber domain • To create a prototype visualization aid for cyber situation awareness • To determine the effectiveness (on inference accuracy and speed) of an implemented visual aid prototype • Accomplishments • Conducted a literature review of cyber situation awareness, data fusion and visual analytics for cyber SA • Conducted an ethnography study and knowledge elicitation of 60 IT professionals regarding tool utilization • Developed a visual analytics cyber SA artifact using a Geo-Visualization toolkit • Developed a framework for measuring the efficacy of cognitive aids and collaboration tools based on SAGAT, SART, HPSM and NASA-TLX • Conducted a human in the loop experiment with 25 subjects in a 2x2 experiment to evaluate effectiveness of a visualization tool for cyber situation awareness • Scientific/Technical Approach • Living laboratory framework involving; • Ethnographic studies • Knowledge elicitation of domain experts • Development of cognitive and process frameworks and theories • Implementation of a scaled world prototype • Conduct of human in the loop experiments • Analysis and transition to real-world environments

12. Ongoing research • Extending the modeling concept from passive to active/prediction • Previous work focused on IDS/Security Analysts • New work shifts to the Threat Analysis Analysts • Extending the threat model to include merging of hard and soft data • How can we use textual data (blogs, tweets, etc.) to inform what kinds of hard data (IDS, flow, logs, etc.)? • Extending into the threat domain analysis of cyber security • How does this kind of cyber-security work differ from the front-line intrusion detection analyst? • Exploration of automated processing aids including CEP and Coherence Network modeling • What tools can we apply to address the needs of this new domain by supporting the human’s fusion and awareness? • Initial concepts for visualization and sonification support • What visualizations support sense making and decision making in the threat analysis domain?

13. Extending the Threat Model Text extraction & processing Human obs. Analysis Tools (CEP, CNP, etc.) Search engines

14. New Concepts for Cyber Trans-Action • Developing Extensible Cognitive Artifacts to Support Distributed Cyber Security Work • Objectives: 1) Building Work Support Tools based on Socio-Technical Systems • 2) Design Ecological Group-Based Interaction Displays to test in teamNETS • 3) Adversarial Team to Team Interaction • Functional Abstraction Hierarchy (Rasmussen, Sanderson) for Cyber Operations • - Functional work in cyber detection with other cross-function areas (threat assess) • Decision Ladder to portray decision making flow and adaptations across team members • Ecological Interface Design for teamNETS to improve SA under changing conditions • - context switching given adaptable emerging priorities • • Models of the Adversary to be embedded within attack framework given above artifacts • - created through hidden knowledge profiles (static implementation) • - extended to intelligent monitors who spoof and deceive the team-force

15. Complex Event Processing (CEP)for Cyber Situation Awareness • Complex event processing (CEP) • Originally created for financial and stock trading applications • Subsequently applied to smart energy, RFID middleware and limited data fusion applications • Capabilities for rapid rule-based filtering, aggregation and event detection • Allows hierarchy of “levels of analysis” Low-level events filtered and aggregated into a higher-level event

16. CEP Processing Infrastructure

17. CEP/CNB Interaction

18. Technology Definitions StreamBase CEP Engine: a performance-optimized, Java-enabled Complex Event Processing (CEP) framework. (see http://www.streambase.com) JADE: A Java-enabled Multi-Agent System (MAS) framework that enables creation, utilization, and administration of scalable “communities” of software agents. (see http://jade.tilab.com) AMQP/RabbitMQ: The Advanced Message Queuing Protocol (AMQP) is a Message Oriented Middleware (MOM) for advanced, context-dependent routing of messages between multiple software tools/nodes. RabbitMQ is an open source client/server implementation of AMQP. (see http://www.rabbitmq.com) TML/EML: The Open Geospatial Consortium (OCG) standard Transducer Markup Language (TML) and Event Pattern Markup Language (EML) protocols encourage structured, standardized metadata and allow snapshots of complex event patterns (EML only). (see http://www.opengeospatial.org/standards/dp) Mulgara: A semantic data store that is optimized for storage and access of “tuples” rather than strictly relational data. (see http://www.mulgara.org)

19. An Emerging Visual Analytics Workbench Network Activity View Data & message view Social Network View Task: Identify Structure and Methods of Syrian Electronic Army Identify individuals, methods used and individual events. Make predictions on whether our organization will be impacted by the SEA. Check the current network and server status for evidence of similar methods being used against our organization. – Website defacement – DNS Hacks – XSS Examples Analysts Notes Timeline View

20. 5th Research Plan • Develop/refine an abstraction hierarchy and decision ladders to represent team cognition/adversarial model dynamics • Refine and evaluate evolving cognitive aids and visualization tools • Complex event processing (CEP) • Coherence Network Builder (CNB) • Emerging visual analytics workbench • Conduct human in the loop experiments related to adversarial dynamics in distributed cyber SA

21. 5th Year Analytical Study:Abstraction Hierarchy /Decision Ladder Composite Models Goals: (1) build an abstraction hierarchy to model cyber security from the socio-technical systems perspective (2) derive specific decision ladder model to represent team cognition – adversarial model dynamics Primary Approach: utilize a composite strategy taking wholistic knowledge across qualitative/quantitative MURI studies to integrate and distill an overall model that provides: ° basis for creating ecological interface designs for team cyberSA ° identification of research gaps for future work in cyber SA areas Outcomes: integrates cyberSA with cognitive systems engineering approaches, qualitative-quantitative-design-modelinh methods are integrated to form systems perspective

22. 5th year Experiment:Adversarial Dynamicsin Distributed Cyber SA Objectives: • Do team members use adversarial intelligence in cognitive processing ? • How do specific models influence cyber SA, decision making patterns, and human performance? Study:Using hidden profiles to emulate adversarial models ° use of the existing teamNETS simulation with medium level time pressure ° IV#1: Extent of Knowledge of Adversarial Intelligence - simple model -- one member has advantageous intel information - compound model -- two members have advantageous intel information - complex model -- three members have advantageous intel information ° IV#2: Extent of Visual Analytics Support (present/absent) ° Dependent variables (DV.s) - When (if) do members use their intelligence? - When does information congeal together in shared form? - When does intelligence of adversary influence performance?

23. Personal ComputerSecurity MOOC - • Developed by IST and Dr. Gerald Santoro, a top expert on cyber-security • Asynchronous delivery and assessment • 4 modules, each with 3 topics • Online quiz for each module • Content includes topic introductions, reading materials and video lectures • Certificate awarded for successful completion of quizzes • Future capability of instructor interaction