WLCG SOC WG Workshop 19-21 February 2019

1. WLCG SOC WG Workshop19-21 February 2019 Summary

2. SOC WG Introduction • Working group designed to enhance site security monitoring in light of virtualized environments (including containers) • Network monitoring • Coupled with threat intelligence and real time search capabilities • Minimally viable Security Operations Centre

3. Introduction • Summary of most recent SOC workshop • https://indico.cern.ch/event/775579/ • The Cosener’s House, Abingdon, 19-21 February • Supported by GridPP and Scientific Computing Department, UKRI – STFC

4. Introduction • Same format as workshop in June last year • 2.5 days • Half day introduction • Two days of technical sessions

5. Goals • Key outcomes • Finalise initial SOC model • Or identify concrete remaining steps • In particular identify any integrations required and complete documentation

6. For the future • Two areas of work for the group • Technology stack • Sharing of threat intelligence • This workshop focused on the technology stack • Aim to concentrate on threat intelligence over coming months and next workshop

7. (Carefully) Growing Scope • Originally mandated to give guidance to WLCG sites • Area of work enhanced by including neighbouring communities • NRENs • University CSIRTs • Hoping to involve EGI Fedcloud

8. (Carefully) Growing Scope • Originally mandated to give guidance to WLCG sites • Area of work enhanced by including neighbouring communities • NRENs • University CSIRTs • Hoping to involve EGI Fedcloud

9. Status talks • Protective • H2020 project • https://protective-h2020.eu • Oxford Physics update • CERN SOC demonstration and update • NCBJ update • AGLT2 update

10. Elasticsearch and integrations • AGLT2 use of Elastiflow to ingest netflow • Also works for sflow • RAL Elasticsearch deployment • Walked through integration steps

11. Alerting • Shawn and Liviu worked through final day to deploy CERN alerting scripts at AGLT2 • Successful! • Can be shared with other sites with a little more work

12. PocketSOC • SOC demonstrator • Docker cluster designed to run on a laptop • Essential components and network components • Minimal traffic to demonstrate workflow • Test new components • https://gitlab.cern.ch/wlcg-soc-wg/PocketSOC

13. PocketSOC

14. PocketSOC • VM made available at workshop • In the process of a few updates then at least making it available on request

15. New developments • Project to explore a SOC deployment at Nikhef (a student working on it) • Another project to deploy a SOC at the STFC Cloud – graduate starting this week • also working on other aspects of the Cloud

16. Initial Model

17. Initial Model • Data ingestion • At least one of • Zeek (Bro): deep packet inspection • Netflow: network metadata

18. Data processing and pipelines • Threat Intelligence • MISP [Essential] • Log ingestion pipelines • One per data source using Logstash

19. Storage and visualisation • Elasticsearch [Essential] • Kibana [Essential]

20. Alerting • At least one of • Enrichment, correlation and aggregation scripts based on CERN example • Elastalert • Trigger on Elasticsearch query • Spike of events, for example

21. Immediate future • Healthy set of actions to improve documentation • Move select repositories outside of CERN (Github/Gitlab.com) • Improve access for non-CERN users • Make contributing as easy as possible • Gather everything together, ideally by ISGC

22. Next few months • Focus on threat intelligence • Workshop later in the year • Hopefully test entire chain • WLCG → Site → Event detection • Discussed how to do this in a test context

23. Final thoughts • Fantastic to have more sites trying out deployments • Start thinking about how we might want to deploy • Always welcome new participants

24. Questions?

25. Backup slides

26. Introduction • Provide some background and look at goals of the workshop • Mindful that we have a mix of previous and new participants • This will also take us through the sessions

27. Background • SOC Working Group: • Identify need to monitor cluster environment in a new context which can include virtualised / containerised systems • Potentially more opaque than existing grid systems • Network monitoring key to understanding cluster state

28. Security Operations Center • The purpose of a Security Operations Center (SOC): • Gather relevant security monitoring data from different sources • Aggregate, enrich and analyse that data for use in the detection of security events and any subsequent actions  • A SOC consists of a set of software tools and the processes connecting them

29. Two strands • Technology stack • Forming the focus of this workshop • Sharing threat intelligence • Focus of next workshop

30. CERN SOC • Closely following and benefitting from work on the CERN SOC • Common features: • Data ingestion • Data analytics • Data storage (short term and long term)

31. CERN SOC

32. Initial model • SOCs are complex (see also Metron project) • Build the full SOC model over time; start with key components

33. Two questions • What is happening in my cluster? • What events are taking place that we need to care about? (internally or externally)

34. Two questions • What is happening in my cluster? • What events are taking place that we need to care about? (internally or externally)

35. Network Monitoring • IDS: Zeek (previously Bro) • Deep packet inspection • Wide use in the US • 100 Gbps setup at Berkeley Lab • https://commons.lbl.gov/display/cpp/100G+Intrusion+Detection • Flexible & Scalable • Configure as single node or cluster

36. Network Monitoring • Netflow and sflow • Network metadata rather than deep packet inspection • Provided by many switch vendors and software clients

37. Two questions • What is happening in my cluster? • What events are taking place that we need to care about? (internally or externally)

38. Threat Intelligence • Second major strand of the working group • The future of academic security (Romain Wartel) • http://indico.cern.ch/event/505613/contributions/2227689/attachments/1349009/2047093/Oral-109.pdf WLCG SOC WG

39. Threat Intelligence

40. Threat Intelligence • Collaborative response • In particular, one goal of this group is to explore collaboration between grid and institute / campus security team

41. MISP • Malware Information Sharing Platform • “A platform for sharing, storing and correlating Indicators of Compromises of targeted attacks. Not only to store, share, collaborate on malware, but also to use the IOCs to detect and prevent attacks.” • Allows development of trust frameworks between sites to allow rapid sharing of threat intelligence • misp-project.org • https://github.com/MISP/MISP

42. Initial model • Build the full SOC model over time; start with key components • Network monitoring • Zeek (Bro) and Netflow/sflow • Threat Intelligence • Malware Information Sharing Platform (MISP) • Storage, search and visualisation • Elastic stack (ELK)

43. Working Group Goals • In that context, WG looking to build stack appropriate for different sites • Alongside, consider way of working to best incorporate these components

44. Workshop Goals • Work now towards initial version of the SOC model: • Network data sources • Storage, search and visualisation • Alerting • Integration of components

45. Next steps • Where to expand the SOC model next?

46. CERN SOC

47. Important questions • Once a security incident is detected how can we get the full picture of the incident (when exactly it started, what’s the extent of the incident, etc)?

48. Important questions • Important questions which need answers through this work (not just today!) • What data do we need? • What sources of data do we need (intersection with traceability) • Where / how to tap network? • How to handle data sharing / protection for different user groups • How to consider different contexts: • Institution / NGI / WLCG / Other • Critical to include sites of different type in this work