Processing intelligence feeds with open source software
This presentation is the property of its rightful owner.
Sponsored Links
1 / 57

Processing Intelligence Feeds with Open Source Software PowerPoint PPT Presentation


  • 109 Views
  • Uploaded on
  • Presentation posted in: General

Processing Intelligence Feeds with Open Source Software. Chris Horsley, SC Leung, Tomas Lima, L. Aaron Kaplan, Raphael Vinot. Overview. Current topics in automatic incident handling for CERTs IFAS HKCERT , IFAS and use-cases IHAP project ContactDB project Current R&D. IFAS.

Download Presentation

Processing Intelligence Feeds with Open Source Software

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Processing intelligence feeds with open source software

Processing Intelligence Feeds with Open Source Software

Chris Horsley, SC Leung, Tomas Lima, L. Aaron Kaplan, Raphael Vinot


Overview

Overview

  • Current topics in automatic incident handling for CERTs

  • IFAS

  • HKCERT , IFAS and use-cases

  • IHAP project

  • ContactDB project

  • Current R&D


Processing intelligence feeds with open source software

IFAS

  • Information Feed Analysis System


Knowing what s going on

Knowing what’s going on


How do national csirts know what s happening

How do national CSIRTs know what’s happening?

  • National CSIRTs need visibility on network in their economy

  • However, many national CSIRTs don’t operate networks themselves, and normally don’t have global (or any) direct visibility

  • How does the CSIRT know what’s going on in their country?


The kindness of strangers

The kindness of strangers

  • Luckily, there are a lot of network operators, research teams, vendors, and other CSIRTs out there that collect information, and will share it with national CSIRTs.

  • And here comes the “but”...


So much data so many formats

So much data, so many formats

  • There are many feeds, all with their own data formats and mediums:

    • Formats: CSV, JSON, XML, STIX, IODEF

    • Mediums: HTML, RSS, email, HTTP APIs

  • While there are efforts to standardise data formats, this will take a long time, and will likely never cover 100% of feeds

  • We can’t change the format of remote feeds - we can only change what we do with the data.


The need for standards

The need for standards

  • Different feeds use many terms to mean the same thing:

    • ip, source_ip, src_ip, endpoint, attacker_ip, cnc_ip...

  • If we receive events from many feeds, we need to normalise so we can compare them together.


The need for storage

The need for storage

  • As a national CSIRT, we’re concerned with the health of national networks: which means measurement.

  • We can only measure longterm if we store events, enabling us to analyse them.

  • We also want to search through events, like:

    • C&C servers in domestic networks in last week

    • Bots infected with Trojan.abc on BigISP

    • Defaced web sites targeting gov.zz


Need for automation

Need for automation

  • There’s way too much network event data out there to manually process

  • Options:

    • a) use lots of analyst time doing tedious log processing

    • b) write lots of small, independent scripts

    • c) ignore inbound logs completely

    • d) use an automated processing system


So what do we need

So what do we need?

  • We need something which automatically:

    • Gathers many different types of feeds

    • Normalises the data in those feeds

    • Stores that data somewhere

    • Allows search and performs statistical analysis


Processing intelligence feeds with open source software

IFAS

  • IFAS = Information Feed Analysis System

  • Project sponsored by HKCERT and developed by HKCERT and CSIRT Foundry

  • An integration of open source tools, released as open source for CSIRTs


Architecture

Architecture


Architecture1

Architecture

  • Abusehelper: gather, process, and enrich feeds, generate events

  • Logstash: process and normalise feeds

  • Elasticsearch: store events in schema-free index server

  • Kibana: search through events

  • IFAS Reporter: get overall statistics, build realtime dashboards


Kibana event searches

Kibana event searches


Freeform statistical reporting

Freeform statistical reporting


Nesting filtering deduplication

Nesting, filtering, deduplication


Ifas dashboard

IFAS – Dashboard

*Drill down right at the chart

  • Visualize information


What you need to start

What you need to start


Software

Software

  • Open source under Apache 2.0 License

  • Only possible with the hard work released under open source licenses from Abusehelper and Elasticsearch teams

  • Contributions, bug reports, feature requests most welcome!


Hardware

Hardware

  • Production: 8-16GB memory machine

  • Dev: 4GB possible

  • Multi-core machine (4+ ideal)

  • Runs in a VM no problem


Out of the box feeds

Out of the box feeds

Out of Box Feed Plugins

(4 publicly available)

  • Abuse.ch

  • CleanMX

  • Millersmiles

  • Phishtank

Other developed Plugins

  • Malc0de

  • Malicious Domain List

  • Arbor SRF

  • Shadowserver

  • Zone-H

    Future

    … more, and your own


Where to get it

Where to get it

  • Currently under closed pilot to trusted CSIRTs

    • Eventually public release

  • Please contact [email protected] for details


Demos

Demos


Ifas and use cases

IFAS and Use Cases

SC Leung, HKCERT


Give a sense of today s events

Give a sense of Today’s Events


Ifas log search

IFAS - Log Search

  • Powerful search on all the information collected

Feed Details

Keywords here

Add columns of

interests


Ifas reporter

IFAS - Reporter

  • Statistical analysis-Trends & Distributions

    • Free form statistical reports

3.

4.

5.

2.

1.

6.


Nesting filtering deduplication1

Nesting, filtering, deduplication

Number of phishings in “.AU” in each ASN by brand


Ifas alert

IFAS - Alert

  • Set tracking criteria – get notify ASAP

    • domain: *.gov.hk

    • Alert lists : educational institutions (hkedu), NGOs (hkorg)

!


Dashboard

Dashboard

Real-time situational awareness for CERT management


Public situational awareness on compromised servers pcs

Public Situational Awarenesson Compromised Servers / PCs


Hong kong security watch report

Hong Kong Security Watch Report


Analysis of trend with events

Analysis of Trend with Events

  • Correlate Cryptolocker 2013-Oct with Zeus


Engage isps for large scale incident handling

Engage ISPs for large scale incident handling

ISP

  • Data do help HKCERT engaging ISPs (their sales team)

  • Data do help a server hosting SP understand their customers’ security problems


Converting security events into incident reports

Converting security events into incident reports

  • Defacement

  • Phishing

  •  Export to CSV for batch processing, with some other scripts

  • Malware hosting – a bit difficult

  • Large volume of incidents – need prioritisation


Future of ifas a collaboration platform

Future of IFAS - a collaboration platform

  • All you can use

  • All you can contribute

    • Add input filters for new feeds

    • Add new plug-in modules

    • Add new chart and visualization

    • Integrate with other systems, e.g. RTIR

  • Standard language: STIX, taxonomy of ENISA


Dsms decision support monitoring system

DSMS (Decision Support & Monitoring System)

  • An ongoing project that turn security events into Actionable Data

    • Set Priority, Choose Monitors, Consolidate Results

Monitoring

Services

Profile

Status Check

(HTTP, DNS) via proxy

Input  URL

Tasks

Status ?

Interface Module

Decision Support

Sub-system

(online /offline)

Interfaces to Monitors

Interface Module

Public analysis sys (VirusTotal, ThreatExpert)

IFAS

Request to monitor

Interface Modules

Story

Private analysis sys

Interface Modules

Web reputation

(D-Shield)

Incident

Mgmt

Output

Consolidated Results


Processing intelligence feeds with open source software

Incident handling automation project

IHAP


Processing intelligence feeds with open source software

IHAP

  • Very similar to IFAS, developed in parallel by CERT.pt, CERT.at

  • Also uses Logstash, Elastic Search and Abusehelper

  • Less work on the Webinterface, more work on Ontology, „Data harmonisation document“


Ihap history

IHAP - History

  • Discussions about CERT.AT developments/documents

  • Discussions about cooperation between CERTs

  • ENISA support


Ihap goals

IHAP - Goals

  • Open Source

  • Maintainable

  • Flexible and Modular - must be possible to integrate existing software and modules (Pastemon, AbuseHelper, etc..)

  • Reusable

  • Easily Extendable - should require little knowledge and basic programming skills

  • Easily Deployable

  • Easily Updatable – easy to share new developments with other CERTs and update the system with that new code

  • Easily Configurable - config files that can be easily modified to fit CERT‘s needs

  • Documented - must be well documented


Links code

Links & Code

http://www.enisa.europa.eu/activities/cert/support/incident-handling-automation


Common field names for ah

Common field names for AH

  • https://bitbucket.org/clarifiednetworks/abusehelper/wiki/Data%20Harmonization%20Ontology

  • A standard set of well defined field names within Abusehelper (AH)

  • Allows CERTs to:

    • Write bots which are interoperable within AH

    • Measure in identical ways

    • Easier to parse different feeds („generic santizer bot“) : you just have to define the mappings


Contactdb

contactDB


Background problem

Background/ problem

  • abuse@ lookups suck (IRT object not in use, no standard; Just now RIPE DB is changing with abuse-c:)

  • Getting the right lookup is non-trivial, complex

  • Many (national) CERTs create their own abuse contact lookup DBs.

  • National CERT DB, TI directory, FIRST data can not be looked up automatically via scripts.


Processing intelligence feeds with open source software

Idea

  • A caching contact database with more specific internal data

  • Some of this data (tel nos, etc) will never be in the public whois

  • Unify with TI, FIRST etc data

  • Make it query-able by scripts


Abuse contact lookup flow

What databases exist? What can we query?

Abuse contact lookup - flow


Processing intelligence feeds with open source software

Number based resource:IP addr, netblock, ASN

Name based resource:domain name, hostname

TI, FIRST, CERT.org DBs

Get country()

Gethostbyname()

Extract ccTLD

IANA ccTLD list

Whois DB (RIPE, ARIN, ..)

Maxmind

RIPE DB

Cymru, ...

Whois DB (registrant, registrar)

National CERT DB

CERT.org

Country code

Country code

National CERT for country

IRT object, abuse-c, ...

Email Address


What exists now

What exists now?

  • Public code repo ;-)

  • Whois server (thx Mauro)

  • RESTful API (Mauro, Rafiot)

  • Some scripts to import TI data (Aaron, David)

  • Still some bugs ;-)


Code document with ripe

Code & document with RIPE

  • Document (WIP):

  • https://github.com/certtools/contactdb/blob/master/doc/contact-databases-for-abuse-handling.mkd

  • Codebase:https://github.com/certtools/contactdb

  • (thx Rafiot, David, Mauro!)


Summary

Summary


Summary1

Summary

  • The CERT community has limited ressources for development

  • We re-implement the same thing all the time

  • Let‘s share code or at least exchange ideas on how to automate incident handling!

  • Let‘s share on how to measure success

  • Thanks HKCERT, ENISA, CERT.at, CERT.pt, CIRCL, etc..

  • Mailinglist: https://tiss.trusted-introducer.org/mailman/listinfo/ihap


Thanks

Thanks!


  • Login