1 / 26

Flow Data Tools and Analysis at Fermilab

This talk outlines the flow data collection and analysis system at Fermilab, including security tools, performance estimation tools, and checking of traffic for PBR’d circuits.

tmalone
Download Presentation

Flow Data Tools and Analysis at Fermilab

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Flow Data Tools and Analysis at Fermilab Andrey Bobyshev / Phil DeMar Internet2/ESCC Joint Techs Workshop Fermilab, July 15-19, 2007

  2. Outline of the talk: • Flow data collection & analysis system at Fermilab • Security tools • Performance estimation tools • Checking of traffic for PBR’d circuits

  3. Netflow Collection and Analysis system Local RAID6 Fermilab Core Services • Based on flow-tools (OSU) • Collecting data from: • Border routers: • 1min flow time outs • Internal core routers and large experiment routers: • 5 min flow time outs • Specific collector for “near” real-time tools/applications • Central storage system accumulating all flow data • Multiple systems for primary processing • Results stored in SQL tables NetFlow Storage EnStore Flow collector Real-Time Appls Long-Term Archiving real-time replication BlueArc NAS 5min samples 1min samples Border & StarLight CMS WorkGroup & Core mySQL Server primary processing data, Application's data Web Presentation Processing and Analysis systems

  4. Data Collection details • ~2.5GB to disk daily • Older data are archived on EnStore, Fermilab’s tape storage facility • Complete flow data collection, not sampled • 10GE backbone & offsite links… • Impact on routers is minimal

  5. Breakdown of traffic and tagging process Origin: onsite, offsite, local, transit Target: CMS, D0, CDF Filter: particular remote site or group of sites. Ex. Caltech, Tier2, US-Tier2 and etc.. Applications: topN, Network Weather Map, ... tableID: router,origin,target,filter,DNS Level mySQL SrcDstOctets SrcDstFlows SrcDstPackets and more Raw data sets accumulated for 1min,5min, 15min intervals Tagging Sources and Destinations are identified by DNS name (host, top level,second level and so on or statically assigned labels

  6. Security Tools • AutoBlocker – quasi real-time detection and automatic block/unblocking onsite and offsite scanners • Automated offsite blocking based on “greedy” data flow pattern • Automated unblocking ‘x’ minutes after behavior stops • Top Scanners GUI • Slow Scanning detection • Raw Flow reader – packets exchange

  7. BLOCK RED Calculate metrics WATCH ORANGE NOTICE YELLOW NONE BLUE GREEN NO scanning – NO actions Evaluate triggers to return threat level AutoBlocker – automatic detection and blocking/unblocking of offsite and onsite scanners The main idea of AB3 is calculating multiple quantified metrics from netflow data to use it for making automated decision on blocking and ublocking of offsite and onsite scanners. In October of this year it will be 5 years since AutoBlocker has been deployed.

  8. Metrics/Triggers/Threats/Actions Metrics: Actions: Triggers: • ipDestinationAddressCount • ipDestinationPortCount • ipSourcePortCount • blockCount • activeBlockCount • detectionCount • consecutiveDetection • consecutiveWatch • watchRate • flowsIn • flowsOut • HitByRemotes • excessivePrcTime • tcpSourcePortOut • tcpSourcePortIn • tcpDestPortOut • tcpDestPortIn • udpSourcePortOut • udpSourcePortIn • udpDestPortOut • udpDestPortIn • excessiveHostCount • excessiveDestinationPort • flowsResponseInconsistency • portScanFlowsResponse • excessiveProcessingRate • DatectionRate • consecutiveDetection • watchRate • consecutiveWatch • BLOCK/unBLOCK • watch/resetWatch • NONE/flushNONE • NOTICE Triggers return the threat identified by a color. Threats are mapped into actions BLOCK RED WATCH ORANGE NOTICE YELLOW NONE BLUE GREEN NO scanning – NO actions

  9. Definition in terms of AB metrics + IP Blocks Definition in terms of AB metrics + IP Blocks AutoBlocker Exceptions System Exceptions System Events with originally assigned actions NO exception found An exception found, original action is converted into unharmed action such as NONE, NOTICE. Evaluating of events triggered actions against defined exceptions Reversed Exception: an “unharmed” action can be converted into BLOCK: AB has triggered an event but did not meet BLOCK criterion. However, AB-Exception system determines a potential dangerous application that needs to be BLOCKed.. Multiple Classes of Exceptions Multiple classes of exceptions: • Network, based on CIDR IP Blocks • Applications defined by combination of event's metrics and specified IP blocks • Groups of applications Network Groups of Applications Applications Core Servers Definitions of applications can be created statically or dynamically Static Definitions Dynamic Definitions Traffic Determine usual traffic behavior

  10. External AutoBlocker detectors • Several external AutoBlocker detectors: • DarkNets - analyze traffic to unallocated Fermilab networks and generate alerts to AB3 via SOAP • SlowScan – detects slow scanning by analyzing flow for a longer periods (1hour, 1 day) and generate alerts to AB3

  11. Raw Flows Reader • WEB interface to generate raw flow data based on specified criteria, such as time range, port, source/destination addresses • Typical use is for forensic analysis of computer security incidents • Access to the tool (and raw flow data itself) is restricted

  12. Sample of RawFlow Output

  13. TopScan: Generate tables of topN Scanners TopScan – on per origin basis (onsite, offsite, local, transit) generate tables of top scanners for specified time intervals: 5min, 1hour, 1day. Information is available via interactive GUI and by E-Mail notifications

  14. Performance Monitoring & Estimation tools • WEB USCMS Network Weather Map • topN • Traffic Summary (ftsumTraffic) • Traffic asymmetry (bfpsum) • Multistream flow analysis

  15. USCMS Network Weather Map Show estimated rates to various sites: Tier0, other Tier1, USCMS Tier2. Features: • popup graphs • clickable icons to direct to other informational sources

  16. USCMS WM : popup graphs Place cursor over UNL icon - Utilization graph appears

  17. USCMS WM: Popup graphs, 16Gbps Place cursor over central USCMS icon - Aggregate Tier-1 center traffic graph appears

  18. USCMS WM: Group's rates

  19. USCMS WM: Clickable icons Click on BlueArc Icon: hourly summary tables for TopN pairs, senders and receivers

  20. USCMS WM: TopN conversations Tables of hourly topN senders, receivers & conversations

  21. bfpsum: ByteFlowPacket Summary bfpsum allows to build graphs and tables for traffic of specified targets, such as USCMS to the various remote sites. Single or multiple routers can be selected as well as multiple targets and filters. Traffic can be seen in the terms of bytes, flows and packets. Both rates or amount can seen.

  22. bfpsum: Verifying symmetry of PBR-ed traffic This tool is used for interactive inspection of USCMS PBR-ed traffic to detect potential asymmetry. When traffic is symmetric flow rates of inbound and outbound traffic is practically the same (see graph on the previous slide). An example of traffic asymmetry is graph on this slide (caused by Caltech when LS was shutdown and outbound traffic was going through the core network.

  23. Test: detection of traffic asymmetry WAN r-s-starlight-fnal (E2E circuits…) r-s-bdr (routed IP via ESnet) r-cms-fcc2 USCMS Tier1 normal (E2E) traffic flow LambdaStation is turned off, no PBR

  24. Breakdown of multistreams GridFTP sessions ftGftp: detects and estimates transfer rates for multistreams gridFTP sessions. - Filtering on remote sites can be selected first before passing it to the detector.

  25. Commercial Products • Always looking for commercial or public domain packages of comparable functionality: • Most commercial packages have similar capabilities & some useful features, but not flexible enough for our purposes • Evaluated AdventNet Netflow Analyzer & NetFlow Tracker from Crannog-Software • Purchased AdventNet , ~$1K for 20 interfaces, allows to define IP groups based on the list of IP blocks

  26. Future flow data developments • Maintaining the existing scope of monitoring • Automate asymmetric path analysis • Integrate flow data analysis into our network performance troubleshooting methodology • High impact data movement detection • Lesson learned from Lambda Station: application awareness is hard • Wouldn’t it be nice to have the network detect recognizable flow patterns and modify path/service/whatever, if appropriate? • But it almost certainly would require real time flow data • Would be happy to collaborate with others developing flow data tools: • Contact us at wan@fnal.gov

More Related