1 / 62

NetFlow and flow-tools @ OARnet

NetFlow and flow-tools @ OARnet. Mark Fullmer maf@splintered.net. Agenda. OARnet - who we are, what we do. Flow-tools summary - software we use to collect and process NetFlow data. NetFlow deployment at OARnet.

neona
Download Presentation

NetFlow and flow-tools @ OARnet

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NetFlow and flow-tools @ OARnet Mark Fullmer maf@splintered.net

  2. Agenda • OARnet - who we are, what we do. • Flow-tools summary - software we use to collect and process NetFlow data. • NetFlow deployment at OARnet. • How we use NetFlow. Detailed examples with flow-tools. Hopefully some tips that can be used or passed on to customers.

  3. OARnet • Internet access to about 100 of Ohio’s colleges and universities. • Some commercial clients / co-lo services. • Internet2 / Abilene access to member school’s in Ohio. • Sink over 1Gb/s of traffic from the Internet during peak hours.

  4. OARnet • Multiple Transit providers, access to two of the NAPs, OC48 to Abilene. • In the middle of a state wide network rebuild, moving from ATM circuits over leased lines to POS and GigE over DWDM using Cisco 15454’s.

  5. Flow-tools • Started back in June of ‘96 working from Darren Kerr’s fdg.c (a very simple example of displaying NetFlow v1 data). No commercial or open source NetFlow collectors available at the time. • Released as an open source toolkit for working with NetFlow data from Cisco routers. At the time mostly used internally at Ohio State University, usually for network forensics.

  6. Flow-tools • Still available as open source, active development on and off as resources (time) are available. Keep working it as long as people are interested. • Many other options available now for collectors and post processing both open source and commercial. • A lot of sites use it, over 700 subscribers to the support mailing list.

  7. NetFlow deployment • Collector running FreeBSD and flow-capture at each POP (distributed collection). • Sampled NetFlow data feeds (where software permits). • Flow-capture stores the exports to disk with line compression at 5 minute intervals. • Some reports run on the collectors, some at back at a central server (combination of distributed and centralized report generation).

  8. NetFlow deployment OC48’s to other POPs. Core router Flow collector. 80GB RAID storage. Aggregation routers.

  9. NetFlow At Work

  10. Traffic Engineering • We have multiple transit providers with various contractual obligations. • With provider T we must purchase a minimum of 300Mb/s / month. • Provider Q recently reduced their prices to half of provider T. • So we want to minimize our transit costs by using provider T for no more than 300Mb/s during peak hours until contract runs out (or they reduce their prices).

  11. Traffic Engineering Clearly above 300Mb/s on on outbound traffic, a little over on inbound.

  12. Traffic Engineering • 170 prefixes that are announced to the internet. • Prefix length does not necessarily correlate to traffic load. I.e. some smaller school’s with /16’s. • So to influence provider T to send us less traffic to a prefix, pad the AS path.

  13. Traffic Engineering • Use NetFlow to determine which prefixes to pad. • Select traffic that has an input interface of the provider circuit. • Add local masks. • Scale the packets and octets by 100 (sampling rate is 1/100). • Summarize on destination IP & mask (zero out the host bits).

  14. Traffic Engineering Collected flows(flow-capture). Select a days data (flow-cat) Filter on provider interface (flow-nfilter) Fix masks (flow-mask) Summarize by destination IP & mask (flow-report) Format tabular data (flow-rptfmt)

  15. Traffic Engineering Flow-nfilter configuration: # ifIndex 46 is the POS interface to the Internet filter-primitive CLMBO-R4-INTERNET type ifindex permit 46 # Match on traffic to the Internet by POS interface. filter-definition CLMBO-R4-INTERNET-OUT match output-interface CLMBO-R4-INTERNET # Match on traffic from the Internet by POS interface. filter-definition CLMBO-R4-INTERNET-IN match input-interface CLMBO-R4-INTERNET

  16. Traffic Engineering Flow-mask configuration: mask-definition AS600-ADV-ALL prefix 128.146.0.0/16 16 prefix 140.254.0.0/16 16 prefix 164.107.0.0/16 16 prefix 192.12.205.0/24 24 prefix 192.68.143.0/24 24 prefix 128.156.0.0/16 16 prefix 139.88.0.0/16 16 prefix 192.55.90.0/23 23 prefix 129.1.0.0/16 16 prefix 129.137.0.0/16 16 . . .

  17. Traffic Engineering Flow-report configuration: stat-report CLMBO-R4-R8-FROM-INTERNET-BY-DESTINATION-PREFIX type ip-destination-address filter CLMBO-R4-INTERNET-IN ip-destination-address-format prefix-mask scale 100 output options +header,+xheader sort +octets fields -duration,+bps,+pps path @P/CLMBO-R4-FROM-INTERNET-BY-DESTINATION-PREFIX.txt output sort +octets fields -duration path |flow-rptfmt -Fip-destination-address,flows,octets,packets > \ @P/CLMBO-R4-FROM-INTERNET-BY-DESTINATION-PREFIX.fmt

  18. Traffic Engineering Run the report: #!/bin/sh FLOW_DATA=../CLMBO-R4/2004-11-08/* flow-cat $FLOW_DATA | flow-report -s report.cfg -Sdaily-summaries \ -vP=daily-summaries/

  19. Traffic Engineering Flow-report output: # --- ---- ---- Report Information --- --- --- # build-version: flow-tools 0.68p # name: CLMBO-R4-FROM-INTERNET-BY-DESTINATION-PREFIX # type: ip-destination-address # scale: 100 # options: +header,+xheader # ip-dst-addr-type: prefix-mask # sort_field: +octets # fields: +key,+flows,+octets,+packets,+other # filter: CLMBO-R4-INTERNET-IN # records: 139 # first-flow: 1099890083 Mon Nov 8 00:01:23 2004 # last-flow: 1099976485 Tue Nov 9 00:01:25 2004 # now: 1100221034 Thu Nov 11 19:57:14 2004 # # mode: streaming # compress: off # byte order: little # stream version: 3 # export version: 5 #

  20. Traffic Engineering Flow-report output: XXX # recn: ip-destination-address*,flows,octets,packets 164.107/16,1555282,227567453500,369333900 64.247.64/18,2276007,145782119200,379959000 129.137/16,678257,132932164100,179344100 134.53/16,786497,110197887600,175669700 198.30/16,523565,91210633100,126912900 129.1/16,651195,77953466900,104152800 130.101/16,389654,60952342600,87133000 132.235/16,379746,55747741000,102050900 140.254/16,328851,54698141100,65434300 131.183/16,416366,53890273300,76920100 128.146/16,511276,52719529000,92782100 199.218/16,746850,52400585900,165697600 130.108/16,308359,49463259900,66772600 205.133/16,441714,44835214400,71746400 206.244/16,268486,37445229500,50199700 206.21/16,380376,35449280700,59676600 …

  21. Traffic Engineering Flow-report-fmt output: # ['./flow-rptfmt', '-Fip-destination-address,flows,octets,packets'] ip-destination-address flows octets packets 164.107/16 1555282 227567453500 369333900 64.247.64/18 2276007 145782119200 379959000 129.137/16 678257 132932164100 179344100 134.53/16 786497 110197887600 175669700 198.30/16 523565 91210633100 126912900 129.1/16 651195 77953466900 104152800 130.101/16 389654 60952342600 87133000 132.235/16 379746 55747741000 102050900 140.254/16 328851 54698141100 65434300 131.183/16 416366 53890273300 76920100 128.146/16 511276 52719529000 92782100 199.218/16 746850 52400585900 165697600 130.108/16 308359 49463259900 66772600 ……….

  22. Traffic Engineering Flow-report-fmt output (top 10 percent totals): # ['./flow-rptfmt', '-m10', '-p', '-Fip-destination-address,flows,octets,packets'] ip-destination-address flows octets packets 164.107/16 10.906643 12.768776 12.746539 64.247.64/18 15.960832 8.179813 13.113235 129.137/16 4.756376 7.458804 6.189566 134.53/16 5.515425 6.183187 6.062754 198.30/16 3.671576 5.117815 4.380048 129.1/16 4.566600 4.373958 3.594546 130.101/16 2.732506 3.420027 3.007155 132.235/16 2.663024 3.127998 3.522005 140.254/16 2.306116 3.069105 2.258284 131.183/16 2.919828 3.023775 2.654685

  23. Traffic Engineering • A day long summary usually works for situations like this, but we may want to look at a time-series report over a day period (or longer). • Gnuplot, or rrdtool, or gdchart (among others) can be used for this. • Flow-rpt2rrd will convert the flow-report output to rrd format. For automated reports rrdtool works well since it bounds the storage requirements.

  24. Traffic Engineering #!/bin/sh FLOW_DATA=../CLMBO-R4/2004-11-08/* for name in $FLOW_DATA; do echo working...$name flow-report -s report.cfg -S5min-summaries \ -vP=5min-summaries < $name done for name in 5min-summaries/*; do echo working...$name ./flow-rpt2rrd -K as600.keys -p rrds < $name done

  25. Traffic Engineering During peak hours moving traffic for 164.107/16 elsewhere will drop inbound load by between 20 and 30Mb/s.

  26. Traffic Engineering • Minor changes to the report to get other results • Outbound traffic by source prefix (busy sources in OARnet). • Outbound traffic by destination prefix or AS (where are our sinks). • Inbound traffic by source prefix (where do our sinks draw traffic from). • Source / Destination pairs.

  27. Traffic Engineering stat-report CLMBO-R4-TO-INTERNET-BY-SOURCE-PREFIX type ip-source-address filter CLMBO-R4-INTERNET-OUT ip-source-address-format prefix-mask scale 100 output options +header,+xheader sort +octets fields -duration path @P/CLMBO-R4-TO-INTERNET-BY-SOURCE-PREFIX.txt

  28. Traffic Engineering stat-report CLMBO-R4-TO-INTERNET-BY-DESTINATION-AS type destination-as filter CLMBO-R4-INTERNET-OUT scale 100 output options +header,+xheader sort +octets fields -duration path @P/CLMBO-R4-TO-INTERNET-BY-DESTINATION-AS.txt

  29. Traffic Engineering # ['./flow-rptfmt', '-p', '-n', '-Fdestination-as,flows,octets,packets'] destination-as flows octets packets SCRR-10796 3.769746 7.881135 4.664673 ATT-INTERNET3 4.636261 5.987038 4.844542 DNEO-OSP1 3.066557 3.309393 3.005959 CCINET-2 2.258435 2.896718 2.529854 ROGERS-AS 1.537620 2.664075 1.913102 CHARTER-NET-HKY-NC 2.369852 2.100978 2.175202 RR-CINCINNATI-ASN-01 0.732147 1.989616 1.190482 SCRR-11427 1.588737 1.646306 1.647938 VIDEOTRON-LTEE 0.735799 1.323654 1.078804 SCRR-11426 1.183881 1.289154 1.234044

  30. Traffic Engineering During peak hours moving traffic destined to AS 10796 elsewhere will drop outbound load by between 20 and 30Mb/s.

  31. RPF checks • Provider Q wants to turn on strict RPF checks on our peerings. • For a variety of reasons this won’t work for us (load distribution for one), but we do agree to implementing a traffic filter. • There are places OARnet has not enforced RPF checks, mostly for historical reasons.

  32. RPF checks • So want to understand the impact of applying the traffic filter before applying it to prevent dropping valid customer traffic. • Expect problems mostly to be multi-homed clients who may not be advertising all their address space to us. • …Plus unexpected surprises.

  33. RPF checks Collected flows(flow-capture). Select a days data (flow-cat) Filter on provider interface (flow-nfilter) Filter on announced address space Summarize results with flow-report Format tabular data (flow-rptfmt)

  34. RPF checks Flow-nfilter configuration: #; OSU filter-primitive AS600-ADV-ALL type ip-address-prefix default permit deny 128.146.0.0/16 deny 140.254.0.0/16 deny 164.107.0.0/16 deny 192.12.205.0/24 deny 192.68.143.0/24 deny 128.156.0.0/16 deny 139.88.0.0/16 deny 192.55.90.0/23 deny 129.1.0.0/16 deny 129.137.0.0/16 ...

  35. RPF checks Flow-report configuration: stat-report CLMBO-R4-RPF-FAIL type ip-source-address filter CLMBO-R4-RPF-FAIL ip-source-address-format prefix-mask scale 100 output options +header,+xheader sort +octets fields -duration path |./flow-rptfmt stat-definition CLMBO-R4-RPF-FAIL report CLMBO-R4-RPF-FAIL

  36. RPF checks Results % FLOW_DATA=../clmbo-r4/2004-11-08/* % % flow-cat $FLOW_DATA | flow-report -s report.cfg -SCLMBO-R4-RPF-FAIL ip-source-address flows octets packets 198.234/16 24689 3478979000 4894300 66.144/15 22661 1422430800 3345200 204.130.176/24 4328 398581200 737000 156.63/16 2387 279678500 387900 208.44/14 992 6527900 105700 65.112/12 845 4986800 88900 66.194.129.106/32 381 2150400 38400 53/8 91 1886400 39300 18/8 90 1737600 36200 45/8 89 1660800 34600 38/8 91 1622400 33800

  37. Access Lists • Same procedure can be used to test a firewall or access list before applying it to live traffic. • Determine what the impact of a change will be before impacting customers.

  38. Traffic Classification • Abilene / Internet2 is a high speed research network that OARnet provides connectivity to for Ohio school’s. • Only some school’s participate. • Of those school’s that participate they can sponsor smaller school’s or other institutions for example a medical college.

  39. Traffic Classification • The participants all share the cost for an OC48 to Abilene. They want to know who’s using it both by school and group. • School’s may have multiple networks. • Not all school’s have AS numbers. • School traffic is combination of prefixes. • Group traffic is combination of schools.

  40. Traffic Classification • Flow-tag adds a new 32 bit source tag and destination tag to the flows. • Tags can be set based on criteria in the flow, such as source or destination prefix. • Reports can then be run on the tagged traffic.

  41. Traffic Classification Collected flows(flow-capture) Select a months data (flow-cat) Add tags based on interface and prefix (flow-tag) Filter on provider interface (flow-nfilter) Summarize results with flow-report Graph tabular data (gnuplot)

  42. Traffic Classification # # tag format # # 0 7 15 23 31 # 0000 0000 0000 0000 0000 0000 0000 0000 (32 bits) # RRRRRRRRRRRRRR TTTT NNNNNNNNNNNNNNNNNNN # | | | Site name # | | Site type # | Reserved # ID Name #--------------------------------- # 0x0001 OSU # 0x0002 CWRU # 0x0003 BGSU # 0x0004 UC # 0x0005 UAKRON # 0x0006 WRIGHT # 0x0007 KENT … # ID Type #------------------------ # 0x01 Participant # 0x02 SEGP # 0x03 Sponsored-Participant # 0x04 Gigapop # 0x05 MULTICAST

  43. Traffic Classification tag-action OHIO-GIGAPOP_DST type dst-prefix # OSU match 128.146/16 set-dst 0x010001 match 164.107/16 set-dst 0x010001 match 140.254/16 set-dst 0x010001 match 192.153.26/24 set-dst 0x010001 match 199.18.22.24/30 set-dst 0x010001 match 199.18.4.24/30 set-dst 0x010001 # CHMCC match 204.152.48/24 set-dst 0x030014 match 204.152.49/24 set-dst 0x030014 match 205.142.196/24 set-dst 0x030014 match 205.142.197/24 set-dst 0x030014 match 205.142.198/24 set-dst 0x030014 match 205.142.199/24 set-dst 0x030014 match 199.18.107.220/30 set-dst 0x030014

  44. Traffic Classification stat-report TS-CLMBQ-R2-FROM-ABILENE-BY-TAG-GROUP type destination-tag tag-mask 0x00FF0000 0x00FF0000 filter CLMBQ-R2-ABILENE-IN scale 100 output options +header,+xheader sort +octets fields -duration path @P/CLMBQ-R2-FROM-ABILENE-BY-TAG-GROUP.%j.txt stat-definition clmbq-r2-daily-summaries tag OHIO-GIGAPOP report TS-CLMBQ-R2-FROM-ABILENE-BY-TAG-GROUP report TS-CLMBQ-R2-FROM-ABILENE-BY-TAG-CUSTOMER report TS-CLMBQ-R2-TO-ABILENE-BY-TAG-GROUP report TS-CLMBQ-R2-TO-ABILENE-BY-TAG-CUSTOMER

  45. Traffic Classification

  46. Traffic Classification

  47. Traffic Classification • Groups of IP addresses - campus department traffic summaries. • Individual IP addresses - Find busy hosts on a campus. • Ports and protocols - find all the SMTP clients or hosts using file sharing services.

  48. Traffic Classification • We have the ability to provision dedicated l’s between many of the larger school’s. • Cost money…Some clients want to know if they should be purchasing dedicated bandwidth between each other.

  49. Traffic Classification

  50. Traffic Classification

More Related