1 / 94

SiLK Installation

SiLK Installation. A Walk-through. Joe McManus joe@cert.org. Topics. Topics Covered in this Training What SiLK is and is not SiLK on a Box SiLK with remote flow collection Building SiLK RPMs Monitoring SiLK Processes Basic SiLK Queries Useful queries for an administrator.

ada
Download Presentation

SiLK Installation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SiLK Installation A Walk-through Joe McManus joe@cert.org

  2. Topics • Topics Covered in this Training • What SiLK is and is not • SiLK on a Box • SiLK with remote flow collection • Building SiLK RPMs • Monitoring SiLK Processes • Basic SiLK Queries • Useful queries for an administrator

  3. Introduction to SiLK • System for internet Level Knowledge • Data collection • Data analysis • NOT: • Intrusion detection/prevention system • Audit tools for your network • Automated report generator

  4. Collection & Analysis Infrastructure • SiLK stores the data and allows access to the data for analysis. • YAF turns packets into flows for SiLK over IPFIX. • SiLK can take input from IPFIX or Netflow. • Netflow is convenient as it implemented on many routers and requires no additional hardware.

  5. Collection & Analysis Infrastructure

  6. SiLK on a BoxStandalone flow collection

  7. Standalone Collection and Analysis • We will now go over installing SiLK and YAF to create a stand alone flow collection and analysis box. • We will assume you are on a linux server with root access.

  8. Downloading SiLK • SiLK can be downloaded from: • http://tools.netsa.cert.org/silk/ • YAF is available from: • http://tools.netsa.cert.org/yaf/ • Fixbuf is available from: • http://tools.netsa.cert.org/fixbuf/

  9. Downloading (continued…) • Log in to your Linux server and use wget to download the software. • [root@silktmp]#cd /tmp • [root@silktmp]# wget \ http://tools.netsa.cert.org/releases/silk-1.1.3.tar.gz • [root@silktmp]# wget \ http://tools.netsa.cert.org/releases/libfixbuf-0.8.0.tar.gz • [root@silktmp]# wget \ http://tools.netsa.cert.org/releases/yaf-1.0.0.tar.gz

  10. Build fixbuf • Fixbuf is required to have YAF talk to SiLK over IPFIX. • [root@silktmp]# tar -zxvf libfixbuf-0.8.0.tar.gz • [root@silktmp]# cd libfixbuf-0.8.0 • [root@silk libfixbuf-0.8.0]# ./configure && make && make install

  11. Errors • checking for cc... no • checking for cl... no • configure: error: no acceptable C compiler found in $PATH • We need to make sure the system has the required packages to build the suite.

  12. YUM • We can use YUM to install the missing C compiler. • [root@silk libfixbuf-0.8.0]# yum install gcc • Try building Fixbuf again. • Which brings us to the next error: • checking whether we are using the GNU C++ compiler... no • checking whether g++ accepts -g... no • checking dependency style of g++... none

  13. YUM • If you do not know which package you are looking for YUM can help with this. • [root@silk libfixbuf-0.8.0]# yum search c++ • [root@silk libfixbuf-0.8.0]# yum install gcc-c++

  14. RPM Confusion • [root@silk libfixbuf-0.8.0]# ./configure • checking for GLIB - version >= 2.4.7... no • *** Could not run GLIB test program, checking why... • *** The test program failed to compile or link. See the file config.log for the • *** exact error that occured. This usually means GLIB is incorrectly installed. • configure: error: Cannot find a suitable glib2 (>= 2.4.7) • This error is misleading.

  15. RPM Confusion • [root@silk libfixbuf-0.8.0]# yum install glib2 • Loaded plugins: refresh-packagekit • Setting up Install Process • Parsing package install arguments • Package glib2-2.16.6-2.fc9.i386 already installed and latest version • Nothing to do • [root@silk ~]# rpm -qa | grep glib2 • glib2-2.16.6-2.fc9.i386 • Wuh?!?

  16. RPM Confusion • What is actually missing is the glib2-devel rpm. • [root@silk ~]# yum install glib2-devel • You may have to install the libpcap-devel rpm as well. This is dependent on which install of linux you have done.

  17. Fixbuf install • [root@silk libfixbuf-0.8.0]# ./configure • [root@silk libfixbuf-0.8.0]# make • [root@silk libfixbuf-0.8.0]# make install • [root@silk libfixbuf-0.8.0]# ls /usr/local/lib • libfixbuf-0.8.0.so.8 • libfixbuf.a • libfixbuf.so • libfixbuf-0.8.0.so.8.0.0 • libfixbuf.la • pkgconfig

  18. YAF Install • YAF is our tool for flow collection, there are others. • [root@silk libfixbuf-0.8.0]# cd .. • [root@silktmp]# tar -zxvf yaf-1.0.0.tar.gz • [root@silktmp]# cd yaf-1.0.0 • checking for libfixbuf >= 0.7.2... configure: error: Cannot find a suitable libfixbuf (>= 0.7.2): Package libfixbuf was not found in the pkg-config search path. • Perhaps you should add the directory containing `libfixbuf.pc' • to the PKG_CONFIG_PATH environment variable • No package 'libfixbuf' found

  19. YAF Install • Didn’t we just install fixbuf? • -Yes, but pkgconfig does not know to look in /usr/local. • [root@silk yaf-1.0.0]# export \ PKG_CONFIG_PATH=/usr/local/lib/pkgconfig • [root@silk yaf-1.0.0]# ./configure • [root@silk yaf-1.0.0]# make • [root@silk yaf-1.0.0]# make install

  20. SiLK Install • Uncompress the SiLKtarball. • [root@silk yaf-1.0.0]# cd .. • [root@silktmp]# tar -zxvf silk-1.1.3.tar.gz • [root@silktmp]# cd silk-1.1.3 • We will build in support for PySiLK. • [root@silk silk-1.1.3]# yum install python-devel

  21. SiLK Install (cont…) • [root@silk silk-1.1.3]# ./configure \ • --with-libfixbuf=/usr/local/lib/pkgconfig/ \ • --with-python • SiLK creates a great summary.

  22. Summary • * Configured package: SiLK 1.1.3 • * Host type: i686-pc-linux-gnu • * Source files ($top_srcdir): . • * Install directory: /usr/local • * Root of packed data tree: /data • * Packing logic: via run-time plugin • * Timezone support: UTC • * Default compression method: SK_COMPMETHOD_NONE • * IPv6 support: NO • * IPFIX collection support: YES (-pthread -L/usr/local/lib -lfixbuf -lgthread-2.0-lrt -lglib-2.0) • * Transport encryption support: NO (gnutls not found) • * IPA support: NO • * LIBPCAP support: YES (-lpcap) • * Python support: YES (-L/usr/kerberos/lib -Xlinker -export-dynamic -ld -lutil -lm -L/usr/lib -lpython2.5 -pthread) • *Python package destination: /usr/lib/python2.5/site-packages • * Build analysis tools: YES • * Build packing tools: YES • * Compiler (CC): gcc • * Compiler flags (CFLAGS): -I$(top_srcdir)/src/include -DNDEBUG -D_GNU_SOURCE=1 • -D_FILE_OFFSET_BITS=64 -O3 -fno-strict-aliasing -Wall -W -Wmissing-prototypes -Wformat=2 • -Wdeclaration-after-statement • * Linker flags (LDFLAGS): • * Libraries (LIBS): -ldl -lm

  23. SiLK Install (cont…) • [root@silk silk-1.1.3]# make • [root@silk silk-1.1.3]# make install • [root@silk silk-1.1.3]# rwfilter --version • rwfilter: part of SiLK 1.1.3; configuration settings: • * Root of packed data tree: /data • * Packing logic: Run-time plug-in • * Timezone support: UTC • * Available compression methods: none [default] • * IPv6 support: no • * IPFIX collection support: yes • * Transport encryption: no • * PySiLK support: /usr/lib/python2.5/site-packages • * Enable assert(): no

  24. SiLK Configuration • We need to create a place to store the flow data. • The default for is /data. • [root@silk ~]# mkdir /data • We will need to create two configuration files for SiLK, sensor.conf and silk.conf

  25. silk.conf • A few sample silk.conf files are included with the distribution. They are installed under /usr/local/share/silk. • We would want to use twoway-silk.conf • [root@silk /data]# cp \ /usr/local/share/silk/twoway -silk.conf .

  26. silk.conf • [root@silk data]# vi /data/silk.conf • #Define Sensors • sensor 0 localhost • class all    sensors localhostend class • version 1

  27. SiLK Configuration • class all    type  0 in      in    type  1 out     out    type  2 inwebiw    type  3 outwebow    type  4 innullinnull    type  5 outnulloutnull    type  6 int2int int2int    type  7 ext2ext ext2ext    type  8 inicmpinicmp    type  9 outicmpouticmp    type 10 other   other •     default-types in inwebinicmpend class

  28. silk.conf storage heirarchy

  29. silk.conf storage hierarchy. • I find it good practice to have silk store its data under /data/SENSOR-NAME/type. • The benefit of storing data this way is that you can use unix groups to control access to flow data on each sensor. • i.e. • /data/engineering • /data/sales • /data/logistics

  30. silk.conf • # The default path format from SILK_DATA_ROOTDIRpath-format "%N/%T/%Y/%m/%d/%x“ • %N= Sensor Name • %T=Type In/Out/int2int… • %Y=Year • %m=month • %d=data • %x=flowtype-sensor_YearMonthDay.Hour

  31. silk.conf • # The plug-in to load to get the packing logic to use in rwflowpack.# The --packing-logic switch to rwflowpack will override this value.# If SiLK was configured with hard-coded packing logic, this value is# ignored. • # The plug-in to load to get the packing logic to use in rwflowpack.# The --packing-logic switch to rwflowpack will override this value.# If SiLK was configured with hard-coded packing logic, this value is# ignored.packing-logic "packlogic-twoway.so“

  32. sensor.conf • We will now have to edit the sensor.conf file. • [root@silk data]# vi /data/sensor.conf • probe localhostipfix    listen-on-port 18001    protocol tcp    accept-from-host 127.0.0.1end probe

  33. sensor.conf • sensor localhostipfix-probes localhost    internal-ipblock 192.168.1.0/24    external-ipblock remainderend sensor

  34. rwflowpack • rwflowpack is the daemon that will listen for traffic from YAF. • A sample file is included, which we will customize. • [root@silk]# cp \ /usr/local/share/silk/etc/rwflowpack.conf \ /usr/local/etc/. • [root@silk data]# vi /usr/local/etc/rwflowpack.conf

  35. rwflowpack.conf • Change the following values: • ENABLED=yes • SENSOR_CONFIG=/data/sensor.conf • SITE_CONFIG=/data/silk.conf • LOG_TYPE=legacy • LOG_DIR=/var/log • CREATE_DIRECTORIES=yes • COMPRESSION_TYPE=best

  36. rwflowpack.conf • To make rwflowpack start on boot we can add it using chkconfig. SiLK includes sample startup scripts with the distribution. • [root@silk data]# cp /usr/local/share/silk/etc/init.d/rwflowpack /etc/init.d • [root@silk data]# chkconfigrwflowpack on • [root@silk data]# chkconfig --list rwflowpack • rwflowpack 0:off 1:off 2:on 3:on 4:on 5:on 6:off

  37. Test YAF • Test that YAF is able to listen on the interface. • [root@silk data]# yaf --live=pcap --in=eth0 --out=- -v | yafscii • [2009-01-08 17:06:10] yaf starting • [2009-01-08 17:06:10] running as root in --live mode, but not dropping privilege

  38. Test YAF • Let that run for a minute. Then enter CTRL-C. • C2009-01-08 17:07:19.900 tcp 10.0.0.2:1005 => 10.0.0.3:2049 90b24967:0b0cbdd9 AF/A:AF/0 (2/104 <-> 1/52) rtt 0 ms • [2009-01-08 17:07:42] Processed 57 packets into 9 flows: • [2009-01-08 17:07:42] Mean flow rate 0.18/s. • [2009-01-08 17:07:42] Mean packet rate 1.11/s. • [2009-01-08 17:07:42] Virtual bandwidth 0.0008 Mbps. • [2009-01-08 17:07:42] Maximum flow table size 8. • [2009-01-08 17:07:42] 6 flush events. • [2009-01-08 17:07:42] Rejected 1 out-of-sequence packets. • [2009-01-08 17:07:42] Assembled 0 fragments into 0 packets: • [2009-01-08 17:07:42] Expired 0 incomplete fragmented packets. • [2009-01-08 17:07:42] Maximum fragment table size 0. • [2009-01-08 17:07:42] Rejected 44 packets during decode: • [2009-01-08 17:07:42] 44 due to unsupported/rejected packet type: • [2009-01-08 17:07:42] 44 unsupported/rejected Layer 3 headers. • [2009-01-08 17:07:42] yaf terminating

  39. IPTables • You must open the firewall for YAF to connect to rwflowpack. • [root@silk data]# iptables -I INPUT \ • -s 127.0.0.1 -p tcp -m tcp \ • --dport 18001 -j ACCEPT • [root@silk data]# service iptables save • iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]

  40. IPTables • We can check to see if the rule has been added correctly. • [root@silk data]# service iptables status • Table: filter • Chain INPUT (policy ACCEPT) • num target prot opt source destination • 1 ACCEPT tcp -- 127.0.0.1 0.0.0.0/0 tcp dpt:18001 • ….

  41. Starting YAF • [root@silk data]# yaf --silk --ipfix=tcp --live=pcap --in=eth0 --out=127.0.0.1 --ipfix-port=18001 &

  42. Starting YAF • [root@silk data]# yaf--silk --ipfix=tcp --live=pcap --in=eth0 --out=127.0.0.1 --ipfix-port=18001 & • The silk option tells YAF to format the output for silk.

  43. Starting YAF • [root@silk data]# yaf --silk --ipfix=tcp--live=pcap --in=eth0 --out=127.0.0.1 --ipfix-port=18001 & • The ipfix=tcp option tells YAF use ipfix over tcp. • Theipfix-port=18001 option specifies which port.

  44. Starting YAF • [root@silk data]# yaf --silk --ipfix=tcp--live=pcap--in=eth0 --out=127.0.0.1 --ipfix-port=18001 & • The live=pcap option tells YAF the capture device uses pcap. • The only other option is dag for use with Endace Dag cards.

  45. Starting YAF • [root@silk data]# yaf --silk --ipfix=tcp--live=pcap --in=eth0 --out=127.0.0.1 --ipfix-port=18001 & • The in=eth0 option tells YAF which input device to use. • If you are using a dag card, the option would be dag0.

  46. Starting YAF • [root@silk data]# yaf --silk --ipfix=tcp--live=pcap--in=eth0 --out=127.0.0.1 --ipfix-port=18001 & • The out=127.0.0.1 option tells YAF where to send its output.

  47. Checking on YAF • You can check the status of YAF with the following command: • [root@silk data]# kill -SIGUSR1 `pgrepyaf` • [2009-01-08 21:53:29] Rejected 1 out-of-sequence packets.

  48. Checking on The Collection Process • When you first start yaf one of the first places to look is in the log file under /var/log. • [root@silk data]# tail /var/log/rwflowpack-20090108.log • Jan 8 16:53:40 silk rwflowpack[10605]: Flushing files after 120 seconds. • Jan 8 16:54:20 silk rwflowpack[10605]: Opening new output file /data/localhost/ext2ext/2009/01/08/ext2ext-localhost_20090108.21

  49. Checking on The Collection Process • Did yaf start correctly? • [root@silk data]# ps -ef | grepyaf • root 10630 10475 0 16:52 pts/1 00:00:00 yaf --silk --ipfix=tcp --live=pcap --in=eth0 --out=127.0.0.1 --ipfix-port=18001

  50. Checking on The Collection Process • If not can yaf connect to rwflowpack? • [root@silk data]# telnet localhost 18001 • Trying 127.0.0.1... • Connected to localhost. • Escape character is '^]'. • Is rwflowpack running? • [root@silk data]# ps -ef | grepyaf

More Related