Advanced Bio-Linux

Dan Swan: Log files and log monitoring Environmental Genomics Thematic Programme Data Centre http://envgen.nox.ac.uk Advanced Bio-Linux

Environmental Genomics Thematic Programme Data Centre http://envgen.nox.ac.uk What, where and why? • Log files are files of 'useful' information that are written in an ongoing fashion by various programs on the system. • Most (but not all) logs are stored in /var/log/ • Apache (webserver logs) are stored in /etc/httpd/logs/ • Logs are your first port of call for detecting things like upcoming hardware issues (such as disk drive failure), accounting visits (hits) on the webserver, detection of intrusion attempts, detection of intrusions. • Going over the log files is good practice, but for an inexperienced admin making sense of them is not always straightforward. • We will try to demistify some of this process and show you how to automate it.

Environmental Genomics Thematic Programme Data Centre http://envgen.nox.ac.uk Apache logs

Environmental Genomics Thematic Programme Data Centre http://envgen.nox.ac.uk Why check these logs? • If you have opened up your web server to the outside world it will be probed by hackers. Apache has had a number of vulnerabilities in the past but is generally considered to be good in terms of security. • Things to look out for are - huge entries trying to get filenames with thousands of characters - these are attempted “buffer overflow” exploits. Anything which contains more strings like ../../../../../../../../../ is attempting to locate files outside of an improperly secured Apache set up. Any requests for executable files, or cgi files that you did not put there are cause for concern. • You can safely ignore anything looking for .com .exe files, these are scans for Microsoft IIS vulnerabilities.

Environmental Genomics Thematic Programme Data Centre http://envgen.nox.ac.uk /var/log • This containst the majority of the log files. • Each one stores different information. • It is important to keep an eye on these. • Some can be viewed with a 'logchecker' to give the administrator essential information. • It is important that as an administrator you learn to understand the information given in the logs. • The only way is to look and learn. You will often not understand the entries, use Google to look them up if you are concerned. • We will go through them one by one briefly.

Environmental Genomics Thematic Programme Data Centre http://envgen.nox.ac.uk

Environmental Genomics Thematic Programme Data Centre http://envgen.nox.ac.uk dmesg • less /var/log/dmesg • keeps a log of printed kernel messages. • Most often this will have information about the boot process, and it should be checked to keep an eye on the actual hardware as it can harbour warnings of upcoming problems. • Whilst the system is running dmesg will print out any other kernel messages, and will include items like firewall incidents etc. • There is a great article demystifying the dmesg output (which can be quite cryptic) here: • http://www.linuxgazette.com/issue59/nazario.html

Log files not to stress about • /var/log/gdm : gdm is the graphical login manager. • /var/log/httpd : the same as /etc/httpd/logs/ • /var/log/iptraf : it's empty! • /var/log/kdm.log : another graphical login manager. • /var/log/ksyms.* : kernel symbols logs (incomprehensible). • /var/log/lastlog : Use the last command instead. • /var/log/mysqld.log : Of course if you start using MySQL you might look! • /var/log/pgsql : PostgreSQL logs - as above. • /var/log/rcd/* : RedCarpet logs - only look if you have issues. • /var/log/rpmpkgs : Same as running rpm -qa • /var/log/sa : output from sar command on cron job • /var/log/scrollkeeper.log : GNOME messages, might be good to look at if you have a GNOME problem! • /var/log/spooler : UUCP spools (no-one uses this anymore!) • /var/log/up2date : RedHat update mechanism (unused). • /var/log/wtmp : not human readable • /var/log/Xfree86.0.log : If you have X errors this is the place to look.

Environmental Genomics Thematic Programme Data Centre http://envgen.nox.ac.uk What is with the .1 files? • The first thing you notice about /var/log/ files is that you have • file • file.1 • file.2 etc. • This is because logs are rotated weekly to make sure they do not clog up the disk. • Rotation is handled by logrotate.sh in /etc/cron/cron.daily • Each week file is moved to file.1, file.1 to file.2 etc. and file.4 is deleted as we work a 4 week cycle of logs. • You can alter the behaviour of logrotate in /etc/logrotate.conf

Environmental Genomics Thematic Programme Data Centre http://envgen.nox.ac.uk Automated log checking • Currently Bio-Linux machines are set to email the EGTDC your logs. Now you are all advanced users :) you can take the responsibility for your own log checking. We wil advise you how to do this later. • Log checking is done by a program called logcheck.sh - appropriately! This is run as a daily cron job and the results emailed to the defined manager. Currently this is channelled to the EGTDC account and forwarded to us. In future this should be sent to the manager account. • This tries to pick out the most obvious security problems and report them, although it still reports a lot of false positives it is imperative you watch this file for curious actions on the system.

Advanced Bio-Linux