1 / 20

To Err Is Human…

To Err Is Human…. Chuck Thompson Manager, CS Technology Services Group. … But To Really Err Takes A Computer (And A Sysadmin). Outline. General Mistakes Environment Matters Real World Problems And Their Causes Detecting Security Incidents And Problems Final Thoughts. General Mistakes.

sana
Download Presentation

To Err Is Human…

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. To Err Is Human… Chuck Thompson Manager, CS Technology Services Group

  2. … But To Really Err Takes A Computer (And A Sysadmin)

  3. Outline • General Mistakes • Environment Matters • Real World Problems And Their Causes • Detecting Security Incidents And Problems • Final Thoughts

  4. General Mistakes • Typos • Cut and Paste Line Wraps (or lack of) • Wrong Window • Right Action, Wrong Location • Incorrect Permissions • Incomplete Changes • Inadequate Preparation

  5. Environment Matters • Epoch Redisplay Port to Lucid Emacs • Software Distribution X11 Build Circa 1991 • TeX/LaTeX PATH and TEXINPUTS • Binary Path Differences Between OSes • Crontab vs. Login Shell Environment It is very difficult to properly recreate production environments in test environments.

  6. Real World Problems And Their Causes • All situations and incidents to be described really happened • They all happened in or to this department • Names may or may not be changed to protect the guilty

  7. Networking Related Problems • UIUCnet 12 hour outage on August 27 • Filtering of dcs-server1 on September 7 • ServerIron MAC address corruption • Router ACLs bug in early 2004 • Switch uplink module problems in 2002 • Cut-and-Paste ACL update failure • DHCP server not running… due to a missing semicolon

  8. OS Bug / Patching Related Problems • Primary CS server crashing due to DHCP-tickled bug • Y2K induced problems (3 bad Solaris patches) • BIND security patch (or lack of) from Sun • Broken binaries after patching, before reboot • DCL elock servers patching incident

  9. Services Startup Problems • missing init scripts / links / chkconfig settings • httpd.pid file not removed before reboot • SSL-capable httpd startup with password protected SSL certificates • dcs-mail.cs.uiuc.edu /etc/init.d/sendmail vs. /etc/init.d/sendmail-dcs

  10. Disk Space Related Problems • FA05 class websites server /var filled up in less than a week: 3GB of debugging info • corrupted alias distribution problems • Bluestem login failures • Runaway programs doing excessive logging • favorite of CS grad students  • /home/student out-of-space in April 2005

  11. Mail Related Problems • @cs.uiuc.edu email handler queue backup in March 2005 • mail.cs.uiuc.edu performance problems • general overload • inefficient SPAM filtering • poor swap setup • possible OS bug • SpamAssassin upgrade incident in March 2005 • Mailman broken by Python upgrade

  12. Miscellanous Problems • “ssh.com vs. OpenSSH scp compatibility” or “When error messages lie” • “rm –rf a *” or “The Ultimate Typo” • “The Case of the Disappearing Software Distribution” or “Why it is important to use locks to prevent multiple running instances of some programs” • “3 days to create 2000 accounts” or “Why CS473 is important” • “Underlying mount point permissions matter” or “What you can’t see might frustrate you”

  13. Detecting Security Incidents And Problems • Port scans • Port monitoring • Network traffic logs • External reports • Intrusion Detection/Prevention Systems • Local system scans

  14. Local System Scans • Modified system binaries • Common ones include /bin/login, ps, ls, find • programs that capture login data, provide backdoors • programs that can be used to detect the intrusion • Modified init files • Used to startup backdoors after reboots • System log checks for known signs • Known compromised / attacking hosts • Obvious signs of missing data (e.g. missing timeframes)

  15. Local System Scans (cont.) • Check for known problem files and directories. /usr/spool/uucppublic/.hushlogin /usr/spool/secretmail/.l /tmp/.a /tmp/... /.zap /usr/sbin/at /usr/spool/secretmail/.log /usr/spool/secretmail/.tty /usr/spool/secretmail/.lock /usr/tmp/.log /usr/spool/uucp/.sys /usr/uucp/.sys /var/crash/... /usr/etc/.getwd /var/crash/.getwd /usr/kvm/... /dev/.tty /dev/.test /dev/.error /dev/.errors /tmp/.w0rm /tmp/.worm /tmp/.X11x /tmp/.w0rm0r /tmp/w0rmishere /tmp/ADMw0rm.tgz /dev/reset /dev/pmcf1 /dev/pmcf2 /dev/pmcf3 /dev/pmcf4 /tmp/fix /dev/fix /usr/lib/libsn.a /.ncftp /dev/izzo /var/tmp/.preserve /var/tmp/a /dev/sad/tmp /dev/sad/tmp/core /usr/sbin/keybd /tmp/bob /usr/src/sh /usr/share/src/sh /usr/sbin/nfds /usr/sbin/nfds.config /etc/core /etc/m /usr/lib/libxxx/.../ttymon /var/named/ADMROCKS /var/adm/Q /var/named/Q /var/spool/lp/buffer /usr/sbin/in.bind /usr/bin/xcat /usr/bin/nfsiod /usr/sbin/find /etc/l1lo /bin/fwl /lib/... /lib/go /tmp/.bash_history /usr/src/.poop /tmp/ramen.tgz /tmp/ramen.tar /bin/in.telnetd /bin/mjy /usr/man/man1/man1/lib/.lib/mjy /usr/man/man1/man1/lib/.lib/in.telnetd /usr/man/man1/man1/lib/.lib/.x /dev/.lib /usr/info/.torn /usr/src/.puta /var/lp/lpacct/lpacct /usr/lib/lpset /dev/pts/01 /usr/lib/lib /usr/bin/adore /usr/lib/klogd.o /tmp/.tmp /tmp/.problem /usr/lib/red.tar /usr/lib/.lnx /tmp/.z /tmp/.z/asu /z /sbin/... /tmp/.bugtraq.c /usr/lib/libX.a/bin /usr/lib/libX11.a/bin /usr/bin/sshd_config /usr/bin/ssh_host_key /usr/bin/ssh_host_key.pub /usr/bin/srload /usr/bin/ssh_random_seed /usr/bin/sshd.pid /dev/tux /usr/sbin/modstat /usr/sbin/modcheck /etc/security/audit_device /usr/lib/libp/libm.n /dev/prom /dev/ttyob /usr/lib/lpstart /usr/lib/ldlibnet.so /sbin/xlogin /sbin/init.xrk /etc/k.xrk /etc/vfsd /etc/vfsd/.viminfo /etc/vfsd/sshd.pid /usr/bin/zap

  16. Encryption Isn’t A Guarantee • Incident last year involved ssh commands being replaced • Intruders also took advantage of SSH host-based authentication

  17. Information Overload • Too much logged data • Too many false positives • Too many vulnerabilities • Not enough time • When there is too much to do, security is easy to leave until there is an incident • We’ve entered the era of zero-day exploits • We’ve already been in the era of rapid exploits • Charley Kline Linux system install story • Problems patching systems against Blaster

  18. How Can Students Help With Security • Patch Your Systems • Patch Your Applications • Install Antivirus Software And Keep It Updated • It’s not just a good idea, it’s the University law • Patch Your Systems • Patch Your Applications • Install Antivirus Software And Keep It Updated

  19. Final Thoughts • Computing environments are run by buggy and incomplete scripts • Scale Matters • Problem symptoms rarely lie but often mislead • Systems administration is still as much art as science • Break-ins Happen

  20. Questions / Comments Contact Info: Chuck Thompson 2332 Siebel Center cthomp@cs.uiuc.edu 333-3632

More Related