Practical Applications in Security Lessons learned from recent security incidents in the Computer Science Department

1. CSCI 6268 December 6, 2005 Practical Applications in SecurityLessons learned from recent security incidents in the Computer Science Department Chris Schenk Mark Dehus Michael Oberg

2. Outline • History and Background of CS Attacks (Chris) • Fall 2004 Compromise • Fall 2005 Compromise • Details on the Recent Compromise (Mark) • Detection • Recovery • Mitigation • Computational Science Center Perspective (Michael) • Systems Inspection • Security Tools Used By The CSC

3. You know it’s a bad day when…

4. It is possible… • Pre-attack on the CSEL, Fall 2004 • Request for user login name change • Password file distribution from central source • Rdist + hosts.allow • *poof* • Password file corruption • 600 accounts lost • Empty root password • Restore from single nightly backup copy (luckily!)

5. That someone may be… • Reload of machine ‘duos’ to support ldap (with help from Dirk Grunwald) • All machines on external network accessible via ssh • toucan, cardinal, woodpecker, etc • ssh woodpecker: Remote host key has changed! • Not initially thought of as a threat (woops) • Host ID in base64 form, not in hex (strange) • Two days later, logged into ‘duos’ • ‘who’ command (some.company.ro) • The Romanians are upon us

6. Doing something nasty!! • ‘freeshell.org’ email account shutdown • Contact ITS to stop mail forwarding • Attempt to email sysadmin at freeshell.org • His response: Chris - Thanks for writing. Your colorado.edu account may been compromised as well. Please change your password. There is currently an investigation being done on you by the FBI and you may be contacted by a special agent Matthew Rowald concerning some efnet IRC activity. I'm not even sure if you will get this, but I certainly wasn't going to email it to your SDF account. I only saw connections from comcast and colorado.edu, which leeds me to believe that the individual (or in the FBI's case, you) are using your colorado.edu acct. Can you please change your passwords? Also, you might want to look into your ssh versions used at colorado.edu.

7. Response • Almost all machines compromised (~90%) • Entire network bogged down with flooding traffic from compromised machines • ‘ls’ command taking 10 seconds to complete • Single 100Mbit copper line to connect all CSEL machines to the world • Very unhappy databases class (some were glad homework couldn’t get done, others were not) • Contacts from a few places around the world about attempted dictionary attacks from CSEL machines.

8. Remediation • Bad Mojo • No logs saved • Unsure of how many accounts were logged (anyone using ssh that week) • All account passwords expired to force password changes (didn’t completely work) • Good Mojo • Reload of machines with Fedora Core 2 • Replacing RedHat 9 (1 ½ years after support dropped for OS) • All machines moved to internal subnet • No machines externally accessible except server csel.cs • No further incidences seen after reload and NATing of machines

9. John the Ripper • Password cracking utility • Ran on password file after remediation • 22 account passwords discovered after 2 hours • 6 found immediately (one dead guy) • 2800 accounts leftover in the lab • Only 300 active users for the year

10. Fall 2005 Compromise • Dirk Grunwald’s machines compromised • Local user with full sudo access logged in from Amsterdam • Sshd immediately replaced on all machines • Took one full day to figure out something was wrong

11. CSEL Second Compromise Overview • CSEL Background • Attack Timeline • Detection • Lockups and Errors • Log Auditing • Recovery • Data migration • Server Recovery • Workstation Recovery • Mitigation • Locking Down OS • Locking Down Dameons • Automated Break-in detection

12. csel.cs.colorado.edu • Primary public server • Used for public ssh shell access • Hosts the CSEL website & student websites • Has e-mail capabilities

13. duos.cs.colorado.edu • Duos is the LDAP & File Server • This means it contains authentication data and home directories • It also manages 'Self-Account Creation' from PLUS • It has access to Uniquid (Unique User ID system)

14. lesc.cs.colorado.edu • Lesc is used as a backup mirror of duos • At the time Lesc was being used to phase out duos • Important information was on it.. such as • Recent LDAP Directory Dump (Contained password hashes) • Workstation Image (The one I used that same day) • Recent Backup of Home directories

15. Csel.cs.colorado.edu Locked up! .. We were oblivious ..

16. CSEL Attack Timeline (What we saw) • Friday, August 19th • csel.cs.colorado.edu locked up at 10:30am • Computer did not want to reboot (Hanging on Shorewall) • Shorewall was disabled and computer was booted with basic iptables • Shrugged off as bad luck • Sunday, August 21st • Chris is concerned and checks sshd on servers by executing.. $ ps aux | grep sshdroot 2092 0.0 0.0 4668 264 ? Ss Aug21 0:00 /usr/sbin/sshd • Everything looks OK! • Monday, August 22nd • Check of log files shows a compromised account logged in on Friday!! • Tuesday, August 23rd • We determined that all the attacker was able to do was crash csel.cs

17. Lesc.cs.colorado.edu

18. Finding out what happened (Log Auditing) • First sign.. csel.cs.colorado.edu was locked up for no reason • Question: Who & How - > /var/log/messages & /var/log/secure • Two ways to find answers • Find by Hand • Automated Log Analysis • This question was answered by a mix of both.. but mostly by hand (I love Grep!) Dec 5 15:43:30 csel OTRS-PM-10[12240]: [Notice][Kernel::System::PostMaster::FollowUp::Run] FollowUp Article to Ticket [2005120510000011] created (TicketID=128, ArticleID=410). Dec 5 15:50:03 csel crond(pam_unix)[5839]: session closed for user otrs Dec 5 15:50:05 csel crond(pam_unix)[5837]: session closed for user root Dec 5 15:50:41 csel unix_chkpwd[5906]: password check failed for user (buschk) Dec 5 15:50:41 csel sshd(pam_unix)[5904]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=camras.colorado.edu user=buschk Dec 5 15:50:41 csel sshd[5904]: pam_ldap: error trying to bind as user "uid=buschk,ou=People,dc=csel,dc=cs,dc=colorado,dc=edu" (Invalid credentials) Dec 5 15:50:50 csel unix_chkpwd[5909]: password check failed for user (trumpler) Dec 5 15:50:50 csel sshd(pam_unix)[5907]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=camras.colorado.edu user=trumpler Dec 5 15:50:50 csel sshd(pam_unix)[5910]: session opened for user trumpler by (uid=0) Dec 5 15:50:51 csel automount[5937]: mount(nfs): nfs: mount failure duos:/home/nsa on /home/nsa

19. CSEL Second Compromise (What the logs told us) • Friday, August 19th • Attempted logins using previously collected usernames started at 5AM • Csel.cs & Lesc.cs were attacked simultaneously • Attacker was io.physics.tamu.edu (obviously compromised) • Payback.. Buffs beat Texas A&M 40 – 21! That'll teach um! • 35 Usernames were attempted, 4 were successful • When io discovered successful login it alerted attacker • Attacker then used login information from 81.181.251.2 • 81.181.251.2 Belongs to RIPE.net (Large ISP) in Amsterdam

20. Lesc Damage • Attackers got into Lesc using root password • SSHD was replaced with chroot jailed trojan • Contents of /root • rhel4-workstation-image.tgz • ldap-data-backup.ldif • All attacker did was replace SSHD!

21. Duos.cs.colorado.edu • Hackers did not bother with Duos • We got lucky.. Would have been very easy to gain root access • PHP was setuid root due to Uniquid requirements • Who cares? Duos has access to Mailing Addresses! $ cat 0wnDuos.php #!/usr/local/sbin/php <?php exec(“passwd root”); ?> $ ./0wnDuos.php Changing password for user root New password: i0wnU $ su Password: i0wnu # rm / -rf ( or worse )

22. Recovery • Copied important data to another computer • Reloaded all three servers • Reloaded all workstations • Easy but time consuming (30 hours)

23. Mitigation (General) • Don't use weak passwords and change them often! • SSH Restrictions and Rate Limiting on incoming connections • Use nosuid, nodev, and noexec to restrict files • Monitor log files frequently with automatic analyzer • Logcheck • Logwatch • Considering using RSA/DSA keys instead of passwords

24. Mitigation (duos.cs.colorado.edu) • Self-Account Creation scripts still need setuid • So-So Solution.. C setuid wrapper designed to call uniquid scripts only • Better solution.. rewrite all scripts in perl • Prefect solution.. fix uniquid so it does not need setuid • Passwords are no longer allowed. • Fine grained sudo controls and su restrictions • SSH restrictions • Blocked Internet Traffic & Remote Logins • Installed Intrusion Detection Software

25. Mitigation (csel.cs.colorado.edu) • Passwords are allowed but restrictions are put on pam_cracklib • Still can’t restrict passwords from SAC.. Yet • Fine grained sudo controls and su restrictions • SSH Restrictions • Rate Limiting on incoming connections • Installed Intrusion Detection Software • Put restrictions on file system

26. Mitigation (lesc.cs.colorado.edu) • Removed from Internet completely • Applied same restrictions as Duos • Installed Intrusion Detection Software

27. Outline • Computational Science Center Perspective on the Compromise • Systems Inspection immediately following notification of the compromise • Security Tools Used By The CSC

28. Computational Science Center • 100 systems under Dr. Henry Tufo • http://csc.cs.colorado.edu/ • Three clusters, plus several additional machines: • 65 node Intel based Computational Cluster (hemisphere) • 28 node PowerPC based Computational Cluster (occam) • 3 node Storage Cluster • 1 x86_64 Big RAM (8GB) machine • Management Server • License Server • Log / Security Server • 8 externally accessible, remainder are on a private LAN

29. CSC Systems Inspection • This work done by Sean McCreary, Matthew Woitaszek and myself • Received notification of compromise of unknown number of systems in the CS department • Manually audited logs to determine if anyone successfully logged in to our servers, found this: Aug 19 06:04:26 hemisphere sshd[4212]: Accepted password for grunwald from 165.91.181.6 port 39256 ssh2 Aug 19 06:11:32 hemisphere sshd[6929]: Accepted password for grunwald from 81.181.251.2 port 2412 ssh2 Aug 19 06:36:27 hemisphere sshd[7861]: Accepted password for grunwald from 81.181.251.2 port 2441 ssh2

30. CSC Systems Inspection • Hacker Successfully logged into our system, now what? • Treat all machines as if they are hacked, full inspection • If any machines are found to be compromised, remove from network and do full recovery • Management server • Only Administrator accounts • Persistent connections to other main servers under ‘GNU screen’, a terminal multiplexer • All Administrators use password protected RSA keys on known client machines to connect to CSC machines • Trojan SSHD can’t capture administrator passwords • Administrators use ‘sudo’ exclusively

31. CSC Systems Inspection • Attempted logins from all 32 compromised accounts • Only user ‘grunwald’ was successful • First, make sure ‘grunwald’ is not logged in and lock the account: • `sudo mkpasswd grunwald` • Change shell to /sbin/nologin • Examine the following looking for any suspicious files: • /home/grunwald • /tmp • /var/tmp • Examine the process table: • `ps auxw`

32. CSC Systems Inspection • Manually Run Tripwire, Osiris, Samhain • These are “Filesystem Integrity Scanners” • Example Tripwire Output (Osiris is similar): Host name: hemisphere Host IP address: 128.138.207.210 Policy file used: /etc/tripwire/tw.pol Configuration file used: /etc/tripwire/tw.cfg Database file used: /var/lib/tripwire/hemisphere.twd Command line used: /usr/sbin/tripwire --check … ------------------------------------------------------------------------------- Rule Name: Critical configuration files Severity Level: 100 ------------------------------------------------------------------------------- Modified: "/etc/crontab" "/etc/fstab" "/etc/motd" …

33. CSC Systems Inspection • These Filesystem Integrity Scanners verify that the system binaries and configuration files are not compromised • Attacker would have to modify the Filesystem Integrity Scanner database to prevent notification • Could also disable the cron entry • If you monitor systems, be sure to watch for missing emails • Tripwire and Osiris use cron entries to run daily checks, which send out a single summary email • Samhain has a client-server architecture, can send out alerts within minutes of finding an issue • We highly recommend Samhain for new installations

34. CSC Systems Inspection "Hemisphere was the only system to survive the attacks” • Why? • Luck • We do what we can to limit vulnerabilities but focus most of our attention on early detection and the ability to recover • Have taken the time to focus on security • Is not easy, or quick • So what have we done?

35. CSC Security Tools and Policies • In addition to the Filesystem/Configuration Integrity Scanning and Reporting: • System Hardening • Foreign login detection and reporting (custom scripts) • “Rootkit” scanning and reporting (custom wrapper scripts) • We have and follow Incident Response Procedures

36. CSC Security Tools and Policies • Centralized Log Server • We were in the process of building this server when the CS compromise happened • All syslog, Samhain, chkrootkit, rkhunter logs to this server • Extremely secure configuration: • Only three Administrators have accounts, use different passwords from any other system • Root cannot login • EAL4+ Common Criteria Installation • IBM RPM/script which hardens SuSE Enterprise Linux 9

37. CSC Security Tools and Policies • System Hardening • Common Service Audit • Remove extraneous installed packages • Only core, necessary services running • Configuration of required software • PAM: ssh, sudo, su • Host based firewalling • Shorewall • User Policies • We run John-the-Ripper Password cracking utility • Users are created with account aging

38. CSC Security Tools and Policies • chkrootkit - a tool to scan for rootkits • We have a wrapper to consolidate the output from the many servers into a single email: • Lack of scalability in security tools is an issue! chkrootkit-0.45 found the following warnings, vulnerabilities, and infections on hosts: node0[01-64],hemisphere transputer occam,blade0[01-27] loaner,store01,store02,store03 Errors occured on the follwing nodes that prevented chkrootkit from completing: NONE The following hosts were not scanned: node008 blade003 blade016 All logs generated for this run are located at: /home/admin/logs/chkrootkit/20051206/ Quick Stats: warnings = 0 vulnerabilities = 0 infections = 0

39. CSC Security Tools and Policies Foreign Login Detection custom script • Detects login activity from new machines • Short summary describes new logins • Administrator follow-up • Look for accesses from outside of known activity • System administrators contact Advisor/PI to verify student’s location (this gets their attention) • We have caught several compromised passwords this way (Source: CSC Group Slide by Matthew Woitaszek)

40. CSC Security Tools and Policies • User training: Don’t type your Hemisphere password in on a remote machine, it might have a trojan SSH! Connect directly from your client! Foreign Login Detection Example -------------------------------------- Host report times hemisphere 2005-12-01 00:02:15 -------------------------------------- First logins for each user from a host oberg 2005-11-30 14:05:50 hemisphere ec240-175-dhcp.colorado.edu 2005-11-30 15:47:43 hemisphere ec240-250-dhcp.colorado.edu sliu 2005-11-30 23:44:33 hemisphere ae203-053.resnet.stonybrook.edu 2005-11-30 23:46:22 hemisphere ae203-053.resnet.stonybrook.edu Why is sliu, usually at a desk in ECCS122, connecting from a dorm in Long Island at 1:45AM Eastern Time? (Source: CSC Group Slide by Matthew Woitaszek)

41. CSC Security Tools and Policies Incident Response Procedure • Secure email: Commercial PGP, GPG for the rest • Cell phones • Upon any notification of compromise, immediately disconnect all machines from the internet, and conduct onsite analysis • Start to download offsite backups for restore

42. Questions?