Not "If" but "When"

1. Not "If" but "When" Essential Incident Handling Techniques for System Administrators John Ives (jives@security.berkeley.edu) System and Network Security

2. Outline • What is Incident Handling • The 6 Step Incident Handling Methodology • Identification • Windows • Unix • Remediation • Containment • Eradication • Some Examples • ToolKits • Q & A • Resources

3. What is Incident Handling In a nutshell, Incident Handling is first aid for compromised or possibly compromised computers.  Put another way, it is the immediate triage that occurs to identify the problem, secure your system and to restore it to active duty while preserving evidence that can be used to calculate the damage and/or take legal action.

4. 6 Step Incident Handling Process The SANS method is a 6 step process • Preparation • Identification • Containment • Eradication • Recovery • Lessons Learned Other organizations, like NIST, may merge or split some of these steps but, in essence everyone covers these same steps.

5. Preparation • Make backups regularly and test the backup system • Document systems, software and data - you should know: • what systems you have  • what software is running on your systems • where your data is stored • what is the criticality of each system, service or data source • Develop incident response plans and policies • how will you respond to a virus, a backdoor, rootkits, etc. • do you have different responses for staff vs. faculty and workstations vs. servers, etc.  • when do you call in the cavalry (SNS, Police, etc.) • use forms and checklists to make sure that you don't forget anything • Develop contingency plans (ie. how long can you afford to have that service, system, data down) • Find, Test and Use emergency software • Helix, Knoppix S-T-D • Handling Legal, SB1386, HR and Student Affairs Cases

6. Legal, SB1386, HR and Student Affairs Cases Legal (criminal, civil or regulatory), SB1386, HR and Student Affairs Cases are all special cases and should not be handled alone by anyone. If you encounter one of these cases call in the appropriate group(s) be they SNS, Policy, Human Resources, Campus Council or even the Police Department. Remember, when your in over your head, there is no shame in asking for help, and not asking for help could be the biggest mistake you will make. Establish a Chain of Custody!

7. Identification • Intrusion Detection • backoors in use (RAT, IRC, FTP, web and browser based, etc.) • spyware • Scanning • backdoors (RAT, IRC, FTP, etc.) • AV software • Trojans & viruses (ask me about Torpig) • Dangerous software (RealVNC?) • Logs • firewall • system • database • web server • 'Something doesn't feel right' • Don't discount users that say their systems are behaving odd

8. Containment Containment is the 'in-the-field' triage • Immediate Triage • Disconnect from network • Block at router • Back-up • Byte-by-Byte copy is best • Can be done with hardware duplicator, dd, ghost • Longer Term Containment • remove/disable accounts • shutdown/remove backdoor • change passwords

9. Eradication • Remove any remnants of the compromise (e.g. rootkits, extra tools, etc) by hand • Catching everything may require a forensic analysis ~ OR ~ • Format hard drive and rebuild system from scratch • Reinstall OS, Applications and all patches • Restore data from clean back-up • Enhance security • harden system • chroot services • implement stronger access and user controls • Vulnerability Scan

10. Recovery • Return the system back to active duty • Verify the system is working as expected • Monitor Logs for continued or missed problems

11. Lessons Learned • Take time to review the incident and document any problems • Address shortcomings in policy and procedure • Be sure to look at how you handled the incident • Write an after action report that outlines the causes and documents recommendations • Everyone involved should be able to sign off on this • Where possible try not to blame individuals

12. A Deeper Look • Identification in Windows • Useful Commands • Identification in Unix • Useful Commands  • Containment • Eradication • Some examples • Web backdoor • Windows Exploit • Toolkits (Helix, JohnDoe) • Other Resources

13. Windows Identification • Logs - Look for non standard or dangerous events • "Event log service was stopped" • "Windows File Protection is not active" • Failed logons • AV logs - may have caught it, but did it catch it in time? • Services • Start-up Items (Registry and Startup) • User accounts • Are there any extra Administrators, Users, etc.? • Unusual Network ports or activity • Is the CPU/Disk/Memory usage running higher than it should?

14. Useful Commands: Windows • netstat -naob • net localgroup administrators • reg query  • HKLM\Software\Microsoft\Windows\CurrentVersion\Run • HKLM\Software\Microsoft\Windows\CurrentVersion\Runonce • HKLM\Software\Microsoft\Windows\CurrentVersion\RunonceEx • wmic • process list full • process [pid] (focuses in on a single pid) • startup list full • service list full • autorunsc -i (command line AutoRun) • fports (openports?)  • find "search string"

15. Unix Identification • Logs - Look for non standard or dangerous events • Software or system crashes/reboots • Successful logons from unusual locations or following a series of unsuccessful logons • entering promiscous mode • Extra/New Services and Processes • Weird/Unusual Files and directories • files or directories with names that will be missed (" ", ". ", "...", etc.) • User accounts • How many people have a uid of 0? • Who is in the wheel, root, admin groups • Unusual Network ports or activity • Network cards in promisc mode • Is the CPU/Disk/Memory usage running higher than it should? 

16. Useful Commands: Unix • netstat -an • find • / -uid 0 -perm 4000 -print • / -uid 0 -name ". " -print (also " ", ".. ") • egrep ':0+:' /etc/passwd • lastcomm • lsof • -i - listening processes • -p [pid] - process details • chkrootkit

17. Containment: Immediate At this stage we are hoping to prevent further damage WITHOUT changing the hard drive. • Disconnect from network (just pull the plug) • Ask SNS to block at the router (still can see the local network) • Change network based firewall rules to block access to the backdoor or from the attacker's address/netblock

18. Containment: Backups • Don't do a clean Shutdown.  Pull the plug! • USE CLEAN DISKS not just new ones  • Hardware duplication • LogicCube • Image MASSter • Tableau TD1 • dd  • dd if=/dev/hda of=/dev/hdb bs=4096 • Adepto is a X-Windows front end to dd that also does drive hashing (found on Helix) • Ghost • Doesn't do forensic duplication by default • use IR (image raw) option (earlier version needed -ID)

19. Containment: Longer Term Ideally you can jump this section and go to Eradication, but that is not always possible. If this system must stay in production while the evidence is being analyzed: • Remove backdoor • Change Passwords (recognize that this may not be the only time you have to do this) • Remove vulnerable software • Install patches • Modify firewall to limit access • Restrict vulnerable service to authenticated users (best) or management IPs (acceptable)

20. Eradication • A rebuild will get rid of everything, everytime* • Just be sure to also address the vulnerability used • Thorough examination of hard drive, logs, etc. can allow you to remove just the crud left by the attacker, however this requires a fair amount of forensic skills and if you don't get everything you may be starting over in a short period of time. The following tools will make this more viable: • Tripwire/AIDE (tell you at least some of what has changed) • Sleuth Kit and Autopsy • Encase from Guidance Software • Meld (compare files from before and after)

21. Example: Web backdoor

22. Example: Web backdoor How would you Identify? How would you Contain? How would you Eradicate?

23. Example: Windows Exploit What is wrong with this System? C:\Documents and Settings\Administrator>netstat -anActive Connections  Proto  Local Address          Foreign Address        State  TCP    0.0.0.0:80             0.0.0.0:0              LISTENING  TCP    0.0.0.0:135            0.0.0.0:0              LISTENING  TCP    0.0.0.0:445            0.0.0.0:0              LISTENING  TCP    127.0.0.1:1026         0.0.0.0:0              LISTENING  UDP    0.0.0.0:445            *:*  UDP    0.0.0.0:500            *:*  UDP    0.0.0.0:1025           *:*  UDP    0.0.0.0:4500           *:*  UDP    127.0.0.1:123          *:*  UDP    127.0.0.1:1900         *:*

24. Example: Windows Exploit nmap -sV 192.168.8.128Starting Nmap 4.20 ( http://insecure.org ) at 2008-12-09 23:37 Pacific Standard TimeInteresting ports on 192.168.8.128:Not shown: 1696 filtered portsPORT   STATE SERVICE  VERSION80/tcp open  winshell Microsoft Windows XP 5.1.2600 cmd.exeMAC Address: 00:0C:29:57:E4:A0 (VMware)Service Info: OS: WindowsService detection performed. Please report any incorrect results at http://insecure.org/nmap/submit/ .Nmap finished: 1 IP address (1 host up) scanned in 40.012 seconds

25. Example: Windows Exploit How would you Identify? How would you Contain? How would you Eradicate?

26. ToolKits Helix • Bootable Linux environment based upon Ubuntu • Adepto (a dd front end specifically for forensic imaging) • Regviewer (Windows Registry viewer) • HFS Support • Much Much More http://www.forensicswiki.org/index.php?title=Helix • Can also download Open Source tools to run in various OS's JohnDoe • A compiled perl script for running any command line tool and then sending the results to an email address.  • Configuration done via text files • Can be run from CD • Source code is avaiable from jives@security.berkeley.edu

27. Questions?

28. Additional Resources: Software Helix http://www.e-fense.com/helix/ Knoppix S-T-D http://www.knoppix-std.org/ Johndoe https://kb.berkeley.edu/kb2600 Security related Live CD list  http://www.livecdlist.com/?pick=All&showonly=Security Encase http://www.guidancesoftware.com/ The Sleuth Kit & Auopsy http://www.sleuthkit.org/ Open Source Digital Forensics http://www.opensourceforensics.org/ Darik's Boot and Nuke http://www.dban.org/ Autoruns and Autorunsc http://technet.microsoft.com/en-us/sysinternals/bb963902.aspx

29. More Resources: Whitepapers, etc. Command line reference for Windows, Bash, Oracle, and SQL Server http://www.ss64.com/ Incident Handling Step by Step: Unix Trojan Programs http://www.windowsecurity.com/whitepapers/Incident_Handling_Step_by_Step_Unix_Trojan_Programs.html NIST: Incident Response Publications http://csrc.nist.gov/publications/PubsTC.html#Incident%20Response NIST: Computer Security Incident Handling Guide http://csrc.nist.gov/publications/nistpubs/800-61-rev1/SP800-61rev1.pdf WMIC articles by Ed Skoudis http://www.google.com/search?hl=en&rls=en&q=site%3Aisc.sans.org+skoudis+wmic&btnG=Search