Containment, Eradication, Recovery, Lessons Learned

1. Containment, Eradication, Recovery, Lessons Learned Incident Response

2. Where we’re at • We’ve hopefully done lots of planning beforehand • We’ve found and verified the presence of an incident • We know what’s affected • Have a good idea how they got in • Determined there’s enough evidence to declare an incident • Before we decide how to contain the incident, let’s characterize the incident (CSIRT Case Classification) • Category • Criticality • Sensitivity Incident Response

3. Category • Denial of Service • DOS or DDOS • Compromised Information • Disclosure of sensitive corporate information or intellectual property • Compromised Asset • Compromised host, network device, user account, application • Malware, root account, rootkit, etc. • Unlawful Activity • Theft, Fraud, Computer Safety • Internal Hacking • Suspicious activity originating from inside the network, excluding malware • External Hacking • Suspicious activity originating from outside the network, excluding malware • Email • Spoofed email, spam, phishing • Malware • Policy Violation Incident Response

4. Criticality • Incident affecting critical systems • Potential to impact revenue or customers • Active hacking, compromised critical asset, virus outbreak, etc. • Initial response time: 60 minutes • Incident affecting non-critical systems • Not revenue or customer impacting • Time-sensitive employee investigations • Policy violations, unauthorized access, compromised asset, inactive hacking • Initial response time; 4 hours • Possible incident affecting non-critical systems • Employee investigations not time sensitive, longer-term investigations • Email, forensics request, policy violation • Initial response time: 48 hours Incident Response

5. Sensitivity • Helps define the “need to know” and who should be in the loop • Level 1: Extremely sensitive • Forensics request, destruction of property, unlawful activity, compromised information, etc. • Informed: CSIRT, Management • Computer Security Incident Response Team • Level 2: Sensitive • Hacking, unauthorized access • Informed: CSIRT, Management, System owners, Operations • Level 3: Not sensitive • Denial of service, virus, email • Informed: Employees can be informed as well of isolated infections Incident Response

6. Inform appropriate entities • Thorough planning should help in identifying who should be informed • A senior manager should sponsor the IR team • CISO, CIO, Legal counsel, etc. • Notify them when declaring an incident • Email • Phone call or visit for the critical incidents • Local or organizational incident-handling team • Impacted business unit • Start tracking the incident • Utilize an incident tracking system Incident Response

7. Containment • Goal is to keep the problem from getting worse • A number of ways to effectively contain the incident • Shut down the system • Disconnect the system from the network • Disable certain affected functions of the system • Specific strategy details will vary based on • The type of incident • The goals of the containment strategy Incident Response

8. Containment method decision criteria • Potential damage or theft of data or resources • Evidence preservation requirements • Availability of services • Effort needed to implement • Feasibility • Effectiveness of the strategy • Duration of the strategy • Temporary solution? • More permanent? Incident Response

9. Costs of containment • Do you want to tip your hand? • Some organizations may setup a sandbox and monitor the attacker • Some organizations may determine leaving everything online with stringent monitoring is a better short-term solution • Further determine what’s happening, what is affected • Dangerous solution – could lead to further compromise • Might cause additional damage • Removing internet access from some malware could cause it to behave differently • Malware could start encrypting files when it loses connectivity • Making changes could damage evidence • May need evidence for determining the full scope of the incident • May need evidence for legal purposes Incident Response

10. Short term containment • Keep a low profile • Avoid tipping your hand to the attacker • Don’t start looking for the attacker from the compromised machine with obvious methods • Ping, traceroute, nslookup • Prevent more damage caused by the attacker • Possible short term actions • Disconnect the network cable • Pull the power • Destroys volatile memory, could damage drive • Isolate the switch port, or put it on an “infected VLAN” • Apply network-based firewalls to isolate the affected machine • If you remove the system or disable it, be sure to inform the business unit Incident Response

11. Gathering evidence • Especially in the case you need to preserve for legal proceedings, documentation is essential • Talk with legal professionals to be sure your evidence can be admissible in court • Account for evidence with a chain of custody • Keep an evidence log • Identifying information - location, model number, hostname, IP address, etc. • Name and title of individuals who collected the evidence • Time and date • Evidence storage locations • NIST guide on evidence preservation • NIST SP 800-86, Guide to Integrating Forensic Techniques into Incident Response Incident Response

12. Create forensic images • The quicker you can create a forensic image of the affected systems, the better • Image the memory and filesystem • You’ll lose data if you turn off the system • Volatility and Memoryze can capture memory • As well as provide some analysis capabilities – more on that later • Bit-by-bit images to get all filesystem data is best • Create hashes of the original and images • Not all incidents will allow for full backups and analysis • More time sensitive incidents will require live forensics Incident Response

13. Determine how to continue • Analyze logs and other forensic information • How far did the attacker go? • Be sure to check neighboring systems for compromise • Lateral movement • Determine longer term containment strategies • Ultimately a business decision informed by the incident handler’s input • Move into long term containment phase after backups and evidence have been acquired Incident Response

14. Long term containment • Can begin making changes to the system once forensic artifacts have been obtained • If the system can be kept offline, move to the eradication phase • If the system must be kept online, perform long term containment options • Patch the system and network • Change passwords • Apply firewalls • Use IPS • Shutdown backdoor processes used by the attacker • Long term containment may be long term, but it should still be temporary • Running in production only until a clean replacement system is ready Incident Response

15. Eradication • The goal of eradication is to actually get rid of the attacker • Containment should stop the bleeding • Eradication is actually removing the attacker’s artifacts • Re-format and re-build the machine • Sure way to remove all of the attacker’s artifacts • Restoring from system backups • Be sure it’s a clean backup! • Remove all the artifacts manually from the existing system • Be sure you don’t miss anything Incident Response

16. Improve your defense • One of the most important, sometimes overlooked portions of the eradication phase • Rebuilding is great, removing malicious software is great • But what is stopping it from happening again? • You MUST fix the vulnerabilities • Put additional defensive measures in place • Firewalls • Move the system to a new IP address • Change DNS names • Apply patches • Harden the system • Apply monitoring solutions Incident Response

17. Assess the system • Before you put a rebuilt system into production, perform a vulnerability analysis • System • Network • Related vulnerabilities • Scan the entire existing network in addition to the rebuilt system • Vulnerability scanners can help • Nessus, OpenVAS, NeXpose, etc. • Be sure to not put a vulnerable system back into production! • Attackers will often leverage the same vulnerability across multiple systems • Look everywhere Incident Response

18. Recovery • The goal of recovery is to bring the impacted systems back online to production • Validate the system is back online and running normally • Typically will have the business unit assist in testing functionality • Restoring in off-hours is best • Easier to monitor • Might be over-ruled, the business often will want to restore asap Incident Response

19. Recovery - monitoring • The other major portion of the recovery phase is continued monitoring • Continually look for evidence of the intrusion • On the recovered systems • On the entire network • Check regularly for re-compromise • Script this! • Look for the specific indicators • Utilize network and host-based IPS/IDS • Add indicators of the compromise Incident Response

20. Lessons Learned • Learning and improving • Develop a follow up report • Start as soon as possible, use notes compiled throughout the IR • Report compiled by the on site team and lead incident handler • Have all parties review the report • Hold a meeting within a few days of the end of the incident • Use what you learned to improve • Fix your processes • Fix your technology Incident Response

21. Questions to address • What happened? • How well did the response work? • Were documented procedures adequate? • What other information could have been used sooner? • What hurdles were there? • What would you do differently next time? • Could information sharing have been improved? • What can we do next time to prevent any problems? • How can we detect similar incidents in the future? • Are any additional tools or resources needed? Incident Response