Introduction to honeypot denial of service and rootkit
This presentation is the property of its rightful owner.
Sponsored Links
1 / 32

Introduction to Honeypot, Denial-of-Service, and Rootkit PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Introduction to Honeypot, Denial-of-Service, and Rootkit. Cliff C. Zou CAP6135 Spring, 2010. What Is a Honeypot?. Abstract definition: “A honeypot is an information system resource whose value lies in unauthorized or illicit use of that resource.” (Lance Spitzner) Concrete definition:

Download Presentation

Introduction to Honeypot, Denial-of-Service, and Rootkit

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Introduction to honeypot denial of service and rootkit

Introduction to Honeypot, Denial-of-Service, and Rootkit

Cliff C. Zou


Spring, 2010

What is a honeypot

What Is a Honeypot?

  • Abstract definition:

    “A honeypot is an information system resource whose value lies in unauthorized or illicit use of that resource.” (Lance Spitzner)

  • Concrete definition:

    “A honeypot is a faked vulnerable system used for the purpose of being attacked, probed, exploited and compromised.”

Example of a simple honeypot

Example of a Simple Honeypot

  • Install vulnerable OS and software on a machine

  • Install monitor or IDS software

  • Connect to the Internet (with global IP)

  • Wait & monitor being scanned, attacked, compromised

  • Finish analysis, clean the machine

Benefit of deploying honeypots

Benefit of Deploying Honeypots

  • Risk mitigation:

    • Lure an attacker away from the real production systems (“easy target“).

  • IDS-like functionality:

    • Since no legitimate traffic should take place to or from the honeypot, any traffic appearing is evil and can initiate further actions.

Benefit of deploying honeypots1

Benefit of Deploying Honeypots

  • Attack analysis:

    • Find out reasons, and strategies why and how you are attacked.

    • Binary and behavior analysis of capture malicious code

  • Evidence:

    • Once the attacker is identified, all data captured may be used in a legal procedure.

  • Increased knowledge

Honeypot classification

Honeypot Classification

  • High-interaction honeypots

    • A full and working OS is provided for being attacked

    • VMware virtual environment

      • Several VMware virtual hosts in one physical machine

  • Low-interaction honeypots

    • Only emulate specific network services

    • No real interaction or OS

      • Honeyd

  • Honeynet/honeyfarm

    • A network of honeypots

Low interaction honeypots

Low-Interaction Honeypots

  • Pros:

    • Easy to install (simple program)

    • No risk (no vulnerable software to be attacked)

    • One machine supports hundreds of honeypots, covers hundreds of IP addresses

  • Cons:

    • No real interaction to be captured

      • Limited logging/monitor function

      • Hard to detect unknown attacks; hard to generate filters

    • Easily detectable by attackers

High interaction honeypots

High-Interaction Honeypots

  • Pros:

    • Real OS, capture all attack traffic/actions

    • Can discover unknown attacks/vulnerabilites

    • Can capture and anlayze code behavior

  • Cons:

    • Time-consuming to build/maintain

    • Time-consuming to analysis attack

    • Risk of being used as stepping stone

    • High computer resource requirement



  • A network of honeypots

  • High-interaction honeynet

    • A distributed network composing many honeypots

  • Low-interaction honeynet

    • Emulate a virtual network in one physical machine

    • Example: honeyd

  • Mixed honeynet

    • “Scalability, Fidelity and Containment in the Potemkin Virtual Honeyfarm”, presented next week

  • Reference:

Honeypot aware botnet zou 07

Honeypot-Aware Botnet [Zou’07]

  • Honeypot is widely used by defenders

    • Ability to detect unknown attacks

    • Ability to monitor attacker actions (e.g., botnet C&C)

  • Botnet attackers will adapt to honeypot defense

    • When they feel the real threat from honeypot

    • We need to think one step ahead

Honeypot detection principles

Honeypot Detection Principles

  • Hardware/software specific honeypot detection

    • Detect virtual environment via specific code

      • E.g., time response, memory address

    • Detect faculty honeypot program

    • Case by case detection

  • Detection based on fundamental difference

    • Honeypot defenders are liable for attacks sending out

      • Liability law will become mature

      • It’s a moral issue as well

    • Real attackers bear no liability

      • Check whether a bot can send out malicious traffic or not

Detection of honeypot bot

Detection of Honeypot Bot

  • Infection traffic

    • Real liability to defenders

    • No exposure issue: a bot needs to do this regardless

  • Other honeypot detection traffic

    • Port scanning, email spam, web request (DoS?)

  • bot

  • Sensor (secret)

  • 1 malicious traffic

  • 2 Inform bot’s IP

  • 3 Authorize

  • C&C

Two stage reconnaissance to detect honeypot in constructing p2p botnets

Two-stage Reconnaissance to Detect Honeypot in Constructing P2P Botnets

  • Fully distributed

    • No central sensor is used

    • Could be fooled by double-honeypot

      • Counterattack is presented in our paper

  • Lightweighted spearhead code

    • Infect + honeypot detection

    • Speedup UDP-based infection

  • 1

  • Host A

  • Host B

  • Host C

  • 2

  • spearhead

  • spearhead

  • request

  • main-force

  • 3

Defense against honeypot aware attacks

Defense against Honeypot-Aware Attacks

  • Permit dedicated honeypot detection systems to send out malicious traffic

    • Need law and strict policy

  • Redirect outgoing traffic to a second honeypot

    • Not effective for sensor-based honeypot detection

  • Figure out what outgoing traffic is for honeypot detection, and then allow it

    • It could be very hard

  • Neverthless, honeypot is still a valuable monitoring and detection/defense tool

Distributed denial of service ddos attack

Distributed Denial of Service (DDoS) Attack

  • Send large amount of traffic to a server so that the server has no resource to serve normal users

  • Attacking format:

    • Consume target memory/CPU resource

      • SYN flood (backscatter paper presented before)

      • Database query…

    • Congest target Internet connection

      • Many sources attack traffic overwhelm target link

      • Very hard to defend

Why hard to defined ddos attack

Why hard to defined DDoS attack?

  • Internet IP protocol has no built-in security

    • No authentication of source IP

      • SYN flood with faked source IP

      • However, IP is true after connection is setup

  • Servers are supposed to accept unsolicited service requests

  • Lack of collaboration ways among Internet community

    • How can you ask an ISP in another country to block certain traffic for you?

Ddos defenses

DDoS Defenses

  • Increase servers capacity

    • Cluster of machine, Multi-CPUs, larger Internet access

  • Use Internet web caching service

    • E.g., Akamai

  • Defense Methods (many in research stage)

    • SYN cookies (

    • SOS

    • IP traceback

Syn cookies

SYN Cookies

  • SYN flood attack

    • Fill up server’s SYN queue

    • Property: attacker does not respond to SYN/ACK from victim.

  • Defense

    • Fact: normal client responds to SYN/ACK

    • Remove initial SYN queue

    • Server encode info in TCP seq. number

      • Use it to reconstruct the initial SYN

Dos spoofed attack defense ip traceback

DoS spoofed attack defense: IP traceback

  • Suppose a victim can call ISPs upstream to block certain traffic

  • SYN flood: which traffic to block?

  • IP traceback:

    • Find out the real attacking host for SYN flood

    • Based on large amount of attacking packets

    • Need a little help from routers (packet marking)

Sos secure overlay service

SOS: Secure Overlay Service

  • Central Idea:

    • Use many TCP connection respondent machines

    • Only setup connections relay to server

    • Identity of server is secrete

The evolution of malware

The Evolution of Malware

  • Malware, including spyware, adware and viruses want to be hard to detect and/or hard to remove

  • Rootkits are a fast evolving technology to achieve these goals

    • Cloaking technology applied to malware

    • Not malware by itself

    • Example rootkit-based viruses: [email protected], [email protected]

  • Rootkit history

    • Appeared as stealth viruses

      • One of the first known PC viruses, Brain, was stealth

    • First “rootkit” appeared on SunOS in 1994

      • Replacement of core system utilities (ls, ps, etc.) to hide malware processes



  • Modern rootkits can cloak:

    • Processes

    • Services

    • TCP/IP ports

    • Files

    • Registry keys

    • User accounts

  • Several major rootkit technologies

    • User-mode API filtering

    • Kernel-mode API filtering

    • Kernel-mode data structure manipulation

    • Process hijacking

  • Visit for tools and information

User mode api filtering


Explorer.exe, Malware.exe, Winlogon.exe

User-Mode API Filtering

  • Attack user-mode system query APIs

  • Con: can be bypassed by going directly to kernel-mode APIs

  • Pro: can infect unprivileged user accounts

  • Examples: HackerDefender, Afx




user mode

kernel mode

Kernel mode api filtering



Explorer.exe, Malware.exe,Winlogon.exe

Kernel-Mode API Filtering



  • Attack kernel-mode system query APIs

  • Cons:

    • Requires admin privilege to install

    • Difficult to write

  • Pro: very thorough cloak

  • Example: NT Rootkit

user mode

kernel mode


Kernel mode data structure manipulation

Kernel-Mode Data Structure Manipulation

  • Also called Direct Kernel Object Manipulation

  • Attacks active process data structure

    • Query API doesn’t see the process

    • Kernel still schedules process’ threads

  • Cons:

    • Requires admin privilege to install

    • Can cause crashes

    • Detection already developed

  • Pro: more advanced variations possible

  • Example: FU





Process hijacking

Process Hijacking

  • Hide inside a legitimate process

  • Con: doesn’t survive reboot

  • Pro: extremely hard to detect

  • Example: Code Red



Detecting rootkits

Detecting Rootkits

  • All cloaks have holes

    • Leave some APIs unfiltered

    • Have detectable side effects

    • Can’t cloak when OS is offline

  • Rootkit detection attacks holes

    • Cat-and-mouse game

    • Several examples

      • Microsoft Research Strider/Ghostbuster

      • RKDetect

      • Sysinternals RootkitRevealer

      • F-Secure BlackLight

Simple rootkit detection

Simple Rootkit Detection

  • Perform a directory listing online and compare with secure alternate OS boot (see )

    • Offline OS is Windows PE, ERD Commander, BartPE

      dir /s /ah * > dirscan.txt

      windiff dirscanon.txt dirscanoff.txt

  • This won’t detect non-persistent rootkits that save to disk during shutdown


Filtered Windows API

omits malware files and keys

Malware files and keys are visible in raw scan


  • RootkitRevealer (RKR) runs online

  • RKR tries to bypass rootkit to uncover cloaked objects

    • All detectors listed do the same

    • RKR scans HKLM\Software, HKLM\System and the file system

    • Performs Windows API scan and compares with raw data structure scan



Windows API

Raw file system, Raw Registry hive

Introduction to honeypot denial of service and rootkit


  • HackerDefender

    • HackerDefender before and after view of file system

    • Detecting HackerDefender with RootkitRevealer

Rootkitrevealer limitations

RootkitRevealer Limitations

  • Rootkits have already attacked RKR directly by not cloaking when scanned

    • RKR is given true system view

    • Windows API scan looks like raw scan

  • SysInternals have modified RKR to be a harder to detect by rootkits

    • RKR is adopting rootkit techniques itself

    • Rootkit authors will continue to find ways around RKR’s cloak

    • It’s a game nobody can win

Dealing with rootkits

Dealing with Rootkits

  • Unless you have specific uninstall instructions from an authoritative source:

  • Don’t rely on “rename” functionality offered by some rootkit detectors

    • It might not have detected all a rootkit’s components

    • The rename might not be effective

Reformat the system and reinstall Windows!

  • Login