Scalable Network
1 / 31

Scalable Network - based Buffer Overflow Attack Detection - PowerPoint PPT Presentation

  • Uploaded on

Scalable Network - based Buffer Overflow Attack Detection. Tzi-cker Chiueh Computer Science Department Stony Brook University Stony Brook, NY, U.S.A. Fu-Hau Hsu Department of Computer Science and Information Engineering National Central University

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Scalable Network - based Buffer Overflow Attack Detection' - dore

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Scalable network based buffer overflow attack detection

Scalable Network-basedBuffer Overflow AttackDetection

Tzi-cker Chiueh

Computer Science


Stony Brook University

Stony Brook, NY, U.S.A.

Fu-Hau Hsu

Department of Computer

Science and Information


National Central University

Taoyuan, Taiwan, R.O.C.

Fanglu Guo

Symantec Research


Cupertino, CA, U.S.A.

ANCS 2006

Virulence of buffer overflow attacks
Virulence of Buffer Overflow Attacks

  • Buffer overflow attack is arguably the most widely used and thus most dangerous attack method used today.

  • Most Internet Worms use it to proliferate themselves.

  • It accounts for more than 50% of all the security vulnerabilities recorded by CERT.

ANCS 2006

Proposed solutions
Proposed Solutions

  • Compiler Transformation

    • Stack Guard, RAD, Address Obfuscation

  • Library Rewriting

  • OS

    • Non-executable Stack

  • Instruction Set

  • Hardware

    • AMD Athlon-64

ANCS 2006

Discrepancy between theory and practice
Discrepancy between Theory and Practice

  • In theory, these efforts have largely solved the buffer overflow attack problem.

  • In practice, however, new buffer overflow vulnerabilities are still discovered and reported on a routine basis.

ANCS 2006

Scalable network based buffer overflow attack detection

ANCS 2006

A solution to the above dilemma nebula
A Solution to the above Dilemma --Nebula

  • Nebula

  • A network-based buffer overflow attack detection mechanism

  • Observe the network traffic only to detect BOAs

  • Currently version is developed for Linux paltforms.

ANCS 2006

Existing network based intrusion detection system nids
Existing Network-based Intrusion Detection System (NIDS)

  • Misuse intrusion detection

    • Zero-day BOAs

    • Labor-Intensive

      • Solution: automatically signature-generating approaches

  • Anomaly intrusion detection

    • False Positive

ANCS 2006

Two factors for a successful buffer overflow style attack
Two Factors for a Successful Buffer Overflow-style Attack

  • A successful buffer overflow-style attack should be able to overflow the right place (e.g. the place to hold a return address with the correct value (e.g. the address of injected code entry point)).

ANCS 2006

Non predicable offset and entry point address
Non-predicable Offset and Entry Point Address

return address

buffer where the

overflow start

injected code

address of injected code

entry point.

offset between the beginning of the

overflowed buffer and the overflow


The offset and the entry point address are non-predicable. They can

not decided by just looking the source code or local binary code.

ANCS 2006

Non predicable offset
Non-predicable Offset

  • For performance concerns, most compilers don’t allocate memory for local variables in the order they appear in the source code, sometimes some space may be inserted between them. (Source Code doesn’t help)

  • Different compiler/OS uses different allocation strategy. (Local binaries don’t help)

  • Address obfuscation insert random number of space between local variables and return address. (Super good luck may help)

ANCS 2006

Non predicable entry point address

Function main()’s stack frame

Non-predicable Entry Point Address

webserver –a –b security


system data


environment variables

argument strings

command line arguments

and environment variables

env pointers

argv pointers


ANCS 2006

Strategies used by attackers to increase their success chance
Strategies Used by Attackers to Increase Their Success Chance

  • Repeat address patterns.

  • Insert NOP (0x90) operations before the entry point of injected code.

ANCS 2006

Indispensable elements of bo style attacks
Indispensable Elements of BO-style Attacks Chance

‘ The Address ’

  • For buffer overflow attacks, it is the address of the entry point of injected code.

ANCS 2006

Linux process memory layout
Linux Process Memory Layout Chance


kernel address space


address space of

addresses of

injected code and frame pointers

(Stack Address Zone)

user stack



for Shared libraries,

including libc functions



run-time heap

data and code

ANCS 2006

Size of stack address zone
Size of Stack Address Zone Chance

  • The default maximum size of a process’s user space stack is 8 Mbytes.

  • However, according to Ditzel et al., the average function frame size is 28 bytes.

  • Therefore, the majority of program are not supposed to use a 2Mbyte stack.

  • In our test, a 8k stack is enough to identify all 10 remote exploit strings.

ANCS 2006

Repeating times and values of return addresses
Repeating Times and Values of Return Addresses Chance

2k stack --- 0xbffffffff ~ 0xbfffe000

ANCS 2006

A property of stack addresses
A Property of Stack Addresses Chance

  • The leading byte of any words that contain a stack address corresponds to a non-printable ASCII character.

ANCS 2006

Generalized signature
Generalized Signature Chance

  • Signature of a stack smashing buffer overflow attack :

    If a sub-string of a traffic string could be interpreted as a stack address that repeats 3 or more times, it is alarmed as a buffer overflow attack string.

ANCS 2006

Scalable network based buffer overflow attack detection

Contextual Analysis Chance

ANCS 2006

Bypassing detection
Bypassing Detection Chance

  • Patient attackers could bypass detection based on repeating address signature by repeating addresses no more than 2 times.

    • PS: All the 10 remote exploit code we tested repeat at least 4 times.

  • Attackers repeat the addresses to increase their chance to success. In other words, it is very likely that without the repeat, attackers will fail many times before getting a successful one.

ANCS 2006

Unsuccessful attacks
Unsuccessful Attacks Chance

  • Buffer overflow-style attacks will destroy targeted process’s address space which in turn usually will crash the attacked process.

  • In order to recycle valuable system resources, OS will close the sockets opened by crash processes automatically.

  • On both Linux and Windows, when a program is crashed, the OS will terminate all the program’s pending socket connections by sending out an RST packet to the communicating hosts on its behalf.

ANCS 2006

Server termination signature
Server Termination Signature Chance

  • After forwarding a sub-string which could be interpreted as a single stack address, Nebula detects that the server closes the TCP connection without sending any data, then the traffic string is deemed as a buffer overflow attack string.

  • Future traffic coming from the same hosts will be blocked or examined thoroughly.

ANCS 2006

Will normal traffic behavior the same way
Will Normal Traffic Behavior the Same Way? Chance

  • HTTP Protocol (RFC 2616) works in the request-reply way. (After the request, there will be a reply before the server close the connection)

  • SMTP protocol (RFC 2821), for e-mail, and FTP protocol(RFC 959) use QUIT command to close a connection. (QUITcan not be interpreted as a stack address.)

ANCS 2006

Scalable network based buffer overflow attack detection

Payload Bypassing Chance

ANCS 2006

Payload bypassing
Payload Bypassing Chance

  • Payload bypassing tries to avoid packet analysis for as much traffic as possible.

  • Because most buffer overflow attacks take place during the exchange of control messages, it is safe to ignore the bulk of data that is downloaded as uninterpreted bytes.

    • For example, in an FTP session, data transferred over the data connection can never be used to mount a buffer overflow attack against the FTP program because the FTP program does not interpret them.

ANCS 2006

Internet traffic statistic
Internet Traffic Statistic Chance

  • From CacheLogic’s measurement on USA, Europe, and Asia backbone in June 2004, HTTP and P2P packets accounted for more than 70% of the total traffic.

ANCS 2006

Percentage of payload
Percentage of Payload Chance

  • Percentage of payload in the traffic when each of the four protocols that Nebula can recognize is used to transfer files of a total size of 1.22 Gbytes.

ANCS 2006

Number of false positives without payload bypassing
Number of False Positives without Payload Bypassing Chance

  • Number of false positives under the our sample as reported by Nebula.

  • The minimal number of times the attack pattern is repeated is assumed to be 1, 2, 3 or 10, and the stack size tested is 2Mbytes, 16Kbytes, or 8Kbytes. In each entry the left is the number of false positives for RTL attacks, whereas the right is the number of false positives for CI attacks.

  • The sample includes 134966 TCP connections and about 1.582 Gbytes of data.

ANCS 2006

Number of false positives with payload bypassing
Number of False Positives with Payload Bypassing Chance

  • The number of false positives in the test traffic associated with different protocols after applying payload bypassing is negligible even when the attack pattern repetition count is 1.

ANCS 2006

Throughput comparison
Throughput Comparison Chance

  • The throughput of Nebula under a test HTTP connection when different options are turned on. With payload bypassing, Nebula can perform buffer overflow attack detection and still achieve a throughput higher than a generic Linux router.

ANCS 2006