Scalable Network
Download
1 / 31

Scalable Network - based Buffer Overflow Attack Detection - PowerPoint PPT Presentation


  • 77 Views
  • Uploaded on

Scalable Network - based Buffer Overflow Attack Detection. Tzi-cker Chiueh Computer Science Department Stony Brook University Stony Brook, NY, U.S.A. [email protected] Fu-Hau Hsu Department of Computer Science and Information Engineering National Central University

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Scalable Network - based Buffer Overflow Attack Detection' - dore


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Scalable Network-basedBuffer Overflow AttackDetection

Tzi-cker Chiueh

Computer Science

Department

Stony Brook University

Stony Brook, NY, U.S.A.

[email protected]

Fu-Hau Hsu

Department of Computer

Science and Information

Engineering

National Central University

Taoyuan, Taiwan, R.O.C.

[email protected]

Fanglu Guo

Symantec Research

Laboratory

Cupertino, CA, U.S.A.

[email protected]

ANCS 2006


Virulence of buffer overflow attacks
Virulence of Buffer Overflow Attacks

  • Buffer overflow attack is arguably the most widely used and thus most dangerous attack method used today.

  • Most Internet Worms use it to proliferate themselves.

  • It accounts for more than 50% of all the security vulnerabilities recorded by CERT.

ANCS 2006


Proposed solutions
Proposed Solutions

  • Compiler Transformation

    • Stack Guard, RAD, Address Obfuscation

  • Library Rewriting

  • OS

    • Non-executable Stack

  • Instruction Set

  • Hardware

    • AMD Athlon-64

ANCS 2006


Discrepancy between theory and practice
Discrepancy between Theory and Practice

  • In theory, these efforts have largely solved the buffer overflow attack problem.

  • In practice, however, new buffer overflow vulnerabilities are still discovered and reported on a routine basis.

ANCS 2006


ANCS 2006


A solution to the above dilemma nebula
A Solution to the above Dilemma --Nebula

  • Nebula

  • A network-based buffer overflow attack detection mechanism

  • Observe the network traffic only to detect BOAs

  • Currently version is developed for Linux paltforms.

ANCS 2006


Existing network based intrusion detection system nids
Existing Network-based Intrusion Detection System (NIDS)

  • Misuse intrusion detection

    • Zero-day BOAs

    • Labor-Intensive

      • Solution: automatically signature-generating approaches

  • Anomaly intrusion detection

    • False Positive

ANCS 2006



Two factors for a successful buffer overflow style attack
Two Factors for a Successful Buffer Overflow-style Attack

  • A successful buffer overflow-style attack should be able to overflow the right place (e.g. the place to hold a return address with the correct value (e.g. the address of injected code entry point)).

ANCS 2006


Non predicable offset and entry point address
Non-predicable Offset and Entry Point Address

return address

buffer where the

overflow start

injected code

address of injected code

entry point.

offset between the beginning of the

overflowed buffer and the overflow

target.

The offset and the entry point address are non-predicable. They can

not decided by just looking the source code or local binary code.

ANCS 2006


Non predicable offset
Non-predicable Offset

  • For performance concerns, most compilers don’t allocate memory for local variables in the order they appear in the source code, sometimes some space may be inserted between them. (Source Code doesn’t help)

  • Different compiler/OS uses different allocation strategy. (Local binaries don’t help)

  • Address obfuscation insert random number of space between local variables and return address. (Super good luck may help)

ANCS 2006


Non predicable entry point address

Function main()’s stack frame

Non-predicable Entry Point Address

webserver –a –b security

[[email protected]]#

system data

0xbfffffff

environment variables

argument strings

command line arguments

and environment variables

env pointers

argv pointers

argc

ANCS 2006


Strategies used by attackers to increase their success chance
Strategies Used by Attackers to Increase Their Success Chance

  • Repeat address patterns.

  • Insert NOP (0x90) operations before the entry point of injected code.

ANCS 2006


Indispensable elements of bo style attacks
Indispensable Elements of BO-style Attacks Chance

‘ The Address ’

  • For buffer overflow attacks, it is the address of the entry point of injected code.

ANCS 2006


Linux process memory layout
Linux Process Memory Layout Chance

0xffffffff

kernel address space

0xc0000000

address space of

addresses of

injected code and frame pointers

(Stack Address Zone)

user stack

%esp

8M

for Shared libraries,

including libc functions

0x40000000

brk

run-time heap

data and code

ANCS 2006


Size of stack address zone
Size of Stack Address Zone Chance

  • The default maximum size of a process’s user space stack is 8 Mbytes.

  • However, according to Ditzel et al., the average function frame size is 28 bytes.

  • Therefore, the majority of program are not supposed to use a 2Mbyte stack.

  • In our test, a 8k stack is enough to identify all 10 remote exploit strings.

ANCS 2006


Repeating times and values of return addresses
Repeating Times and Values of Return Addresses Chance

2k stack --- 0xbffffffff ~ 0xbfffe000

ANCS 2006


A property of stack addresses
A Property of Stack Addresses Chance

  • The leading byte of any words that contain a stack address corresponds to a non-printable ASCII character.

ANCS 2006


Generalized signature
Generalized Signature Chance

  • Signature of a stack smashing buffer overflow attack :

    If a sub-string of a traffic string could be interpreted as a stack address that repeats 3 or more times, it is alarmed as a buffer overflow attack string.

ANCS 2006


Contextual Analysis Chance

ANCS 2006


Bypassing detection
Bypassing Detection Chance

  • Patient attackers could bypass detection based on repeating address signature by repeating addresses no more than 2 times.

    • PS: All the 10 remote exploit code we tested repeat at least 4 times.

  • Attackers repeat the addresses to increase their chance to success. In other words, it is very likely that without the repeat, attackers will fail many times before getting a successful one.

ANCS 2006


Unsuccessful attacks
Unsuccessful Attacks Chance

  • Buffer overflow-style attacks will destroy targeted process’s address space which in turn usually will crash the attacked process.

  • In order to recycle valuable system resources, OS will close the sockets opened by crash processes automatically.

  • On both Linux and Windows, when a program is crashed, the OS will terminate all the program’s pending socket connections by sending out an RST packet to the communicating hosts on its behalf.

ANCS 2006


Server termination signature
Server Termination Signature Chance

  • After forwarding a sub-string which could be interpreted as a single stack address, Nebula detects that the server closes the TCP connection without sending any data, then the traffic string is deemed as a buffer overflow attack string.

  • Future traffic coming from the same hosts will be blocked or examined thoroughly.

ANCS 2006


Will normal traffic behavior the same way
Will Normal Traffic Behavior the Same Way? Chance

  • HTTP Protocol (RFC 2616) works in the request-reply way. (After the request, there will be a reply before the server close the connection)

  • SMTP protocol (RFC 2821), for e-mail, and FTP protocol(RFC 959) use QUIT command to close a connection. (QUITcan not be interpreted as a stack address.)

ANCS 2006


Payload Bypassing Chance

ANCS 2006


Payload bypassing
Payload Bypassing Chance

  • Payload bypassing tries to avoid packet analysis for as much traffic as possible.

  • Because most buffer overflow attacks take place during the exchange of control messages, it is safe to ignore the bulk of data that is downloaded as uninterpreted bytes.

    • For example, in an FTP session, data transferred over the data connection can never be used to mount a buffer overflow attack against the FTP program because the FTP program does not interpret them.

ANCS 2006


Internet traffic statistic
Internet Traffic Statistic Chance

  • From CacheLogic’s measurement on USA, Europe, and Asia backbone in June 2004, HTTP and P2P packets accounted for more than 70% of the total traffic.

ANCS 2006


Percentage of payload
Percentage of Payload Chance

  • Percentage of payload in the traffic when each of the four protocols that Nebula can recognize is used to transfer files of a total size of 1.22 Gbytes.

ANCS 2006


Number of false positives without payload bypassing
Number of False Positives without Payload Bypassing Chance

  • Number of false positives under the our sample as reported by Nebula.

  • The minimal number of times the attack pattern is repeated is assumed to be 1, 2, 3 or 10, and the stack size tested is 2Mbytes, 16Kbytes, or 8Kbytes. In each entry the left is the number of false positives for RTL attacks, whereas the right is the number of false positives for CI attacks.

  • The sample includes 134966 TCP connections and about 1.582 Gbytes of data.

ANCS 2006


Number of false positives with payload bypassing
Number of False Positives with Payload Bypassing Chance

  • The number of false positives in the test traffic associated with different protocols after applying payload bypassing is negligible even when the attack pattern repetition count is 1.

ANCS 2006


Throughput comparison
Throughput Comparison Chance

  • The throughput of Nebula under a test HTTP connection when different options are turned on. With payload bypassing, Nebula can perform buffer overflow attack detection and still achieve a throughput higher than a generic Linux router.

ANCS 2006


ad