malware detection based on application behavior modeling n.
Skip this Video
Download Presentation
Malware Detection based on Application Behavior Modeling

Loading in 2 Seconds...

play fullscreen
1 / 77

Malware Detection based on Application Behavior Modeling - PowerPoint PPT Presentation

  • Uploaded on

Malware Detection based on Application Behavior Modeling. NWMTD’11 Jun 20–21, 2011. Mrs P.R.Lakshmi Eswari C-DAC, Hyderabad. Evolution of Malware Attacks. Malware Definition (Wikipedia). A software which is designed to infiltrate a computer system without the owner’s informed consent

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Malware Detection based on Application Behavior Modeling' - diem

Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
malware detection based on application behavior modeling

Malware Detection based on Application Behavior Modeling


Jun 20–21, 2011

Mrs P.R.Lakshmi Eswari

C-DAC, Hyderabad

malware definition wikipedia
Malware Definition (Wikipedia)

A software which is designed to infiltrate a computer system without the owner’s informed consent

Refers to a variety of forms of hostile, intrusive, annoying software code

MALicious softWARE

threat from the malware
Threat from the Malware
  • A code
    • which collects the credit card number or any other personal info
    • Which makes an application do the buffer overflow and crash
    • Loosing the private and sensitive information
    • which shows annoying advertisements without your consent
    • Which encrypts the data and asks for money to decrypt it
a typical malware
A Typical Malware

Exploit Logic

  • Motivational Logic
    • Spam
    • Data theft
    • Ransom
    • Disrupt the routine
  • Protection Logic
    • Packing
    • Anti Debugging
    • Anti Virtualization

Propagation Logic



attacks classified
Attacks - Classified
  • Untargeted attacks
    • Attacking websites
    • Infecting portable storage devices
    • Attacking social networking websites
    • Wild malware (worms etc)
  • Botnets
  • Targeted Attacks
a typical attack
A Typical Attack

Originally a executable

Doc file

Opens the file,

and executes the malware

Whenever updates windows, also downloads the malware, sends the data out etc.


Changes the windows update program


IRC Server

4. Attacker will also join this channel

(preferably through a program) and

issue commands (for e.g. update)

3. Join a channel on IRC

Receives the command (update)

1. Exploit / Attack


2. Download malware (bot)


DDoS (distributed denial of service attacks)

Collecting lot of bank related data

Spidering attacks (on websites)


Using victim for other sensitive attack

Shutdown the computer etc

malware detection techniques
Malware Detection Techniques
  • Black listing
    • Anti Virus
    • Intrusion Detection System
    • Behavior Based Malware Detection
  • White listing
    • Specification Based Detection
    • Anomaly Detection
end system security suites
End System Security Suites
  • Centralized configuration on all clients
  • Centrally controlled
    • Firewall
    • Encryption
    • Device Control
    • Anti Malware
    • Security policies
white listing solutions
White listing Solutions

Core Trace Bouncer

Bit9 Parity

Robot Genius

Microsoft App Locker

McAfee Application Control

don t want to pay
Don’t want to pay ? !

Free Anti Virus [AVG, AVIRA, AVAST]

Free Firewall [Zone Alarm]

URL Scanner [AVG, WOT, RG Guard]

Trend Micro Web Protection Add on

Disable Auto runs

Returnil Virtual System / Windows Steady State

Wehn-Trust HIPS [MUST for Windows XP – ASLR Tool]

Win-pooch HIPS [Windows XP]


WinPatrol [BillP Studios]

how anti malware works
How anti malware works?

Behaviors database

Behavior Based Engine

(On Process Activities)

Basic Activity Scanning *

Malware Signature database

Anti Virus Scanning

(On file content)

Known Applications database

White listing

(On process creation)

( * Process activity, file read or write )

mps approach
MPS - Approach

Each application makes sequence of system calls for accessing various OS resources through multiple control paths (normal behaviour)

When the application is infected with malware, its behaviour changes

User Process n

User Process 1

User Process 2


User Space

System Calls

Kernel Space

Operating System

Detects malicious activity before it causes damage to end system i.e. before the system calls are executed by the operating system


Malware Prevention System

1. Application Profiling and Model Generation Process in a Sandbox

4. Client

Protection against overall threats -

Process Execution Control

Model Enforcement Module

2. Server Manages the models and admin can set the policies here

Server communication module

3. Based on the policies the model gets pushed to clients



Resource - A

System calls : {1,2,4}

Resource - B

System calls: {1,3,4,2}

Resource - C

System calls: {1,2,4}

Model Generation

operations hooked in mps
Operations Hooked in MPS

File System Calls

Process hooks

Network Calls

Registry Calls

update request
Update Request

MPS Server

MPS Client



Major No,

Minor No,

OS type,


Db Major No,

Db Minor No

No.of Model Files,

Model File names,

ModelFile Path

file transfer request
File Transfer Request

MPS Server

MPS Client



Model File

Name with


Contents of

the Model File

log message request
Log Message Request

Application name,

OS type,








client and server technologies used
Client and Server – Technologies used

Server on Linux

Apache Server 2.2

Virtual Machine

Windows XP, Vista and 7 images

Linux 2.6.23 kernel image

Java runtime environment


HTTP message format


Windows Client

Mini Filter Driver

Call out Drivers

Win32 programming

C, C++ programming

PE Executable format Open SSL

Linux Client

Linux Security Modules

C, C++ programming

Qt Programming


malicious pdf
Malicious Pdf

Creation of Axsle.dll

Creation of Icucnv34.dll

Write file on cvs.exe

The malware repeatedly tries to write cvs.exe file and it gets blocked. The document doesn’t open until the write file operation on cvs.exe is completed.



  • Behaviors Detected
    • Hides view of system files
    • Hidden image file
    • File has system attribute
    • Creates logon entry
    • Unsigned binary
    • Drops executable
    • Modifies internet settings
    • Spawns process
att27390 doc file
ATT27390 doc file
  • Activities blocked
    • Dropping of zipfldr.dll in system32 folder
    • Dropping of wuaueng.dll in system32 folder
field testing report
Field Testing Report
  • MPS is compared with similar best commercial tools available in the market like NovaShield, Mamutu, Malware Defender, Sana Security Primary Response, Safe Connect, Threat fire etc.
field testing report1
Field Testing Report
  • MPS is found sensitive against blended MS office and PDF documents wherein the MPS solution alone identified the malicious activity as the other industry product remain silent
  • Application has a tendency to raise false alarm against benign documents as it might match the enforcement policies defined
  • Overall it is felt that the solution is detecting high level targeted malware behaviours, but there is a need to improve the capabilities by suppressing the false alarms.
malware resist simplifying and strengthening security
Malware ResistSimplifying and Strengthening Security
  • Detection Based on Runtime Behaviour. All running programs are monitored for a set of critical behaviors that could affect the normal functioning

Salient Features

Detection Based on Runtime Behavior

Small memory footprint and high detection rate

Co-exists with Anti Virus Solutions

Low False Positive Rate

Easy to Deploy and Use

ongoing research @ c dac hyderabad
Ongoing Research @ C-DAC Hyderabad

Design and Development of Anti Malware Solution for Web Applications and Mobiles

the approach to analyze the malware
The approach to analyze the Malware

Run the malware in isolated lab

Monitor network and system connections

Understand the program’s code

Repeat until satisfied with gathered info

how to
How to?
  • Manual
    • Dedicated system (ready to be compromised)
    • Virtualized System
  • Automated Analysis
automated analysis
Automated Analysis

Anubis [analyzing unknown binaries]


Virus total [analyze suspicious file]


Bit-Blaze [Malware Analysis Service]


Norman Sandbox

Joe Box Sandbox

Sunbelt CWSandBox

Comodo [Comodo Instant Malware Analysis]

two steps phases
Two Steps / Phases

Behavioral (Dynamic) Analysis

Code (Static) Analysis

Gather as much as from behavioral analysis

Fill the gaps from the code analysis

malware analysis1
Malware Analysis

To analyze malware, we requires basic and advanced knowledge in Windows and Linux concepts (depends)

For example: while doing behavioral analysis of the malware, we find malware modifies file A. – To get more out of it, we must know what is the significance of file A

prepare the system
Prepare the System
  • Use VMWare and use the snapshot feature to restore state after malware execution
  • Use Virtual PC – execute the malware – Close and Delete changes
  • Physical System State Restore
    • Returnil Virtual System
    • Windows Steady State
behavioral analysis
Behavioral Analysis
  • Activate various monitoring tools
  • Execute the malware
  • Terminate / suspend the malware process
    • Sometimes malware process comes again and again
  • Observe the results of monitoring tools
process explorer
Process Explorer
  • Free from Microsoft TechNet
  • Super Task Manager
  • Shows process tree
    • We can know if malware created the new processes
  • Also shows files which a process is using
  • Can see the strings also
process monitor
Process Monitor
  • Free from Microsoft TechNet
  • Monitors the following activities
    • Process creation
    • File related
    • Registry
    • Network related
  • Captures for all the process
    • Best is to do it for all and then apply the filters
using idapro
Using IDAPro

Can reveal a lot of information

Great tool if user can reverse the C/C++ code

use ollydbg
Use OllyDbg
  • OllyDbg is a great debugger
  • Open the sample using OllyDbg
  • Either use snort in a separate virtual machine to monitor its network activity
  • Or use tools like wire shark
  • Find
    • IRC server to whom this sample connects
    • Web servers?
  • May notice DNS queries
packed malicious executables
Packed Malicious Executables
  • Packers compress / encrypt the executable
  • This is used
    • Difficult to analyze
    • Smaller size on hard disk
  • However runs unpacked and original in memory
how it executes
How it executes?

Small Decryptor extracts

the packed code and

executes the code

Executable Decryptor

Unpacked program in memory

Packed program stored as data

pe format
PE Format


MS-DOS Stub Program









if it is packed
If it is packed



MS-DOS Stub Program

This is Decryptor code

MS-DOS Stub Program










Original PE


packers availiable
Packers Availiable






process dumping with lordpe
Process dumping with LordPE
  • LordPE shows all the processes and can dump there images from memory
  • We can run the process from packed executable
    • Anyways it has to unpack itself in the memory
  • We can dump from memory using LordPE