Malware Detection based on Application Behavior Modeling - PowerPoint PPT Presentation

Malware detection based on application behavior modeling
1 / 77

  • Uploaded on
  • Presentation posted in: General

Malware Detection based on Application Behavior Modeling. NWMTD’11 Jun 20–21, 2011. Mrs P.R.Lakshmi Eswari C-DAC, Hyderabad. Evolution of Malware Attacks. Malware Definition (Wikipedia). A software which is designed to infiltrate a computer system without the owner’s informed consent

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Malware Detection based on Application Behavior Modeling

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Malware detection based on application behavior modeling

Malware Detection based on Application Behavior Modeling


Jun 20–21, 2011

Mrs P.R.Lakshmi Eswari

C-DAC, Hyderabad

Evolution of malware attacks

Evolution of Malware Attacks

Malware definition wikipedia

Malware Definition (Wikipedia)

A software which is designed to infiltrate a computer system without the owner’s informed consent

Refers to a variety of forms of hostile, intrusive, annoying software code

MALicious softWARE

Threat from the malware

Threat from the Malware

  • A code

    • which collects the credit card number or any other personal info

    • Which makes an application do the buffer overflow and crash

    • Loosing the private and sensitive information

    • which shows annoying advertisements without your consent

    • Which encrypts the data and asks for money to decrypt it

Malware categories

Malware Categories

A typical malware

A Typical Malware

Exploit Logic

  • Motivational Logic

    • Spam

    • Data theft

    • Ransom

    • Disrupt the routine

  • Protection Logic

    • Packing

    • Anti Debugging

    • Anti Virtualization

Propagation Logic



Attacks classified

Attacks - Classified

  • Untargeted attacks

    • Attacking websites

    • Infecting portable storage devices

    • Attacking social networking websites

    • Wild malware (worms etc)

  • Botnets

  • Targeted Attacks

Targeted attacks

Targeted Attacks

A typical attack

A Typical Attack

Originally a executable

Doc file

Opens the file,

and executes the malware

Whenever updates windows, also downloads the malware, sends the data out etc.


Changes the windows update program



IRC Server

4. Attacker will also join this channel

(preferably through a program) and

issue commands (for e.g. update)

3. Join a channel on IRC

Receives the command (update)

1. Exploit / Attack


2. Download malware (bot)



DDoS (distributed denial of service attacks)

Collecting lot of bank related data

Spidering attacks (on websites)


Using victim for other sensitive attack

Shutdown the computer etc

Motivation and business

Motivation and Business

Motivation and business1

Motivation and Business


Vulnerability exploit and race

Vulnerability, Exploit and Race

Vulnerability exploit and race1

Vulnerability, Exploit and Race

Malware detection techniques

Malware Detection Techniques

  • Black listing

    • Anti Virus

    • Intrusion Detection System

    • Behavior Based Malware Detection

  • White listing

    • Specification Based Detection

    • Anomaly Detection

Commercial solutions

Commercial Solutions

End system security suites

End System Security Suites

  • Centralized configuration on all clients

  • Centrally controlled

    • Firewall

    • Encryption

    • Device Control

    • Anti Malware

    • Security policies

White listing solutions

White listing Solutions

Core Trace Bouncer

Bit9 Parity

Robot Genius

Microsoft App Locker

McAfee Application Control

Don t want to pay

Don’t want to pay ? !

Free Anti Virus [AVG, AVIRA, AVAST]

Free Firewall [Zone Alarm]

URL Scanner [AVG, WOT, RG Guard]

Trend Micro Web Protection Add on

Disable Auto runs

Returnil Virtual System / Windows Steady State

Wehn-Trust HIPS [MUST for Windows XP – ASLR Tool]

Win-pooch HIPS [Windows XP]


WinPatrol [BillP Studios]

How anti malware works

How anti malware works?

Behaviors database

Behavior Based Engine

(On Process Activities)

Basic Activity Scanning *

Malware Signature database

Anti Virus Scanning

(On file content)

Known Applications database

White listing

(On process creation)

( * Process activity, file read or write )

Malware prevention system mps

Malware Prevention System (MPS)

Mps approach

MPS - Approach

Each application makes sequence of system calls for accessing various OS resources through multiple control paths (normal behaviour)

When the application is infected with malware, its behaviour changes

User Process n

User Process 1

User Process 2


User Space

System Calls

Kernel Space

Operating System

Detects malicious activity before it causes damage to end system i.e. before the system calls are executed by the operating system

Mps architecture

MPS - Architecture



Malware detection based on application behavior modeling

Malware Prevention System

1. Application Profiling and Model Generation Process in a Sandbox

4. Client

Protection against overall threats -

Process Execution Control

Model Enforcement Module

2. Server Manages the models and admin can set the policies here

Server communication module

3. Based on the policies the model gets pushed to clients


Malware detection based on application behavior modeling

Resource - A

System calls : {1,2,4}

Resource - B

System calls: {1,3,4,2}

Resource - C

System calls: {1,2,4}

Model Generation

Operations hooked in mps

Operations Hooked in MPS

File System Calls

Process hooks

Network Calls

Registry Calls

Deployment scenario

Deployment Scenario

System architecture

System Architecture

Database structure @ server

Database Structure @ Server

Database structure @ client

Database Structure @ Client

Index file @ server

Index File @ Server

Update request

Update Request

MPS Server

MPS Client



Major No,

Minor No,

OS type,


Db Major No,

Db Minor No

No.of Model Files,

Model File names,

ModelFile Path

File transfer request

File Transfer Request

MPS Server

MPS Client



Model File

Name with


Contents of

the Model File

Log message request

Log Message Request

Application name,

OS type,








Client and server technologies used

Client and Server – Technologies used

Server on Linux

Apache Server 2.2

Virtual Machine

Windows XP, Vista and 7 images

Linux 2.6.23 kernel image

Java runtime environment


HTTP message format


Windows Client

Mini Filter Driver

Call out Drivers

Win32 programming

C, C++ programming

PE Executable format Open SSL

Linux Client

Linux Security Modules

C, C++ programming

Qt Programming


Server gui

Server GUI

Malware detection based on application behavior modeling

Client GUI

Malicious pdf

Malicious Pdf

Creation of Axsle.dll

Creation of Icucnv34.dll

Write file on cvs.exe

The malware repeatedly tries to write cvs.exe file and it gets blocked. The document doesn’t open until the write file operation on cvs.exe is completed.

Malware detection based on application behavior modeling

Malicious Pdf

Malware detection based on application behavior modeling


  • Behaviors Detected

    • Hides view of system files

    • Hidden image file

    • File has system attribute

    • Creates logon entry

    • Unsigned binary

    • Drops executable

    • Modifies internet settings

    • Spawns process

Malware detection based on application behavior modeling


Malware detection based on application behavior modeling


Att27390 doc file

ATT27390 doc file

  • Activities blocked

    • Dropping of zipfldr.dll in system32 folder

    • Dropping of wuaueng.dll in system32 folder

Field testing report

Field Testing Report

  • MPS is compared with similar best commercial tools available in the market like NovaShield, Mamutu, Malware Defender, Sana Security Primary Response, Safe Connect, Threat fire etc.

Field testing report1

Field Testing Report

  • MPS is found sensitive against blended MS office and PDF documents wherein the MPS solution alone identified the malicious activity as the other industry product remain silent

  • Application has a tendency to raise false alarm against benign documents as it might match the enforcement policies defined

  • Overall it is felt that the solution is detecting high level targeted malware behaviours, but there is a need to improve the capabilities by suppressing the false alarms.

Malware resist simplifying and strengthening security

Malware ResistSimplifying and Strengthening Security

  • Detection Based on Runtime Behaviour. All running programs are monitored for a set of critical behaviors that could affect the normal functioning

Salient Features

Detection Based on Runtime Behavior

Small memory footprint and high detection rate

Co-exists with Anti Virus Solutions

Low False Positive Rate

Easy to Deploy and Use

Malware prevention system mps1

Malware Prevention System (MPS)

Ongoing research @ c dac hyderabad

Ongoing Research @ C-DAC Hyderabad

Design and Development of Anti Malware Solution for Web Applications and Mobiles

Malware analysis

Malware Analysis

The approach to analyze the malware

The approach to analyze the Malware

Run the malware in isolated lab

Monitor network and system connections

Understand the program’s code

Repeat until satisfied with gathered info

How to

How to?

  • Manual

    • Dedicated system (ready to be compromised)

    • Virtualized System

  • Automated Analysis

Automated analysis

Automated Analysis

Anubis [analyzing unknown binaries]


Virus total [analyze suspicious file]


Bit-Blaze [Malware Analysis Service]


Norman Sandbox

Joe Box Sandbox

Sunbelt CWSandBox

Comodo [Comodo Instant Malware Analysis]


Two steps phases

Two Steps / Phases

Behavioral (Dynamic) Analysis

Code (Static) Analysis

Gather as much as from behavioral analysis

Fill the gaps from the code analysis



Malware analysis1

Malware Analysis

To analyze malware, we requires basic and advanced knowledge in Windows and Linux concepts (depends)

For example: while doing behavioral analysis of the malware, we find malware modifies file A. – To get more out of it, we must know what is the significance of file A

Prepare the system

Prepare the System

  • Use VMWare and use the snapshot feature to restore state after malware execution

  • Use Virtual PC – execute the malware – Close and Delete changes

  • Physical System State Restore

    • Returnil Virtual System

    • Windows Steady State

Behavioral analysis

Behavioral Analysis

  • Activate various monitoring tools

  • Execute the malware

  • Terminate / suspend the malware process

    • Sometimes malware process comes again and again

  • Observe the results of monitoring tools

Process explorer

Process Explorer

  • Free from Microsoft TechNet

  • Super Task Manager

  • Shows process tree

    • We can know if malware created the new processes

  • Also shows files which a process is using

  • Can see the strings also

Process monitor

Process Monitor

  • Free from Microsoft TechNet

  • Monitors the following activities

    • Process creation

    • File related

    • Registry

    • Network related

  • Captures for all the process

    • Best is to do it for all and then apply the filters



Using idapro

Using IDAPro

Can reveal a lot of information

Great tool if user can reverse the C/C++ code

Use ollydbg

Use OllyDbg

  • OllyDbg is a great debugger

  • Open the sample using OllyDbg



  • Either use snort in a separate virtual machine to monitor its network activity

  • Or use tools like wire shark

  • Find

    • IRC server to whom this sample connects

    • Web servers?

  • May notice DNS queries

Packed malicious executables

Packed Malicious Executables

  • Packers compress / encrypt the executable

  • This is used

    • Difficult to analyze

    • Smaller size on hard disk

  • However runs unpacked and original in memory

How it executes

How it executes?

Small Decryptor extracts

the packed code and

executes the code

Executable Decryptor

Unpacked program in memory

Packed program stored as data

Pe format

PE Format


MS-DOS Stub Program









If it is packed

If it is packed



MS-DOS Stub Program

This is Decryptor code

MS-DOS Stub Program










Original PE


Packers availiable

Packers Availiable






Malware detection based on application behavior modeling


Process dumping with lordpe

Process dumping with LordPE

  • LordPE shows all the processes and can dump there images from memory

  • We can run the process from packed executable

    • Anyways it has to unpack itself in the memory

  • We can dump from memory using LordPE

Thank you

Thank You

  • Login