An overview of malware packing
1 / 88

An Overview of Malware Packing - PowerPoint PPT Presentation

  • Uploaded on

An Overview of Malware Packing. Jenna Kallaher Costas Akrivoulis Raul Gonzalez. Packers. "Programs that transform an input binary's appearance without affecting its execution semantics" [1]. Packers.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' An Overview of Malware Packing' - brie

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
An overview of malware packing

An Overview of Malware Packing

Jenna Kallaher

Costas Akrivoulis

Raul Gonzalez


"Programs that transform an input binary's appearance without affecting its execution semantics"



"Programs that transform an input binary's appearance without affecting its execution semantics"

... For the purposes of


Packers motivation
Packers - Motivation

  • Frustrate reverse engineer!

  • Increase barrier to entry for analysis

    • Cost, Technical understanding

  • Avoid signatures

  • Avoid classifications/triage


Era 3

Block-based Packing

Instruction Virtualization

Era 2




Era 1



Era 1

Era 1

Alpha Pack



Alpha Pack

Encryption motivation
Encryption - Motivation


Evade static signature


  • Encrypt the binary

  • Different keys == different signatures


What one thinks about when they hear "packer"

Packer encrypts all of the data and/or text,

then decrypts at runtime

Encryption key generation
Encryption – Key Generation

  • Embed key

  • Fetch key

  • Derive key

Encryption embed key
Encryption – Embed Key

B = Original Binary

K = Encryption Key

E = Encryption Function

Be = Ek(B)

D = Decryption Code

Malware = Be | K | D

Encryption fetch key
Encryption - Fetch Key

Decryptor code "phones home”

  • Retrieves key from master

    Master can authenticate malware

  • Verify environment

  • Attestation

Encryption derive key
Encryption - Derive Key

No phones

  • Evade attribution

  • No network constraints


  • Wrong environment == wrong key

  • Encryption limitations
    Encryption - Limitations

    Requires decryption before run

    • Decryptor can be signatured

      Encrypt the decryptor?

    • Need decryptor' for the decryptor...

      Some stub code needs to be "plain text"



    Alpha Pack

    Polymorphism motivation
    Polymorphism - Motivation


    Make every signature unique


    Mutate the encrypted binary

    • Re-key

    • Obfuscate decryptor code

    Polymorphism techniques
    Polymorphism - Techniques

    • Dead Code Insertion

    • Variable Renaming

    • Register Reordering

    • Code Reordering

    • Code Transposition

    • Instruction Substitution

    • Branch Inversion

    • ...

    Polymorphism dead code
    Polymorphism - Dead Code

    pop ebx

    pop ecx

    push .haha


    db 0xab

    db 0xbc

    db 0xcdff

    db 0xe8

    db 0xa9


    xor eax, eax

    mov ecx, 1


    xchg eax, ecx

    push edx

    pop edx


    Polymorphism limitations
    Polymorphism - Limitations

    Unencrypted binary is constant

    • Evades static signatures


    • Run malware, freeze upon decryption

    • Memory dump can be signatured

      Not robust enough

    Era 2

    Era 2

    Alpha Pack



    Alpha Pack

    Metamorphism motivation
    Metamorphism - Motivation


    Make every signature unique


    Mutate the plaintext binary

    • Physically different

    • Semantically equivalent

    Metamorphism techniques
    Metamorphism - Techniques

    • Dead Code Insertion

    • Variable Renaming

    • Register Reordering

    • Code Reordering

    • Code Transposition

    • Instruction Substitution

    • Branch Inversion

    • ...

    Metamorphism substitution

    mov $0, %eax

    test %eax, %eax

    jz .hello

    Metamorphism - Substitution

    • A * 4 == A << 2

    • jmp .hello ==

    Metamorphism branch inversion
    Metamorphism - Branch Inversion

    if (false)




    if (true) ==




    Metamorphism limitations
    Metamorphism - Limitations

    Hidden Markov Models:

    System being modeled is assumed to be a Markov Process with hidden states.

    "State is not directly visible, but output, dependant on the state, is visible.“


    Metamorphism limitations1
    Metamorphism - Limitations

    Hidden Markov Models:

    Use output, to reason about the properties of the hidden states in the system.

    Allows for Statistical Pattern Analysis

    Metamorphism limitations2
    Metamorphism - Limitations

    • Treat variants of a single binary as a family

    • Train a model, use HMM to represent the statistical properties of that family

    • Use trained model to decide whether a new variant belongs to the family or not - based on its statistical properties

    Metamorphism limitations3
    Metamorphism - Limitations


    Metamorphism limitations4
    Metamorphism - Limitations

    Similarity Index:

    Pair program P with malware m, of family M

    Find their similarity index, Spm

    If somewhat similar,

    • Compare mwith its variants to find average similarity index, Sm, amongst family M

      If Sm < Spm, then P is in M

    Metamorphism limitations5
    Metamorphism - Limitations

    Detectable, variants are:

    • Too similar to each other

    • Too different from normal programs

      Good metamorphism requires high degree of:

    • Metamorphic variant differences

    • Regular programs similarities

      Defensive techniques exist (and work)

    Anti reversing


    Alpha Pack

    Anti re motivation
    Anti-RE - Motivation

    Enemies of malware:

    • Virtual Machines

    • Debuggers

    • Disassemblers

    • Emulators

    • Memory Dumpers

      Any of the above imply:

    • “Non-production" settings

    • Reverse engineering likely occurring

    Anti re motivation1
    Anti-RE - Motivation


    Run ONLY on production systems


    Find metrics for answering:

    • "Am I in a VM?"

    • "Am I in a debugger?"

    • "Am I being disassembled?"

    • "Am I being emulated?"

    • "Am I being dumped?"

    Anti disassembly


    Alpha Pack

    Anti disassembly push ret
    Anti-Disassembly - Push Ret



    pop eax

    pop ebx

    pop ecx

    push .continue


    db 0xab

    db 0xbc

    db 0xcd

    db 0xff

    db 0xaa

    db 0x1

    db 0x2

    db 0xff

    db 0xef

    db 0xe9

    db 0xab

    db 0xe8

    db 0xa9


    xor eax,eax

    mov ecx,1


    xchg eax,ecx

    push edx

    pop edx


    .end start

    • [4]

    Anti disassembly push pop math
    Anti-Disassembly - Push Pop Math

    • Push known immediate to stack

    • Immediately pop to a register

    • Do some math on the register

    • Register holds desired value, but that value never appears directly in the code

    Anti virtualization techniques
    Anti-Virtualization - Techniques

    Common x86 Instructions for VM Detection

    • cpuid

    • sidt (Red Pill)

    • sldt

    • sgdt

    • smsw

    • str

    • in

      System Artifacts

      Invalid Instructions

    Anti virtualization in
    Anti-Virtualization - in

    #define VMWARE_HYPERVISOR_MAGIC 0x564D5868

    #define VMWARE_HYPERVISOR_PORT 0x5658


    #define VMWARE_PORT(cmd, eax, ebx, ecx, edx) \

    __asm__("inl (%%dx)" : \

    "=a"(eax), "=c"(ecx), "=d"(edx), "=b"(ebx) : \


    "1"(VMWARE_PORT_CMD_##cmd), \



    int hypervisor_port_check(void) {

    uint32_t eax, ebx, ecx, edx;

    VMWARE_PORT(GETVERSION, eax, ebx, ecx, edx);

    int in_hyp = (ebx == VMWARE_HYPERVISOR_MAGIC);

    printf(“%sVMWare\n”, in_hyp ? “”: “Not ”);

    return in_hyp;


    • [6]

    Anti virtualization cpuid
    Anti-Virtualization - CPUID

    mov $0x1, %eax


    and $0x80000000, %ecx

    test %ecx, %ecx

    jnz virtualized

    mov $0x0, %eax



    mov $0x1, %eax


    Anti virtualization artifacts
    Anti-Virtualization - Artifacts


    • VMWare Tools Process

    • 300+ registry references to VMWare in WinXP

    • 50+ file system references to VMWare or VMX in WinXP

      Virtual Hardware

    • MAC Address

    • USB Controller Type

      Anomalies in clock synch between host and guest

    • [7]

    Anti debugging


    Alpha Pack

    Anti debugging techniques
    Anti-Debugging - Techniques

    • Windows

      • IsDebuggerPresent

      • CheckIsRemoteDebuggerPresent

      • NtQueryProcessInformation

      • PEB

    • Linux

      • Ptrace of Self

    Anti debugging techniques1
    Anti-Debugging - Techniques

    • Windows + Linux

      • Rogue 0xCC/0x2D/0xF1

      • 0xCC Scanning

      • popf and trap

      • SS Register

      • Signal Handling

    Anti debugging 0xf1
    Anti-Debugging - 0xF1

    static void catch_function(int signal) {

    puts(“Signal caught.”);


    int main(void) {

    if (signal(SIGTRAP, catch_function) == SIG_ERR) {

    fputs(“An error occurred! \n”, stderr);

    return EXIT_FAILURE;


    puts(“Ice breakpoint added.”);

    asm __volatile__ (.byte 0xf1);

    puts(“This will never print if this program is being debugged.”);

    return 0;


    Anti debugging rdtsc
    Anti-Debugging - RDTSC

    if(dbg() == 0)

    puts("No Debugger");



    Int dbg(){

    asm volatile("rdtsc\n\t"

    "mov %%eax,%%ecx\n\t"


        "sub %%ecx,%%eax\n\t"

        "cmp $0xfff,%%eax\n\t"

        "jae debugged\n\t"

        "mov $0x0,%%eax\n\t"

        "jmp done\n\t"


        "mov    $0x1,%%eax\n\t"

        "jmp done\n\t"


        "mov %%eax,%0\n\t"

        : "=g" (debugged)

        :: "eax");


    Anti emulation


    Alpha Pack

    Anti emulation1

    • Emulators aren't perfect

    • Some instructions aren't handled correctly and the differences can be used to detect emulators

    • Finding these differences is oftentimes done by fuzzing

    Anti dumping


    Alpha Pack

    Anti dumping1

    • Nanomites

    • SizeOfImage

    • Guard Pages

    • Removing the PE Header

    Anti dumping nanomites
    Anti-Dumping - Nanomites

    Used by Armadillo

    Replaces branch instructions with 0xCCs

    Stores all jump information in an encrypted table

    • Jump Destination

    • Flags

      Uses self-debugging to handle 0xCCs:

    • Determines if it is a debug or nanomite breakpoint

    • Skips to jump destination if the jump should be taken

    Anti dumping sizeofimage
    Anti-Dumping - SizeOfImage

    // Any unreasonably large value will work say for example

    // 0x100000 or 100,000h

    void ChangeSizeOfImage(DWORD NewSize)


    __asm {

    mov eax, fs:[0x30] // PEB

    mov eax, [eax + 0x0c] // PEB_LDR_DATA

    mov eax, [eax + 0x0c] // InOrderModuleList

    mov dword ptr [eax + 0x20],NewSize // SizeOfImage



    • [8]

    Anti dumping guard pages
    Anti-Dumping - Guard Pages

    • Not all code is decrypted immediately

    • These pages are replaced by guard pages

    • Guard pages generate exceptions

    • Upon exception, relevant code is decrypted

    Anti dumping remove pe header
    Anti-Dumping - Remove PE Header

    // This function will erase the current images

    // PE header from memory preventing a successful image

    // if dumped

    inline void ErasePEHeaderFromMemory()

    {DWORD OldProtect = 0;// Get base address of module

    char *pBaseAddr = (char *)GetModuleHandle(NULL);

    // Change memory protection (assume x86 page size)

    VirtualProtect(pBaseAddr, 4096, PAGE_READWRITE, &OldProtect);

    // Erase the header

    ZeroMemory(pBaseAddr, 4096);


    • [8]

    Anti re limitations
    Anti-RE - Limitations

    VM assumptions no longer hold

    • Amazon EC2

      Defenders can hook specific "anti" checks

    • No general techniques...

      Endless cat and mouse game

    Era 3

    Era 3

    Alpha Pack

    Era 3 motivation
    Era 3 - Motivation

    Past techniques pack payload statically

    Reverse Engineers

    • Assume payload will at some point be present and unprotected

    • Wait until payload unpacks => analysis

      Era 3 packers violate this assumption

    • Never present entirety of code

    • Must now RE payload before applying SA

    Block based packing motivation
    Block-based Packing - Motivation


    Entire payload should never be present in memory at any given moment


    Unpack small block of malware at a time

    • Re-pack previous blocks

      Only small code block inspectable by AV

    • not enough to meaningfully signature

    Block based packing limitations
    Block-based Packing - Limitations


    • Smaller blocks need to be (re-)packed more often

      • code that loops across block boundaries

        Binary disassembly can be difficult[BIRD paper]

        Not all binary present for scan at once

    • ... but can save incrementally unpacked code

    Instruction virtualization motivation
    Instruction Virtualization - Motivation


    Obfuscate malware through emulation


    • Translate input binary into bytecode

    • Packed binary := emulator + bytecode payload

    • At runtime, emulator interprets bytecode

      Themida, VMProtect, Code Virtualizer

    Instruction virtualization process
    Instruction Virtualization - Process

    • Construct virtual instruction set arch (V-ISA)

      • (compatible with x86 for what payload requires)

    • Take input binary and translate x86 instructions to bytecode for V-ISA

    • Generate emulator (virt. processor) for V-ISA

      • "implementation" of ISA

    Opportunities to frustrate res
    Opportunities to Frustrate REs

    • Construct virtual instruction set arch (V-ISA)

      • Compatible ISA => randomize opcodes

      • Superfluous ISA features

        • Duplicate instructions

        • Unused instruction parameters

    • Generate emulator (virt. processor) for V-ISA

      • Can implement an ISA in many different ways

        • Put on your Computer Architect hats...

        • Single-cycle, pipelined, OoO, VLIW, dataflow

          Generated signatures are meaningless

    Instruction virtualization detection
    Instruction Virtualization - Detection

    Even if REs suspect that an emulator is used:

    Must detect all 3 parts of the process

    • Virtual ISA (semantics, syntax)

    • Bytecode (location)

    • Emulator (behavior)

      Remember: packed binary is a black box!

    Instruction virtualization rotalum
    Instruction Virtualization - Rotalumé

    • Emulator (behavior)

      • Dynamic instruction tracing

      • Abstract variable binding (forward, backward)

      • Read clustering

    • Bytecode (location)

      • Cross-testing clusters to determine VPC

    • Virtual ISA (semantics, syntax)

      • Dynamic taint analysis on read instructions

    Instruction virtualization limitations
    Instruction Virtualization - Limitations

    • Hard to identify emulator

      • Heuristics employed to detect emulator

        • Traditional decode/dispatch architecture

        • Abstract variable lifetimes

    • Hard to identify V-ISA fully (semantics,syntax)

      • Bytecode does not traverse all emulator paths

    • Performance

    Instruction virtualization limitations1
    Instruction Virtualization - Limitations

    Goal (as the unpacker)

    • Determine original, unpacked malware

    • Perform traditional static analysis

      Can't identify + understand V-ISA

      => Can't convert bytecode to native instructions

      => Can't analyze malware with static techniques

    Instruction virtualization thoughts
    Instruction Virtualization - Thoughts

    "We argue that this trend will continue and that a large portion of malware in the near future will be emulation based."

    "There is no existing technique that can reliably counter an emulation-based obfuscation technique."



    Alpha Pack

    Masquerading motivation
    Masquerading - Motivation


    • Mislead malware triage

    • Confuse Reverse Engineers


      Mimic statistical features of a different packer

    Masquerading techniques
    Masquerading - Techniques

    • Signature Injection

    • Statistical Feature Spoofing

    Masquerading signature injection
    Masquerading - Signature Injection

    Majority of packer signatures are regex

    Inject others' signatures in our malware

    [Armadillo 3.00a -> Silicon Realms Toolworks]

    signature =

    60 E8 00 00 00 00 5D 50 51 EB 0F ?? EB 0F ?? EB 07 ?? EB 0F ?? EB 08 FD EB 0B F2 EB F5 EB F6 F2 EB 08 FD EB E9 F3 EB E4 FC ?? 59 58 50 51 EB 0F ?? EB 0F ?? EB 07 ?? EB 0F ?? EB 08 FD EB 0B F2 EB F5 EB F6 F2 EB 08 FD EB E9 F3 EB E4 FC ?? 59 58 50 51 EB 0F

    ep_only = true

    • [10]

    Masquerading feature spoofing
    Masquerading - Feature Spoofing

    Use Instruction Virtualization

    as previously described

    DO NOT generate random ISA

    DO generate ISA that's statistically similar to the output of other packers.

    Future techniques
    Future Techniques

    • What's next?

    What it looks like

    What It Looks like

    Alpha Pack


    • Guo, Fanglu, Peter Ferrie, and Tzi-Cker Chiueh. "A study of the packer problem and its solutions." Recent Advances in Intrusion Detection. Springer Berlin Heidelberg, 2008.

    • Rabiner, Lawrence R. "A tutorial on hidden Markov models and selected applications in speech recognition." Proceedings of the IEEE 77.2 (1989): 257-286.

    • Wong, Wing, and Mark Stamp. "Hunting for metamorphic engines." Journal in Computer Virology 2.3 (2006): 211-229.

    • Branco, Rodrigo Rubira, Gabriel Negreira Barbosa, and Pedro Drimel Neto. "Scientific but Not Academical Overview of Malware Anti-Debugging, Anti-Disassembly and Anti-VM Technologies."

    • Garfinkel, Tal, et al. "Compatibility is not transparency: VMM detection myths and realities." Proceedings of the 11th Workshop on Hot Topics in Operating Systems (HotOS-XI). 2007.




    • Sharif, Monirul, et al. "Automatic reverse engineering of malware emulators."Security and Privacy, 2009 30th IEEE Symposium on. IEEE, 2009.