An overview of malware packing
This presentation is the property of its rightful owner.
Sponsored Links
1 / 88

An Overview of Malware Packing PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

An Overview of Malware Packing. Jenna Kallaher Costas Akrivoulis Raul Gonzalez. Packers. "Programs that transform an input binary's appearance without affecting its execution semantics" [1]. Packers.

Download Presentation

An Overview of Malware Packing

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

An overview of malware packing

An Overview of Malware Packing

Jenna Kallaher

Costas Akrivoulis

Raul Gonzalez



"Programs that transform an input binary's appearance without affecting its execution semantics"




"Programs that transform an input binary's appearance without affecting its execution semantics"

... For the purposes of


Packers motivation

Packers - Motivation

  • Frustrate reverse engineer!

  • Increase barrier to entry for analysis

    • Cost, Technical understanding

  • Avoid signatures

  • Avoid classifications/triage


Era 3

Block-based Packing

Instruction Virtualization

Era 2




Era 1



Era 1

Era 1

Alpha Pack



Alpha Pack

Encryption motivation

Encryption - Motivation


Evade static signature


  • Encrypt the binary

  • Different keys == different signatures



What one thinks about when they hear "packer"

Packer encrypts all of the data and/or text,

then decrypts at runtime

Encryption key generation

Encryption – Key Generation

  • Embed key

  • Fetch key

  • Derive key

Encryption embed key

Encryption – Embed Key

B = Original Binary

K = Encryption Key

E = Encryption Function

Be = Ek(B)

D = Decryption Code

Malware = Be | K | D

Encryption fetch key

Encryption - Fetch Key

Decryptor code "phones home”

  • Retrieves key from master

    Master can authenticate malware

  • Verify environment

  • Attestation

Encryption derive key

Encryption - Derive Key

No phones

  • Evade attribution

  • No network constraints


  • Wrong environment == wrong key

  • Encryption limitations

    Encryption - Limitations

    Requires decryption before run

    • Decryptor can be signatured

      Encrypt the decryptor?

    • Need decryptor' for the decryptor...

      Some stub code needs to be "plain text"



    Alpha Pack

    Polymorphism motivation

    Polymorphism - Motivation


    Make every signature unique


    Mutate the encrypted binary

    • Re-key

    • Obfuscate decryptor code

    Polymorphism techniques

    Polymorphism - Techniques

    • Dead Code Insertion

    • Variable Renaming

    • Register Reordering

    • Code Reordering

    • Code Transposition

    • Instruction Substitution

    • Branch Inversion

    • ...

    Polymorphism dead code

    Polymorphism - Dead Code











    xor eax, eax

    mov ecx, 1


    xchg eax, ecx

    push edx

    pop edx


    Polymorphism register reorder

    Polymorphism - Register Reorder

    Polymorphism limitations

    Polymorphism - Limitations

    Unencrypted binary is constant

    • Evades static signatures


    • Run malware, freeze upon decryption

    • Memory dump can be signatured

      Not robust enough

    Era 2

    Era 2

    Alpha Pack



    Alpha Pack

    Metamorphism motivation

    Metamorphism - Motivation


    Make every signature unique


    Mutate the plaintext binary

    • Physically different

    • Semantically equivalent

    Metamorphism techniques

    Metamorphism - Techniques

    • Dead Code Insertion

    • Variable Renaming

    • Register Reordering

    • Code Reordering

    • Code Transposition

    • Instruction Substitution

    • Branch Inversion

    • ...

    Metamorphism code reorder

    Metamorphism - Code Reorder

    Metamorphism substitution

    mov $0, %eax

    test %eax, %eax

    jz .hello

    Metamorphism - Substitution

    • A * 4 == A << 2

    • jmp .hello ==

    Metamorphism branch inversion

    Metamorphism - Branch Inversion

    if (false)




    if (true) ==




    Metamorphism limitations

    Metamorphism - Limitations

    Hidden Markov Models:

    System being modeled is assumed to be a Markov Process with hidden states.

    "State is not directly visible, but output, dependant on the state, is visible.“


    Metamorphism limitations1

    Metamorphism - Limitations

    Hidden Markov Models:

    Use output, to reason about the properties of the hidden states in the system.

    Allows for Statistical Pattern Analysis

    Metamorphism limitations2

    Metamorphism - Limitations

    • Treat variants of a single binary as a family

    • Train a model, use HMM to represent the statistical properties of that family

    • Use trained model to decide whether a new variant belongs to the family or not - based on its statistical properties

    Metamorphism limitations3

    Metamorphism - Limitations


    Metamorphism limitations4

    Metamorphism - Limitations

    Similarity Index:

    Pair program P with malware m, of family M

    Find their similarity index, Spm

    If somewhat similar,

    • Compare mwith its variants to find average similarity index, Sm, amongst family M

      If Sm < Spm, then P is in M

    Metamorphism limitations5

    Metamorphism - Limitations

    Detectable, variants are:

    • Too similar to each other

    • Too different from normal programs

      Good metamorphism requires high degree of:

    • Metamorphic variant differences

    • Regular programs similarities

      Defensive techniques exist (and work)

    Anti reversing


    Alpha Pack

    Anti re motivation

    Anti-RE - Motivation

    Enemies of malware:

    • Virtual Machines

    • Debuggers

    • Disassemblers

    • Emulators

    • Memory Dumpers

      Any of the above imply:

    • “Non-production" settings

    • Reverse engineering likely occurring

    Anti re motivation1

    Anti-RE - Motivation


    Run ONLY on production systems


    Find metrics for answering:

    • "Am I in a VM?"

    • "Am I in a debugger?"

    • "Am I being disassembled?"

    • "Am I being emulated?"

    • "Am I being dumped?"

    Anti re in malware

    Anti-RE - in Malware

    • [4]

    Anti re techniques

    Anti-RE - Techniques

    • [4]

    Anti disassembly


    Alpha Pack

    Anti disassembly prevalence

    Anti-Disassembly - Prevalence

    • [4]

    Anti disassembly push ret

    Anti-Disassembly - Push Ret





























    .end start

    • [4]

    Anti disassembly push pop math

    Anti-Disassembly - Push Pop Math

    • Push known immediate to stack

    • Immediately pop to a register

    • Do some math on the register

    • Register holds desired value, but that value never appears directly in the code

    Anti virtualization


    Alpha Pack

    Anti virtualization techniques

    Anti-Virtualization - Techniques

    Common x86 Instructions for VM Detection

    • cpuid

    • sidt (Red Pill)

    • sldt

    • sgdt

    • smsw

    • str

    • in

      System Artifacts

      Invalid Instructions

    Anti virtualization prevalence

    Anti-Virtualization - Prevalence

    • [4]

    Anti virtualization in

    Anti-Virtualization - in

    #define VMWARE_HYPERVISOR_MAGIC 0x564D5868

    #define VMWARE_HYPERVISOR_PORT 0x5658


    #define VMWARE_PORT(cmd, eax, ebx, ecx, edx) \

    __asm__("inl (%%dx)" : \

    "=a"(eax), "=c"(ecx), "=d"(edx), "=b"(ebx) :\


    "1"(VMWARE_PORT_CMD_##cmd), \



    int hypervisor_port_check(void) {

    uint32_t eax, ebx, ecx, edx;

    VMWARE_PORT(GETVERSION, eax, ebx, ecx, edx);

    int in_hyp = (ebx == VMWARE_HYPERVISOR_MAGIC);

    printf(“%sVMWare\n”, in_hyp ? “”: “Not ”);

    return in_hyp;


    • [6]

    Anti virtualization cpuid

    Anti-Virtualization - CPUID

    mov $0x1, %eax


    and $0x80000000, %ecx

    test %ecx, %ecx

    jnz virtualized

    mov $0x0, %eax



    mov $0x1, %eax


    Anti virtualization artifacts

    Anti-Virtualization - Artifacts


    • VMWare Tools Process

    • 300+ registry references to VMWare in WinXP

    • 50+ file system references to VMWare or VMX in WinXP

      Virtual Hardware

    • MAC Address

    • USB Controller Type

      Anomalies in clock synch between host and guest

    • [7]

    Anti debugging


    Alpha Pack

    Anti debugging techniques

    Anti-Debugging - Techniques

    • Windows

      • IsDebuggerPresent

      • CheckIsRemoteDebuggerPresent

      • NtQueryProcessInformation

      • PEB

    • Linux

      • Ptrace of Self

    Anti debugging techniques1

    Anti-Debugging - Techniques

    • Windows + Linux

      • Rogue 0xCC/0x2D/0xF1

      • 0xCC Scanning

      • popf and trap

      • SS Register

      • Signal Handling

    Anti debugging prevalence

    Anti-Debugging - Prevalence

    • [4]

    Anti debugging 0xf1

    Anti-Debugging - 0xF1

    static void catch_function(int signal) {

    puts(“Signal caught.”);


    int main(void) {

    if (signal(SIGTRAP, catch_function) == SIG_ERR) {

    fputs(“An error occurred! \n”, stderr);

    return EXIT_FAILURE;


    puts(“Ice breakpoint added.”);

    asm __volatile__ (.byte 0xf1);

    puts(“This will never print if this program is being debugged.”);

    return 0;


    Anti debugging rdtsc

    Anti-Debugging - RDTSC

    if(dbg() == 0)

    puts("No Debugger");



    Int dbg(){

    asm volatile("rdtsc\n\t"

    "mov %%eax,%%ecx\n\t"


        "sub %%ecx,%%eax\n\t"

        "cmp $0xfff,%%eax\n\t"

        "jae debugged\n\t"

        "mov $0x0,%%eax\n\t"

        "jmp done\n\t"


        "mov    $0x1,%%eax\n\t"

        "jmp done\n\t"


        "mov %%eax,%0\n\t"

        : "=g" (debugged)

        :: "eax");


    Anti emulation


    Alpha Pack

    Anti emulation1


    • Emulators aren't perfect

    • Some instructions aren't handled correctly and the differences can be used to detect emulators

    • Finding these differences is oftentimes done by fuzzing

    Anti dumping


    Alpha Pack

    Anti dumping1


    • Nanomites

    • SizeOfImage

    • Guard Pages

    • Removing the PE Header

    Anti dumping nanomites

    Anti-Dumping - Nanomites

    Used by Armadillo

    Replaces branch instructions with 0xCCs

    Stores all jump information in an encrypted table

    • Jump Destination

    • Flags

      Uses self-debugging to handle 0xCCs:

    • Determines if it is a debug or nanomite breakpoint

    • Skips to jump destination if the jump should be taken

    Anti dumping sizeofimage

    Anti-Dumping - SizeOfImage

    // Any unreasonably large value will work say for example

    // 0x100000 or 100,000h

    void ChangeSizeOfImage(DWORD NewSize)


    __asm {

    mov eax, fs:[0x30] // PEB

    mov eax, [eax + 0x0c] // PEB_LDR_DATA

    mov eax, [eax + 0x0c] // InOrderModuleList

    mov dword ptr [eax + 0x20],NewSize // SizeOfImage



    • [8]

    Anti dumping guard pages

    Anti-Dumping - Guard Pages

    • Not all code is decrypted immediately

    • These pages are replaced by guard pages

    • Guard pages generate exceptions

    • Upon exception, relevant code is decrypted

    Anti dumping remove pe header

    Anti-Dumping - Remove PE Header

    // This function will erase the current images

    // PE header from memory preventing a successful image

    // if dumped

    inline void ErasePEHeaderFromMemory()

    {DWORD OldProtect = 0;// Get base address of module

    char *pBaseAddr = (char *)GetModuleHandle(NULL);

    // Change memory protection (assume x86 page size)

    VirtualProtect(pBaseAddr, 4096, PAGE_READWRITE, &OldProtect);

    // Erase the header

    ZeroMemory(pBaseAddr, 4096);


    • [8]

    Anti re limitations

    Anti-RE - Limitations

    VM assumptions no longer hold

    • Amazon EC2

      Defenders can hook specific "anti" checks

    • No general techniques...

      Endless cat and mouse game

    Era 3

    Era 3

    Alpha Pack

    Era 3 motivation

    Era 3 - Motivation

    Past techniques pack payload statically

    Reverse Engineers

    • Assume payload will at some point be present and unprotected

    • Wait until payload unpacks => analysis

      Era 3 packers violate this assumption

    • Never present entirety of code

    • Must now RE payload before applying SA

    Block based packing

    Block-based Packing

    Alpha Pack

    Block based packing motivation

    Block-based Packing - Motivation


    Entire payload should never be present in memory at any given moment


    Unpack small block of malware at a time

    • Re-pack previous blocks

      Only small code block inspectable by AV

    • not enough to meaningfully signature

    Block based packing limitations

    Block-based Packing - Limitations


    • Smaller blocks need to be (re-)packed more often

      • code that loops across block boundaries

        Binary disassembly can be difficult[BIRD paper]

        Not all binary present for scan at once

    • ... but can save incrementally unpacked code

    Instruction virtualization

    Instruction Virtualization

    Alpha Pack

    Instruction virtualization motivation

    Instruction Virtualization - Motivation


    Obfuscate malware through emulation


    • Translate input binary into bytecode

    • Packed binary := emulator + bytecode payload

    • At runtime, emulator interprets bytecode

      Themida, VMProtect, Code Virtualizer

    Instruction virtualization process

    Instruction Virtualization - Process

    • Construct virtual instruction set arch (V-ISA)

      • (compatible with x86 for what payload requires)

    • Take input binary and translate x86 instructions to bytecode for V-ISA

    • Generate emulator (virt. processor) for V-ISA

      • "implementation" of ISA

    Instruction virtualization process1

    Instruction Virtualization - Process

    • [9]

    Opportunities to frustrate res

    Opportunities to Frustrate REs

    • Construct virtual instruction set arch (V-ISA)

      • Compatible ISA => randomize opcodes

      • Superfluous ISA features

        • Duplicate instructions

        • Unused instruction parameters

    • Generate emulator (virt. processor) for V-ISA

      • Can implement an ISA in many different ways

        • Put on your Computer Architect hats...

        • Single-cycle, pipelined, OoO, VLIW, dataflow

          Generated signatures are meaningless

    Instruction virtualization detection

    Instruction Virtualization - Detection

    Even if REs suspect that an emulator is used:

    Must detect all 3 parts of the process

    • Virtual ISA (semantics, syntax)

    • Bytecode (location)

    • Emulator (behavior)

      Remember: packed binary is a black box!

    Instruction virtualization rotalum

    Instruction Virtualization - Rotalumé

    • Emulator (behavior)

      • Dynamic instruction tracing

      • Abstract variable binding (forward, backward)

      • Read clustering

    • Bytecode (location)

      • Cross-testing clusters to determine VPC

    • Virtual ISA (semantics, syntax)

      • Dynamic taint analysis on read instructions

    Instruction virtualization limitations

    Instruction Virtualization - Limitations

    • Hard to identify emulator

      • Heuristics employed to detect emulator

        • Traditional decode/dispatch architecture

        • Abstract variable lifetimes

    • Hard to identify V-ISA fully (semantics,syntax)

      • Bytecode does not traverse all emulator paths

    • Performance

    Instruction virtualization limitations1

    Instruction Virtualization - Limitations

    Goal (as the unpacker)

    • Determine original, unpacked malware

    • Perform traditional static analysis

      Can't identify + understand V-ISA

      => Can't convert bytecode to native instructions

      => Can't analyze malware with static techniques

    Instruction virtualization thoughts

    Instruction Virtualization - Thoughts

    "We argue that this trend will continue and that a large portion of malware in the near future will be emulation based."

    "There is no existing technique that can reliably counter an emulation-based obfuscation technique."



    Alpha Pack

    Masquerading motivation

    Masquerading - Motivation


    • Mislead malware triage

    • Confuse Reverse Engineers


      Mimic statistical features of a different packer

    Masquerading techniques

    Masquerading - Techniques

    • Signature Injection

    • Statistical Feature Spoofing

    Masquerading signature injection

    Masquerading - Signature Injection

    Majority of packer signatures are regex

    Inject others' signatures in our malware

    [Armadillo 3.00a -> Silicon Realms Toolworks]

    signature =

    60 E8 00 00 00 00 5D 50 51 EB 0F ?? EB 0F ?? EB 07 ?? EB 0F ?? EB 08 FD EB 0B F2 EB F5 EB F6 F2 EB 08 FD EB E9 F3 EB E4 FC ?? 59 58 50 51 EB 0F ?? EB 0F ?? EB 07 ?? EB 0F ?? EB 08 FD EB 0B F2 EB F5 EB F6 F2 EB 08 FD EB E9 F3 EB E4 FC ?? 59 58 50 51 EB 0F

    ep_only = true

    • [10]

    Masquerading feature spoofing

    Masquerading - Feature Spoofing

    Use Instruction Virtualization

    as previously described

    DO NOT generate random ISA

    DO generate ISA that's statistically similar to the output of other packers.

    Future techniques

    Future Techniques

    • What's next?

    What it looks like

    What It Looks like

    Alpha Pack

    An overview of malware packing

    • [11]



    • Guo, Fanglu, Peter Ferrie, and Tzi-Cker Chiueh. "A study of the packer problem and its solutions." Recent Advances in Intrusion Detection. Springer Berlin Heidelberg, 2008.

    • Rabiner, Lawrence R. "A tutorial on hidden Markov models and selected applications in speech recognition." Proceedings of the IEEE 77.2 (1989): 257-286.

    • Wong, Wing, and Mark Stamp. "Hunting for metamorphic engines." Journal in Computer Virology 2.3 (2006): 211-229.

    • Branco, Rodrigo Rubira, Gabriel Negreira Barbosa, and Pedro Drimel Neto. "Scientific but Not Academical Overview of Malware Anti-Debugging, Anti-Disassembly and Anti-VM Technologies."

    • Garfinkel, Tal, et al. "Compatibility is not transparency: VMM detection myths and realities." Proceedings of the 11th Workshop on Hot Topics in Operating Systems (HotOS-XI). 2007.




    • Sharif, Monirul, et al. "Automatic reverse engineering of malware emulators."Security and Privacy, 2009 30th IEEE Symposium on. IEEE, 2009.



    An overview of malware packing


  • Login