Date of Award

Spring 3-2022

Document Type

Dissertation

Degree Name

Doctor of Philosophy in Cyber Operations (PhDCO)

Department

Computer Science

First Advisor

Shengjie Xu

Second Advisor

Austin O'Brien

Third Advisor

Josh Stroschein

Abstract

This dissertation proposes several improvements to existing adversarial attacks against MalConv, a raw-byte malware classifier for Windows PE files. The included contributions greatly improve the success rates and performance of gradient-based file overlay attacks. All improvements are included in a new open-source attack utility called BitCamo.

Several new payload initialization strategies for use with gradient-based attacks are proposed and evaluated as potential replacements for the randomized initialization method used by current attacks. An algorithm for determining the optimal payload size is also proposed. The resulting improvements achieve a 100% evasion rate against eligible target executables using an average payload size of only 300 bytes. The results are substantially better than those reported by other open-source tools or attacks proposed within the research literature.

Existing gradient attacks against MalConv contain a long-running byte reconstruction phase necessary to map backwards across a non-differentiable embedding layer used by the model. Three proposals are presented to significantly improve the runtime of this phase, including the addition of parallelism, limiting the scope of reconstruction to the payload only, and introducing a K-D tree data structure to allow for blazing fast spatial searches in comparison to the L2 distance metric used by current attacks.

A pre-detection mechanism proposed in previous research checks if executables have the same code section hash but a different overall hash with respect to known malicious files, allowing adversarial examples to be immediately rejected by a detection pipeline before MalConv evaluates the sample. This dissertation proposes a single-byte code section attack that can completely bypass this defense mechanism in over 63% of samples. The pre-detection attack can be used in conjunction with the other new improvements to offer a formidable attack capability against MalConv and other detection models sharing a similar architecture.

Download

Share

COinS