Austin Norby

Date of Award

Spring 3-2024

Document Type

Dissertation

Degree Name

Doctor of Philosophy in Cyber Operations (PhDCO)

First Advisor

Tyler Flaagan

Second Advisor

Yong Wang

Third Advisor

Mark Spanier

Abstract

Anti-debugging techniques are often used in malware samples and software protection frameworks. From the malware perspective, the malware author includes these checks to hinder the analysis of the sample to increase the time to affect their target. Software protection frameworks on the other hand do this to protect sensitive information such as intellectual property, music, movies, and other proprietary information. The anti-debugging techniques serve to deter reverse engineers from compromising the sensitive data in this case. Modern anti-debugging research focuses on creating new techniques or defeating techniques during analysis. Additionally, modern anti-debugging research is paired with other anti-analysis techniques and used as a component in novel software protection frameworks. This dissertation seeks to better understand anti-debugging techniques in terms of CPU execution time and provide an artifact for anti-debugging identification within target executables based on performance data. This research will use a design science methodology to investigate performance data anti-debugging techniques and generate an artifact that can inform cybersecurity defenders to the presence of anti-debugging techniques based on performance data and statistical tests. The results of this research will be implementations of the anti-debugging techniques, many datasets representing the performance data of Anti-Debugging techniques, statistical results regarding the difference in performance of the anti-debugging techniques compared to a control group, and an artifact that can identify the presence of anti-debugging techniques by measuring the similarity of data variances in performance datasets. In conclusion, the generated artifact was able to detect 27 out of the 58 total techniques across operating systems and architectures with any degree of sensitivity. Out of those 27, seven were significantly less sensitive to changes in execution duration and detected every time in the artifact experiment.

Download

Share

COinS