Date of Award
Spring 3-2025
Document Type
Dissertation
Degree Name
Doctor of Philosophy in Cyber Operations (PhDCO)
First Advisor
Varghese Vaidyan
Second Advisor
Yong Wang
Third Advisor
Gurcan Comert
Abstract
Automated identification of malicious JavaScript is a core problem within modern malware analysis. Code obfuscation is a common tactic used in order to evade detection. Obfuscation hinders both manual and automated detection methods, including neural network techniques. In order for these methods to effectively classify malware, it is beneficial to reduce the effects of obfuscation as well as to optimize the configuration and structure of the neural network to be well suited for the task. To overcome these challenges, a new framework is presented: “PyRHOH”, a metalearning framework that implements Bayesian optimization. This framework adds structure and rigor to the selection of neural network hyperparameters, providing assurance that an optimal design has been implemented. This framework was used to determine optimal recurrent neural network architectures for the differentiation of malicious and benign JavaScript samples. These neural networks were then used to determine the degree to which using Google’s V8 JavaScript compiler to process raw JavaScript samples into compiled bytecode affected classification accuracy. Classifying in-the-wild samples, compilation was measured to increase the detection rate from 76.88% to 95.84%. When obfuscation was performed against the full data set, the detection rate increased from an average of 76.76% to an average of 91.24% once compilation was performed. This shows that pre-processing JavaScript into compiled bytecode has a clear positive impact on neural network categorization.
Recommended Citation
Fulkerson, Eli T., "The Effect of Compilation on the Identification of Obfuscated Malware" (2025). Masters Theses & Doctoral Dissertations. 483.
https://scholar.dsu.edu/theses/483