Tjada Nelson

Date of Award

Spring 2025

Document Type

Dissertation

Degree Name

Doctor of Philosophy in Cyber Operations (PhDCO)

First Advisor

Austin O'Brien

Second Advisor

Cherie Noteboom

Third Advisor

Shengji Xu

Abstract

Malware remains a primary cybersecurity threat because traditional signature-based detection methods have difficulty matching the pace of evolving malicious code implementing complex obfuscation methods to evade detection. The current detection methods fail to identify new malware variations, which expose systems to damaging cyber attacks that result in major security breaches and delayed incident response capabilities. This research investigates the speed-performance gap between signature-based detection and control flow graph behavioral understanding through developing Weighted Control Flow Graph (WCFG), which merges structural program analysis with signature-based detection capabilities. The design science research method enabled this study to create two main research outcomes: a complete WCFG dataset and an enhanced machine learning-based malware detection system. The research methodology encompasses a PE feature extraction from the PEMML dataset, followed by CAPA signature matching for identifying malicious functions, then generating control flow graphs through Radare2 before combining data with signature-based weights. The processed dataset consisted of samples divided between five malware families, including Zbot, Locker, Mediyes, Winwebsec, and ZeroAccess, as well as benign software, which underwent XGBoost classification with SMOTE Tomek balancing and Random Forest feature selection. The WCFG method reached 90% accuracy in classification, which outperformed the unweighted control flow graph-only method by 4% because it achieved 86% accuracy. The weighted model demonstrated better performance with precision and recall for malicious and benign samples. The SHAP analysis established that signature-based weighting features played a major role in determining classification outcomes, proving the effectiveness of the integration approach. The research findings deliver major practical value to cybersecurity defenders through improved automated malware triage systems that reduce analyst time waste from false positives while minimizing threats that evade detection due to false negatives. The WCFG methodology presents a deployable solution that unites static analysis speed with improved detection precision to fulfill the essential requirement for flexible malware classification systems operating in advanced threat environments.

Download

Share

COinS