Date of Award

Spring 5-2022

Document Type

Dissertation

Degree Name

Doctor of Philosophy in Cyber Operations (PhDCO)

Department

Computer Science

First Advisor

Joshua Stroschein

Second Advisor

Austin O'Brien

Third Advisor

Shengjie Xu

Fourth Advisor

Viki Johnson

Fifth Advisor

Thomas McGuire

Abstract

The macOS operating system is increasingly targeted by malware. Software written for macOS, both benign and malicious, is in the Mach-O executable format. Malware authors may frustrate analysts through obfuscation methods such as packing. The field of malware research on Windows is well-established but is less so on the macOS platform. Thus far, no research has been identified that studies how machine learning can be used to detected packed Mach-O malware. This research applies supervised machine learning techniques to the classification of packed Mach-O malware. This research will answer three research questions. First, whether machine learning can classify packed Mach-O binaries. Second, whether machine learning can classify packed Mach-O malware. Third, whether machine learning can classify the family that a malware sample belongs to. A design science methodology is used to develop an artifact and apply it against the target problem. Both malware and benignware samples are collected and processed to extract useful information. This information is enriched and parsed into a feature vector. Machine learning models are trained against the three problems identified by the research questions. The model hyperparameters are tuned and relevant features are selected. The results of the experiments show that machine learning can classify packed Mach-O binaries with 100% F1 and can classify packed Mach-O malware with 94.61% F1. However, machine learning can only perform multiclass classification of packed malware family with 69.1% F1.

Download

Share

COinS