Robert Riskin

Date of Award

Spring 3-2025

Document Type

Dissertation

Degree Name

Doctor of Philosophy in Cyber Operations (PhDCO)

First Advisor

Kyle Cronin

Second Advisor

Michael Ham

Abstract

The Microsoft Windows operating system dominates a majority market share in both business and consumer computing. Among the various types of threats that exist to the Windows operating system, banking trojans have been evolving and continue to be a significant threat. A plethora of detection and preventative technologies exist to combat the various threats that target the Windows operating system, including banking trojans. Application control mechanisms are an often-overlooked technology that can augment existing security tools. Built-in Windows application control security mechanisms exist that can have a significant impact on the outcome of banking trojan attacks.

Current academic research presents a knowledge gap in evaluating the use of application control mechanisms in preventing malicious code execution. The purpose of this study was to evaluate the effectiveness of utilizing a Microsoft Windows built-in application control mechanism, AppLocker, in preventing the successful code execution of banking trojans. This study provides technical knowledge into the functionality and statistical outcomes of code execution prevention via AppLocker against various banking trojan samples and their file-types.

This study used a design science methodology to create a laboratory environment for testing multiple single-case mechanism experiments. A real-world environment was built that consisted of a networking layer, physical testing machines and a virtualized Active Directory domain server with logging capabilities. Microsoft Windows based banking trojan sample files were obtained and classified by their banking trojan family category, originating year, file-type and payload staging type. Two separate AppLocker rulesets were configured and deployed alongside a baseline system without an AppLocker configuration. All three systems attempted execution of sample files and the outcomes were logged and analyzed.

The results of this study convey the statistical significance that a properly tuned application control mechanism has in preventing malicious code execution. This study evaluated two AppLocker configurations: a default ruleset (BAL) and a tuned ruleset (FAL). The findings show the BAL ruleset demonstrated an average efficacy of 97% prevention in Stage 1 aggregate banking trojan sample files and an average efficacy of 65% prevention in Stage 2 aggregate banking trojan sample files. Conversely, the FAL ruleset displayed perfect prevention efficacy of 100% prevention in both Stage 1 and Stage 2 banking trojan sample files. These findings highlight the effectiveness that an application control mechanism, such as AppLocker, can have by preventing malicious code execution.

This study concludes that AppLocker can be used as an effective mechanism for preventing the execution of banking trojans on Windows-based operating systems. This study and its framework can be used by future researchers to examine the impacts and effectiveness of utilizing AppLocker against other types of Windows based threats. Microsoft is actively developing other application control mechanisms, including App Control for Business for newer operating systems such as Windows 11 and Server 2025. Future research into application control should be explored as discerned from this study, it can be highly effective at preventing malicious code execution.

Download

Share

COinS