Date of Award

Spring 3-2023

Document Type

Dissertation

Degree Name

Doctor of Philosophy in Cyber Operations (PhDCO)

Department

Computer Science

First Advisor

Cody Welu

Second Advisor

Cherie Noteboom

Third Advisor

Tyler Flaagan

Abstract

This quasi-experimental before-and-after study measured the performance impact of using Process Instrumentation Callback (PIC) to detect the use of manual system calls on the Windows operating system. The Windows Application Programming Interface (WinAPI), the impacts of system call monitoring, and the limitations of current detection mechanisms and their downsides were reviewed in-depth. Previous literature was evaluated that identified PIC as a unique solution to monitor system calls entirely from User-Mode, being able to rely on the Windows Kernel to intercept a target process. Unlike previous monitoring techniques, PIC must handle all system calls when performing analysis which requires an increase in processing. The impact on a single process was evaluated by recording CPU time, memory utilization, and clock time. Three different iterations that performed additional analysis were developed and tested to determine the cost of increased fidelity in detection. Results showed a statistically significant increase when PIC was applied in each version. However, the rate of impact was drastically reduced by restricting dynamic lookups to process initialization and the elimination of the Microsoft Debugging Engine. Future integration with existing detection mechanisms such as User-Mode hooks and Event-Tracing for Windows is encouraged and discussed.

Download

Share

COinS