Date of Award

Fall 12-2023

Document Type

Dissertation

Degree Name

Doctor of Philosophy in Computer Science (PHDCS)

Department

Computer Science

First Advisor

Michael Ham

Second Advisor

Kyle Murbach

Third Advisor

Yong Wang

Abstract

Return-Oriented Programming (ROP) is an exploitation technique that is commonly used by malicious users. It works by leveraging return statements in binaries to gain control over the execution of programs. Some mitigations for ROP include changing the binary during compilation time, rewriting the binary after compilation, and adding runtime checks to the binary. The focus of this study was rewriting the binary after compilation. Rewriting during compilation time requires end users to have access to source code, which, in most cases, they will not. Adding runtime checks adds additional overhead to the target binary.

The areas this study aimed to improve in the binary rewriting space were twofold. The first was improving static binary rewriting. This was done by attempting to see if the amount of information needed to correctly rewrite a binary could be reduced compared to other tools. The second area was attempting to use static binary rewriting to reduce the number of potential ROP gadgets in a binary. The ROP gadgets that were targeted were those created by splitting an instruction that contains a return in them to create new ROP gadgets. This was chosen because most current tools focused on the safety of standard returns from function ends.

To determine if static binary rewriting could be used to reduce the amount of ROP gadgets created from mid-instruction ROP gadgets, a design science approach was taken. There were two artifacts that were created through two design cycles. The first artifact aimed to create a static binary rewriter that collected minimal amount of information from binaries. The second artifact built upon the first artifact and attempted to use it to remove instructions that contained a mid-instruction return. After the removal of the mid-instruction return, the second artifact inserted instructions that allowed for the same functionality of the binary, but without the return byte.

Download

Share

COinS