Recently, I was building a library that needed to offer API instrumentation capabilities for native binaries. Normally, I would use a solution such as Microsoft Detours, but I wanted to give a try at writing my own library.
I needed to support binaries compiled for both native x86 and x64. This meant that I needed to have control over the stack and registers in my detour, which would require the use of assembly. In the past, one way of doing this was using inline assembly with the naked attribute. These features are not supported in the Microsoft C++ compiler for x64, so I decided that writing my detours in pure assembly would be the best avenue forward.
I was able to write the detour code that I needed in x86 assembly with minimal issues. However, x64 assembly proved to be a challenge.
After adjusting my x64 detour code to account for some slight changes in bit-ness, I kept running into an issue where my stack would become corrupted when invoking external functions. I debugged through the assembly of the external functions and identified the functions seemed to be carelessly writing to the stack at any point past the return address (RSP+8 and thereafter). This made no sense to me, as I had never seen this behavior when debugging x86 assembly. Thus, I decided to conduct some additional research, and it turned out the answer was within our own MSDN document on x64 Stack Usage:
“All memory beyond the current address of RSP is considered volatile: The OS, or a debugger, may overwrite this memory during a user debug session, or an interrupt handler. Thus, RSP must always be set before attempting to read or write values to a stack frame.”
I decided that the best way to not encounter this issue was not to invoke external functions, and write anything that I needed in assembly. However, that's easier said than done when you're needing to implement a myriad of logic in your detour. One workaround that didn't require porting all my external functions to assembly was to adjust the stack in the prolog of my function, and restore it in the epilog:
SomeFunction PROC
SUB RSP, 80h;
[truncated x64 code]
ADD RSP, 80h;
SomeFunction ENDP
While this seemed effective, I would not use this in a product unless I had a full understanding of the ways the external functions manipulate the stack (due to the possibility that they could still overwrite values past my stack adjustment).
If there was a lesson to be learned during this project, it was this: always read the manual.
Thanks to Chris Kirk (MSFT) for some guidance on this issue.