How can I safely unhook a Win32 API that blocks?

Question

I have an application that does a bunch of API hooking into a Win32 target application (which uses ASIO) for watching named pipe traffic handled by the target. It sets up a beginning for a transaction with a ReadFile or a WriteFile call and gets the result of that call if the call was made synchronously. If the call is asynchronous, it also traps the end of that call through a hook to GetQueuedCompletionStatus. The retrieved data gets sent to my own ASIO thread pool which services a named pipe connection to Wireshark. It's actually pretty sweet.

Everything works great, except when I have to get the program back to its original state. Unhooking the functions works fine, except for the fact that existing calls can be blocked indefinitely in GetQueuedCompletionStatus unless I force the target application to process a ton of requests through the named pipe. If I don't wait around for those threads to unblock, I'll cause AV when GetQueuedCompletionStatus does eventually unblock and return to a code cave that no longer exists.

The other thing I tried was to track each call made to GetQueuedCompletionStatus and have the hooking function notify a signal whenever the LpOverlapped parameter of GetQueuedCompletionStatus matches a corresponding call to PostQueuedCompletionStatus. This unblocks everything, sure, but it seriously messes up code that made the GetQueuedCompletionStatus call, resulting in an AV.

Does anyone know of a good way to handle this? If I could do one of the following, this would work:

• Make a call with PostQueuedCompletionStatus that ASIO will ignore
• Trap a dummy call made by PostQueuedCompletionStatus and overwrite the return value in the stack
• Create a semi-permanent code cave containing a trampoline for hooked blocking calls which passes a function pointer accessor to the hooking calls. The trampoline function would see that if this value was not null, it would instead call that function pointer instead of return execution to the caller. Upon unblocking dispatch, the hooking code could set that function pointer to the address of GetQueuedCompletionStatus and then we'd be able to return a proper invocation result to the caller.

The first option would be easy, except that it would be nice to be able to generalize this application for use with other targets. The second would be easy, except for the fact that I want to write the code to be as safe as possible. The last one might be doable, I just don't want to pollute a foreign memory space with code caves.

Answer 1

I doubt a general and safe solution - other than simply keeping the DLL loaded - exists. Or refraining from all those hooks in the first place... (skip to the end of the answer for a major spoiler!)

---

The argument is along the lines of what I wrote in the comment to the qeustion.

To free the DLL you have to make sure nobody is currently executing code in the DLL and that nobody is going to going to execute code in the DLL (besides the tear-down thread that does what we're discussing).

Code may be going to be executing in the DLL for two reasons, either we're about to call it, or we're going to return to it (or return to code that going to return to the DLL, etc.). Lets say we solve the first problem by unhooking. We're still left with the second cause. Maybe we're already inside the hook.

You might think of doing something like this:

1. Allocate a “code cave” that does something like

    PreHookWriteFile: LOCK INC [ref_count] POP R15 CALL HookWriteFile PostHookWriteFile: LOCK DEC [ref_count] JMP R15 

2. Hook WriteFile with JMP [PreHookWriteFile]

3. Perform the release in a dedicated thread that unhooks WriteFile and wait for the recount to go to zero. Then it deallocates the code cave.

But there are two problems with that. First, we don't really have a place to store the original return address. We can't put it on the stack, since HookWriteFile doesn't expect to see it there, and we can't store it in any of the non-volatile registers, because we need to restore it in PostHookWriteFile before jumping back, but the problem is that we don't have a place to store things.

Second, the release thread might notice the decrease before the jump took place and release the code cave prematurely.

I don't think it matters how clever we try to be (either with __declspec(naked) functions or by modifying the prototype) to have the hook functions expect the original return address on the stack. The scond problem remains - you decrease the reference count before you return. It's not safe to remove the code before you returned, but you can't notify your return since after you've returned you're no longer in control. That's exactly the reason FreeLibraryAndExitThread was created.

One might think of something crazy like suspending all threads in the DLL and walking their stacks to make sure no code from the DLL is in any of them, but that's just opening another can of worms.

---

But there's good news too.

If what you really want is to monitor named pipe communication and you're willing to limit yourself to Windows 8 and newer versions there's a solution that's much more robust, documented and supported, take considerably less lines of code and isn't intrusive. Koby Kahane wrote a filesystem mini-filter that does exactly that .

It's output is ETW events that you can view in Microsoft Message Analyzer or one of the other tools that consume ETW events from manifest-based providers. If you really need to see it in Wireshark you can write a small ETW consumer that consumes the events and send them through a pipe back to Wireshark. It will still be easier than all those hooks, and certainly safer and a lot less messy.