How do I unhook a global hook from another process?

Question

Here's my current setup: I have a C++ DLL that hooks one of its functions globally to every process running on the computer. The hooking is done in DLLMain using the SetWindowsHookEx winapi function, and I'm hooking to the WH_CBT and WH_SHELL events. I also have a C# application that loads the DLL with p/invoke ( LoadLibrary() ) which triggers the installation of the hooks from DLLMain . The handlers in the DLL are sending event information to the C# app through a named pipe.

Based on what I've read on the microsoft documentary , these events will be handled on the target process's thread, and have to be installed by a standalone C++ DLL (unlike WH_MOUSE_LL and WH_KEYBOARD_LL that can be installed by any application, even straight from a C# app using p/invoke ).

So far everything works fine; the managed app is receiving data as it should. The problem arises when I shut down the application, because the handlers are still hooked, and therefore the DLL file is in use and can't be deleted.

Since the handler is not running in my application, but instead it's injected into other processes running on my computer, the C# app can't just simply call UnhookWindowsHookEx or FreeLibrary , because the pointer of the event handler belongs to other processes.

The question:

How can I trigger an unhook routine from the managed application that makes sure that the DLL is not in use anymore by any process?

Here's what i've tried:

The only solution that I wan able to come up with is to create an exit event (with CreateEvent ) and every time the handler receives WH_CBT or WH_SHELL message, it checks if the exit event is set, in which case it unhooks itself from the process it belongs to and returns before processing the message.

The problem with this approach is that after I shut down my application and unload the DLL, I have to wait until the remaining processes receive an WH event at least once, so the handler belonging to them can unhook itself.

Here's the code of the DLL:

#include <windows.h>
#include <sstream>

HANDLE hTERM;
HHOOK hCBT;
HHOOK hShell;

void __declspec(dllexport) InstallHooks(HMODULE h);
void __declspec(dllexport) RemoveHooks();

int Continue()
{
    return WAIT_TIMEOUT == WaitForSingleObject(hTERM, 0);
}

LRESULT FAR PASCAL _cbtProc(int c, WPARAM w, LPARAM l)
{
    if (!Continue()) { RemoveHooks(); return 0; }   
    // Handling the message ...
    return CallNextHookEx(0, c, w, l);
}

LRESULT FAR PASCAL _shellProc(int c, WPARAM w, LPARAM l)
{
    if (!Continue()) { RemoveHooks(); return 0; }
    // Handling the message ...
    return CallNextHookEx(0, c, w, l);
}

void InstallHooks(HMODULE h)
{
    hTERM = OpenEvent(EVENT_ALL_ACCESS, 0, __TEXT("{0C3ED513-F38C-4996-8130-F9A3C93D890B}"));
    if (!Continue())
        return;
    hCBT = SetWindowsHookEx(WH_CBT, _cbtProc, h, 0);
    hShell = SetWindowsHookEx(WH_SHELL, _shellProc, h, 0);
}

void RemoveHooks()
{
    UnhookWindowsHookEx(hCBT);
    UnhookWindowsHookEx(hShell);
    if (hTERM) CloseHandle(hTERM); hTERM = 0;
}

int FAR PASCAL DllMain(HMODULE h, DWORD r, void* p)
{
    switch (r)
    {
        case DLL_PROCESS_ATTACH: InstallHooks(h); break;
        case DLL_PROCESS_DETACH: RemoveHooks(); break;
        default: break;
    }
    return 1;
}

There is nothing special about the source code of the managed C# app, because the only thing it does is that it calls LoadLibrary on start, handles the messages coming from the pipe, and finally sets the exit event with an other p/invoke call when needed.

Answer 1

" The hooking is done in DLLMain " - that is the completely wrong place to handle this.

Every time the DLL is loaded into a new process, it is going to install a new set of Shell/CBT hooks, which you DO NOT want/need to happen. You only need 1 set.

The correct solution is to have your DLL export its InstallHooks() and RemoveHooks() functions, and then have ONLY your C# app call them AFTER it has loaded the DLL into itself. That single set of hooks will handle loading the DLL into all running processes as needed WITHOUT you having to call SetWindowsHookEx() each time.

Also, DO NOT call UnhookWindowsHookEx() from inside the hook callbacks themselves. Before the C# app exits, it should call RemoveHooks() , which can then signal the hTerm event before calling UnhookWindowsHookEx() . The callbacks should simply exit if Continue() returns false, nothing more. But DO NOT skip calling CallNextHookEx() even if Continue() returns false, as there may be additional hooks installed by other apps, and you DO NOT want to break them.

Try something more like this instead:

#include <windows.h>

HMODULE hModule = NULL;
HANDLE hTERM = NULL;
HHOOK hCBT = NULL;
HHOOK hShell = NULL;

static bool Continue()
{
    return (WAIT_TIMEOUT == WaitForSingleObject(hTERM, 0));
}

LRESULT CALLBACK _cbtProc(int code, WPARAM wParam, LPARAM lParam)
{
    if (Continue()) {
        // Handle the message ...
    }
    return CallNextHookEx(NULL, code, wParam, lParam);
}

LRESULT CALLBACK _shellProc(int code, WPARAM wParam, LPARAM lParam)
{
    if (Continue()) {
        // Handle the message ...
    }
    return CallNextHookEx(NULL, code, wParam, lParam);
}

__declspec(dllexport) BOOL WINAPI InstallHooks()
{
    if (!Continue())
        return FALSE;

    if (!hCBT)
        hCBT = SetWindowsHookEx(WH_CBT, _cbtProc, hModule, 0);

    if (!hShell)
        hShell = SetWindowsHookEx(WH_SHELL, _shellProc, hModule, 0);

    return ((hCBT) && (hShell)) ? TRUE : FALSE;
}

__declspec(dllexport) void WINAPI RemoveHooks()
{
    if (hTERM)
        SetEvent(hTERM);

    if (hCBT) {
        UnhookWindowsHookEx(hCBT);
        hCBT = NULL;
    }

    if (hShell) {
        UnhookWindowsHookEx(hShell);
        hShell = NULL;
    }
}

BOOL WINAPI DllMain(HMODULE hinstDLL, DWORD fdwReason, void* lpvReserved)
{
    hModule = hinstDLL;

    switch (fdwReason)
    {
        case DLL_PROCESS_ATTACH:
            hTERM = CreateEvent(NULL, TRUE, FALSE, TEXT("{0C3ED513-F38C-4996-8130-F9A3C93D890B}"));
            if (!hTERM) return FALSE;
            break;

        case DLL_PROCESS_DETACH:
            if (hTERM) {
                CloseHandle(hTERM);
                hTERM = NULL;
            }
            break;
    }

    return TRUE;
}

Then, your C# app can simply load the DLL and call InstallHooks() and RemoveHooks() when needed. For instance, using PInvoke calls at app startup and shutdown, respectively.