SetWindowsHookEx not working with WH_CBT on another process thread ID (but works on own thread ID)

Question

I've been racking my brain on this for days, and could use some help! I'm able to detect a CBT hook using SetWindowsHookEx when it is hooked to the thread ID of my WPF application, but I'm not able to get it to hook to the Thread ID of another process window when that window becomes the foreground app.

Image 1: Shows that I CAN get the CBT hook for detecting window maximize on the main app's Thread ID

Image 2: Shows that I CANNOT get the CBT hook when listening to another app's Thread ID, and it will crash the app!

I want to avoid sending ThreadId=0 and making it a complete global hook, because I know that I only want to listen to the foreground app, not all apps on the desktop. I want to be able to listen to a few window events before they happen (WH_CBT does this from what I understand) for any window that currently has the foreground focus.

Again, the following code works when the current WPF app becomes the foreground app, but it fails when another app's window becomes the foreground (eg Notepad, Internet Explorer, File Explorer, Chrome, etc.).

Full Code: (link to github zip file)

Here are some snippets of the code to show what I did:

DLL that defines the callback (inject.dll):

inject.h: Snippet

typedef void(__stdcall* MYFUNCPTR)(int code, WPARAM wparam, LPARAM lparam); extern "C" __declspec(dllexport) void Init(MYFUNCPTR funcPtr); extern "C" __declspec(dllexport) LRESULT CALLBACK CbtProcCallback(int code, WPARAM wparam, LPARAM lparam); WPARAM wparam, LPARAM lparam); MYFUNCPTR _handler = 0;

inject.cpp: Snippet

public delegate void MessageCallback(string message);
public delegate void CodeCallback(int code, IntPtr wParam, IntPtr lParam);
[DllImport("dllwrapper.dll")]
public extern static bool StartHooks(uint threadId, MessageCallback messageHandler, CodeCallback codeHandler);
[DllImport("dllwrapper.dll")]
public extern static void StopHooks();

DLL that sets the CBT hook (dllwrapper.dll):

dllwrapper.cpp: Snippet

 // Load library in which we'll be hooking our functions. HMODULE dll = LoadLibrary(L"inject.dll"); if (dll == NULL) { char errorMessage[100]; sprintf_s(errorMessage, "ERR-LoadLibrary failed! ErrorCode=%d", GetLastError()); SendManagedMessage(errorMessage); return false; } SendManagedMessage("LoadLibrary passed!"); // Get the address of the function inside the DLL. MYPROC iaddr = (MYPROC)GetProcAddress(dll, "Init"); if (iaddr == NULL) { char errorMessage[100]; sprintf_s(errorMessage, "ERR-GetProcAddress for Init failed! ErrorCode=%d", GetLastError()); SendManagedMessage(errorMessage); return false; } SendManagedMessage("GetProcAddress for Init passed!"); iaddr(OnInjectionCallback); // Get the address of the function inside the DLL. HOOKPROC cbtProcAddress = (HOOKPROC)GetProcAddress(dll, "CbtProcCallback"); if (cbtProcAddress == NULL) { char errorMessage[100]; sprintf_s(errorMessage, "ERR-GetProcAddress for CbtProcCallback failed! ErrorCode=%d", GetLastError()); SendManagedMessage(errorMessage); return false; } SendManagedMessage("GetProcAddress for CbtProcCallback passed!"); // Hook the function cbtProcHook = SetWindowsHookEx(WH_CBT, cbtProcAddress, dll, _threadId); if (cbtProcHook == NULL) { char errorMessage[100]; sprintf_s(errorMessage, "ERR-SetWindowsHookEx cbtProcAddress failed! ErrorCode=%d", GetLastError()); SendManagedMessage(errorMessage); return false; } SendManagedMessage("SetWindowsHookEx for cbtProcAddress passed!");

Snippet exporting to C#

 typedef void(__stdcall* CodeCallback)(int code, WPARAM wparam, LPARAM lparam); typedef void(__stdcall* MessageCallback)(const char* message); #ifdef __cplusplus extern "C" { // only need to export C interface if // used by C++ source code #endif __declspec(dllexport) bool StartHooks(unsigned int threadId, MessageCallback messageCallback, CodeCallback codeCallback); __declspec(dllexport) void StopHooks(); #ifdef __cplusplus } #endif

NativeMethods.cs: Snippet of C# dll imports to the WPF app

public delegate void MessageCallback(string message); public delegate void CodeCallback(int code, IntPtr wParam, IntPtr lParam); [DllImport("dllwrapper.dll")] public extern static bool StartHooks(uint threadId, MessageCallback messageHandler, CodeCallback codeHandler); [DllImport("dllwrapper.dll")] public extern static void StopHooks();

I can see from the messages in the WPF app window that the hook is passing and not returning any Win32 errors, but it's just not executing the callback when another window has focus (even when using the debugger).

Any help would be greatly appreciated!

Development Environment:

• Windows 10 1909
• VS2019 16.7.4
• C# .NET Framework 4.7.2, C++

Answer 1

When the main app calls Init() , it is calling into its own instance of the DLL, and so is setting its own copy of _handler .

When your hook DLL is injected into another process, that process will get its own copy of the DLL, and thus its own copy of _handler . But Init() is never called on that DLL instance, so its copy of _handler is still 0 when your hook function is invoked inside that process. That is why the process crashes, since the hook is not checking for that condition.

Your DLL in another process can't call a function in your main app across process boundaries. You are going to have to change your hook function to instead use an IPC mechanism of your choosing to communicate with the main app process when _handler is 0. Window messages, pipes, mailslots, sockets, etc can be used for that communication.