SetWindowsHookEx() doesn't return an error when hooking 64 bit processes from a 32 bit process?

Question

My application is compiled as 32-bit, and since I run on 64-bit Windows 7, my target( notepad.exe ) is 64-bit. When I call SetWindowsHookEx() on the first thread that I find of notepad.exe , the DLL doesn't get injected at all, but there is no error returned. I know it's not being injected because on DLL_PROCESS_ATTACH I display message box with the message Attached , and for DLL_PROCESS_DETACH I display a Detached message in a message box. These messages are only displayed once for when I call LoadLibrary() and another time for when my application exits.

According to the MSDN documentation here :

Because hooks run in the context of an application, they must match the "bitness" of the application. If a 32-bit application installs a global hook on 64-bit Windows, the 32-bit hook is injected into each 32-bit process (the usual security boundaries apply). In a 64-bit process, the threads are still marked as "hooked." However, because a 32-bit application must run the hook code, the system executes the hook in the hooking app's context; specifically, on the thread that called SetWindowsHookEx. This means that the hooking application must continue to pump messages or it might block the normal functioning of the 64-bit processes.

Does this mean that it's hooking my own process successfully instead of actually returning an error?

Edit : My hook is of WH_CBT type.

Answer 1

You need to read further the docs:

For a specified hook type, thread hooks are called first, then global hooks. Be aware that the WH_MOUSE, WH_KEYBOARD, WH_JOURNAL*, WH_SHELL, and low-level hooks can be called on the thread that installed the hook rather than the thread processing the hook. For these hooks, it is possible that both the 32-bit and 64-bit hooks will be called if a 32-bit hook is ahead of a 64-bit hook in the hook chain.

In short it depends on the hook type if your dll is injected to the target process at all. If you only want to spy on keyboard and mouse events there is no need to inject yourself into other processes. Windows will call back your hook in your own process.

I suspect your hook type is one of these:

• WH_MOUSE_LL
• WH_KEYBOARD_LL

which do not cause any library injection into the target process.

Answer 2

文档的那部分内容没有明确指出的是，当SetWindowsHookEx无法将32位DLL注入64位进程中时，它没有返回错误，而是使用了调用SetWindowsHookEx的线程的消息循环来执行挂钩过程，就像它与低级鼠标/键盘挂钩（WH_MOUSE_LL / WH_KEYBOARD_LL）一样。