SetWindowsHookEx injection failed on release mode but worked on debug mode

Question

I'm writing a program which monitors Keystrokes of a target process using SetWindowsHookEx . (IDE: Visual Studio 2013)Here's an overview of my program:

1. Obtain a HWND of the target process using FindWindow() .
2. If HWND is valid, obtain the process id using GetWindowThreadProcessId()
3. Obtain a thread id by traversing the thread list with CreateToolhelp32Snapshot(TH32CS_THREAD)
4. Call SetWindowsHookEx() .

Actual code:

//obtain the window handle
HWND hwnd = FindWindowA(NULL, "A valid title");
DWORD pid = 0;
//obtain the process id.
GetWindowThreadProcessId(hwnd, &pid);
//obtain the thread id.
DWORD threadId = GetThreadId(pid);
printf("Injecting to Process: %d Thread: %d\n", pid, threadId);
HMODULE hDll = LoadLibraryA("TestDLL.dll");
if (hDll == INVALID_HANDLE_VALUE)
{
    printf("LoadLibrary() failed! %d!\n", GetLastError());
    return 0;
}
HOOKPROC hookproc = (HOOKPROC)GetProcAddress(hDll, "KeyboardProc");
if (!hookproc)
{
    printf("GetProcAddress() failed\n");
    return 0;
}

HHOOK hook = SetWindowsHookEx(WH_CALLWNDPROC, hookproc, hDll, threadId);
if (!hook)
{
    printf("SetWindowsHookEx() failed! %d\n", GetLastError());
    return 0;
}
//post a message. This will trigger the hook and cause the target process 
//to load my dll. Actual key monitoring code is inside the dll.
printf("SendMessage() returns:%d", SendMessage(hwnd, WM_NULL, 0, 0));

printf("Success!\n");
UnhookWindowsHookEx(hook);
getchar();

Under Debug mode, the output shows:

Injecting to process 4052 Thread:460
SendMessage() returns:0
Success!

A simple analysis shows that the target process did load my dll. Which means the program works. However, under release mode, the output is the same but dll is not loaded into the target process. I tried this multiple times with restarting target process each time. But still doesn't work. How do I resolve this problem?

Answer 1

When you say "under Debug mode" - does it mean that you are debugging inside VS? If so, my guess that the problem might be in permission set - you can run VS with elevated permissions or under another user/group. Try to run your release version of the app in Admin mode. Otherwise it would be a security flaw if any process can inject code into any another process.