Local keyboard hook terminates the target process

Question

I'm trying to install a LOCAL keyboard hook into a process using a C++ DLL from managed C# code, as follows:

public class KeyboardHook
{
    [UnmanagedFunctionPointer(CallingConvention.StdCall)]
    public delegate IntPtr HookCallback(int nCode, IntPtr wParam, IntPtr lParam);

    [DllImport("DLL.dll", CallingConvention = CallingConvention.Cdecl)]
    protected static extern IntPtr Install(int idHook, IntPtr windowHandle, HookCallback callback);

    private IntPtr instance;
    private HookCallback handler;

    public KeyboardHook()
    {
        instance = IntPtr.Zero;
        handler = Callback;
    }

    public void Install(Process process)
    {
        instance = Install(WH_KEYBOARD, process.MainWindowHandle, handler);
    }

    public void Uninstall()
    {
        UnhookWindowsHookEx(instance);
    }

    private IntPtr Callback(int nCode, IntPtr wParam, IntPtr lParam)
    {
        // TODO Use hook data here

        return CallNextHookEx(instance, nCode, wParam, lParam);
    }
}

The C++ DLL code should be just enough to dispatch the hook data to the C#'s Callback function, like this:

// dll.h
#pragma data_seg(".foo")
HOOKPROC _hookCallback = NULL;
#pragma comment(linker, "/SECTION:.foo,RWS")
#pragma data_seg()

static HINSTANCE _moduleHandle = NULL;

extern "C" __declspec(dllexport)
HHOOK Install(int idHook, HWND window, HOOKPROC hookCallback);

extern "C" __declspec(dllexport)
LRESULT CALLBACK HookProc(int code, WPARAM wparam, LPARAM lparam);

// dll.cpp
HHOOK Install(int idHook, HWND window, HOOKPROC hookCallback)
{
    auto processId = 0ul;
    auto threadId = GetWindowThreadProcessId(window, &processId);

    _hookCallback = hookCallback;
    _hookCallback(-1, NULL, NULL); // Test callback (works)

    return SetWindowsHookExA(idHook, HookProc, _moduleHandle, threadId);
}

LRESULT CALLBACK HookProc(int code, WPARAM wParam, LPARAM lParam)
{
    // The following line terminates the target process
    return _hookCallback(code, wParam, lParam);
}

BOOL APIENTRY DllMain(HMODULE hModule, DWORD ul_reason_for_call, LPVOID lpReserved)
{
    switch (ul_reason_for_call)
    {
        case DLL_PROCESS_ATTACH:
            _moduleHandle = hModule;
            break;
        case DLL_PROCESS_DETACH:
        case DLL_THREAD_ATTACH:
        case DLL_THREAD_DETACH:
            break;
    }

    return TRUE;
}

The local hook is successfuly installed as the DLL KeyboardProc function is triggered, however, calling the C# delegate from the C++ DLL terminates the application. Why?

NOTES:

1. Both the DLL and the application are 32 bits
2. _hookCallback_ is not null when HookProc is triggered (although I'm not sure whether it points to a valid memory address)
3. KeyboardProc::handler shouldn't be garbage collected, as the KeyboardProc instance lives for as long as the C# application does
4. Using _hookCallback function pointer within the DLL's Install function works flawlessly, but terminates the process when used inside the HookProc function.
5. There's no exception or whatsoever, the process just terminates abruptly

WHAT ALSO HAVE BEEN TRIED:

Making HookCallback a UnmanagedFunctionPointer , as well as using Marshal.GetFunctionPointerForDelegate and telling the garbage collector not to collect the handler property by using GCHandle.Alloc() and GC.KeepAlive() :

public class KeyboardHook
{
    [UnmanagedFunctionPointer(CallingConvention.StdCall)]
    public delegate IntPtr HookCallback(int nCode, IntPtr wParam, IntPtr lParam);

    [DllImport("DLL32.dll", CallingConvention = CallingConvention.Cdecl)]
    protected static extern IntPtr Install(int idHook, IntPtr windowHandle, IntPtr delegatePointer);

    // ...
    
    protected readonly GCHandle garbageCollectorHandle;
    
    public KeyboardHook()
    {
        instance = IntPtr.Zero;
        handler = new HookCallback(Callback);
        garbageCollectorHandle = GCHandle.Alloc(handler); // Or GC.KeepAlive(handler)
    }
    
    ~KeyboardHook()
    {
        garbageCollectorHandle.Free();
    }
    
    public void Install(Process process)
    {
        IntPtr delegatePointer = Marshal.GetFunctionPointerForDelegate(handler);
        
        instance = Install(WH_KEYBOARD, process.MainWindowHandle, delegatePointer);
    }
    
    // ...
}

Using handler directly into SetWindowsHookExA (C++):

HHOOK Install(int idHook, HWND window, HOOKPROC hookCallback)
{
    auto processId = 0ul;
    auto threadId = GetWindowThreadProcessId(window, &processId);

    _hookCallback = hookCallback;

    return SetWindowsHookExA(idHook, hookCallback, _moduleHandle, threadId);
}

Answer 1

The C# code is calling the DLL's Install() function with the 2nd and 3rd parameters flipped.

Change this line:

instance = Install(WH_KEYBOARD, handler, process.MainWindowHandle);

To this:

instance = Install(WH_KEYBOARD, process.MainWindowHandle, handler);

UPDATE : Also, you did not show the C# code's declaration of the DLL'S Install() function. But in the C++ code, the Install() function is not specifying a calling convention, so it will (likely, depending on compiler config) default to __cdecl ( HookProc() is using the __stdcall calling convention). So make sure the C# code is specifying the correct calling convention when importing the Install() function. C# delegate s use __stdcall by default.

Answer 2

The process terminates due to accessing an invalid memory address.

Every Windows process have different virtual memory regions. In other words, the 0x1234 memory address within the process A does not point to the same value/function as 0x1234 memory address within the process B , since 0x1234 is a virtual memory address bound to it's corresponding process.

In order to achieve that communication between a C++ DLL and a C# application (any different process overall), an inter-process communication (IPC) is required.

For those interested on this specific case, I ended up creating an invisible dummy window to serve as a central point for receiving messages through SendMessage calls from the DLL.