Thread-Safe hooking function

Question

I am implementing little monitoring application, so i am hooking CreateWindowExA/W in process, so i can control the windows creation. The method i use to hook is to replace first 5 bytes from the call with assembler JMP instruction to my hook function. (Yes i know assembler, i used same method many times before). I use EnterCriticalSection at beginning of my hook code and i use InterlockedExchange to restore the stolen bytes aka replacing the JMP i wrote at beginning of CreateWindowExA/W with the real 5 bytes so i can call the function correctly. According to my experience everything has to be fine, but what happen is that at the moment when i just replaced the JMP with the real bytes, some other thread/s call the function, looks like the bytes are replaced for them too....

I know i can use IAT/EAT tables hooking but i want to know whats wrong with my current method... Maybe the problem that InterlockedExchange is not working is that, CreateWindowExA/W is called from dll's(comctl32.dll,shell32.dll...) but not the main executable module.

I hope someone help me, if you dont understand well my explanation please ask and i will re-explain.

Answer 1

If you are hooking windows functions, IAT hooks are far better and safer. however, if you insist on using detours, its generally better to use the hotpatching builtin on the windows side of things (this makes writing of the detour atomically possible, requiring no synchronization).

Your problems is exactly as you say, your lock only suspends your thread(s) of execution, but not those controlled by you. to fix this you either need to suspend all those threads (via PSAPI/toolhlp32), or more efficiently, add a check to the function you detoured to that checks if the callee address lies in the address space of the modules you want detoured, this can be done with GetModuleHandle, some PE functions from WinNT and the _ReturnAddress intrinsic.

Answer 2

As soon as you change back the bytes the hooking will be lost and a critical section won't help because this is after the jmp.

Have a look at http://dxhook.googlecode.com/svn/trunk/dxhook.cpp in DXHooks as I think it'll do what you need

Answer 3

If you can control every single call to CreateWindowEx then you can wrap them with a critical section. However, if you could do that, then you would not need to hook the function.

What's left is to hook the function before any other threads have started. Do this right at the beginning of your application, possibly by using static initialization.

One other thought about this particular function. In many apps, all windows are created in the main thread. If that is so for you then you have no need to synchronize.