Hooked VirtualAlloc returns nullptr when called by system DLLs

Question

I would like to track memory allocations done by VirtualAlloc . For that I tried to use mhook and easyhook . Both work fine as long as I call VirtualAlloc myself. The registered hook is called, from there I call the real VirtualAlloc function and memory is returned as expected.

My problem now is that I try to open a SFML -window and the VirtualAlloc call in there is captured but no memory is returned. The function call goes through sfml-window-2.dll and from there it calls wglCreateContext that calls CreateAnyContext which calls VirtualAlloc . wglCreateContext is in Opengl32.dll. I guess my userspace code doesn't have the permission to allocate memory for it. My question is, why is a system library going through the same stub (that I hooked) if I am not allowed to allocate memory for it? And how could I distingush these calls and prevent that from happening?

I mean, in my particular case I could wait until the window is open and hook VirtualAlloc after that. But this would be a really bad solution because I would like to track arbitrary programs maybe even inject my dll to track the memory.

EDIT: I not only want to track the programs memory but I also want to be able to reset it to a recorded state, all while the programs execution should not be slowed down too much because in my case it will be games. Therefore I wanted to start with VirtualAlloc and set the MEM_WRITE_WATCH flag and track the pages returned by GetWriteWatch manually. I already wrote a vector that can be reset this way but I need it to be more general. /EDIT

Another thing to note is, that I have tried it with HeapAlloc which is called in said function too but it returns memory. This is the reason why I tried two libraries. Any help is appreciated.

Answer 1

Turns out that I wasn't allowed to alter the flags. My workaround, since I can't know who the caller is in a reasonable way, I allocate it with altered flags and if that fails I allocate it again with the original flags.