How to check whether new threads created inside third party DLL in visual c++ application

Question

I am currently struggling with a problem of verifying whether new threads created inside a dll file we call from our code in visual c++ application. Preferably i wanted see resource consumption and call stacks if there are any threads created inside the dlls. I need this to happen in running application not in debugging mode. Do any of you know how we can approach to solve this ?

Answer 1

Have the app hook the CreateThread API in its startup code, preferably before the DLLs initialize. This is deep magic though; you gotta know exactly what you're doing.

EDIT: if you have to ask, you're probably not up to the task. Proceed at your own risk.

The general idea would go: use the disassembler to see the first few commands of CreateThread() in a running process. In the main exe code, patch the memory of CreateThread to insert a JMP command to your own trampoline function. Write a trampoline in assembly that would call your own hook function, maybe repeat some commands from the patched portion, and jump back to the unpatched portion of CreateThread().

This is not meant for production - only for running the executable on your own machine, with a known version of CreateThread, since it would depend on the contents of CreateThread. In a way, the application becomes its own debugger, but this is not a debugger the DLL can easily notice.

For patching the executable part of the process memory, you might have to adjust the memory protection flags, it won't be writable by default.

Naturally, you'd have to mind the bitness.

Alternatively, you can patch the beginning of CreateThread to cause an exception (INT 3 for example) and use a vectored exception handler to catch that. Win32 API functions have a 2 byte NOP in the beginning, just enough to patch.

EDIT: tried the VEH approach, it seems cleaner. Now, on my machine, the first command of CreateThread is MOV EDI, EDI - that's effectively a 2 byte no-op, perfect for patching. So the exception handler goes:

LONG NTAPI OnExc(_EXCEPTION_POINTERS* Exc)
{
    if (Exc->ExceptionRecord->ExceptionCode == EXCEPTION_BREAKPOINT)
    { //Feel free to add an extra check for exception address
      //Just in case there are rogue INT 3's elsewhere

        wprintf(L"Yay, thread created\n");
        //Not a good idea to do I/O from the exception handler :)

        //Continue from the next command after INT 3
        Exc->ContextRecord->Eip++; 
        return EXCEPTION_CONTINUE_EXECUTION;
    }
    else
        return EXCEPTION_CONTINUE_SEARCH;
}

And the hooking code goes:

AddVectoredExceptionHandler(1, OnExc);

HMODULE hKernel32 = GetModuleHandle(L"kernel32.dll");
unsigned char* pCreateThread = (unsigned char*)GetProcAddress(hKernel32, "CreateThread");

//Allow writing to the memory block where CreateThread is
MEMORY_BASIC_INFORMATION mi;
VirtualQuery(pCreateThread, &mi, sizeof mi);
DWORD dw;
VirtualProtect(mi.BaseAddress, mi.RegionSize, PAGE_EXECUTE_READWRITE, &dw);

//Check if the first two bytes are indeed MOV EDI, EDI
if (pCreateThread[0] == 0x8b && pCreateThread[1] == 0xff)
{
    //And patch!
    pCreateThread[0] = 0xcc; //Replace with INT 3 
    pCreateThread[1] = 0x90; //Replace with NOP
}

That's it, hooked. Feel free to test it by calling _beginthread or _beginthreadex - but not under the debugger. The debugger would catch and process INT 3 before the exception handler does.

This is all 32-bit code. Haven't even looked at what the Win64 equivalent would be like.

There are other dimensions to the job that I didn't cover. Specifically, if the DLL in question is loaded statically, and it creates thread(s) in its startup code, and if the hook is installed in [Win]main(), then the hook won't catch those threads. Since this is your project, I have no way of checking that.

Another avenue of enhancement - if you want to catch the result of thread creation (eg the thread ID), this technique alone won't do. You'd need to hook not just the entry point of CreateThread, but also the exit.