How to Debug a COM DLL

Question

I made changes to a function in a COM DLL. I've been unable to figure out how to debug the changes I made.

Background:

I recently inherited a Visual Studio 2012 C++ project that dates back many years. I'm running Win7 Professional 64-bit.

The top-level design of the project is this:

• The code that does most of the work is encapsulated in COM DLLs.

• For each DLL there's a separate wrapper function that calls:

   (1) CoInitialize (2) CoCreateInstance (3) CoUninitialize 

• There's a main program that presents a dialog to allow a user to select an option. Based on the selected option, the main program calls the appropriate wrapper function, which then runs the code in the corresponding COM DLL.

Problem Details:

(1) I've been unable to step through the code in the Visual Studio debugger. (Trying to run in the debugger produces the error "Unable to start program ", where the named DLL is different from the modified one.)

(2) I put "fprintf(stderr, ...)" calls in the modified DLL code, but didn't get any output from the "fprintf" calls. (I do see output from "fprintf" calls I added to the wrapper function that invokes the DLL.)

I also tried opening a temporary debug file using "fopen", writing debug statements to the file, then fflush, and fclose. Also no output.

(3) I noticed a post ( Calling fprintf from dynamic library (c++) ) that suggested that, although "fprintf(stderr, ...)" should work, it would be better to implement a callback to a debug function in the main program. I attempted to do that.

The changes compile, but the linker reports an undefined reference to the name of the function in the DLL that was intended to allow the main program to pass in a pointer to a callback function.

I'm confused by the undefined reference, because the modified DLL has a different exported function that the linker is able to resolve. Specifically:

• __declspec(dllexport) void SetLogFunc(LogFunc LogFuncPtr) [The new function.]
• __declspec(dllexport) BOOL DoRosSum(SRosSumData* pRosSumData) [The existing function.]

I used the "ack" utility to search the entire codebase for the project, including Visual Studio project files and binary files, looking for references to "DoRosSum", and can't find any place where there's a reference to "DoRosSum", but not also a reference to "SetLogFunc".

("SetLogFunc" is listed by "dumpbin" as an exported function.)

I should also mention that I reverted all my changes except for:

• the "SetLogFunc" function in the COM DLL,

• the callback debug function in the main program, and

• the call to ""SetLogFunc"" in the main program.

so I don't think the problem I'm having getting debug output, or running in the VS2012 debugger, is related to the modification I originally made to the DLL code.

Apologies for the long post. In recent years I've mostly been working in C#, or working on linux systems. I have no experience with COM DLLs, so I may be missing something simple.

If anyone has any thoughts on how to proceed I would appreciate it.

Answer 1

For the first question,

https://docs.microsoft.com/en-us/visualstudio/debugger/how-to-debug-from-a-dll-project?view=vs-2017

I think you can check if you have done this yet. Usually, if you haven't set up command correctly in project property/debugging, running debugger in VC always show you the message:

Unable to start program '......\\your DLL'.

Since DLL need to be execute with an executable(.exe).

I noticed that you said the error message 'the named DLL is different from the modified one' .

Your main program will run with multiple DLL right? I think you also need to make sure you put other DLL(.dll) file in the folder of your main program application.

For undefined reference issue ,

If you can compile the functions in DLL sides, it means the logic here is ok. However, linker report undefined error then it should be some issues for exporting this function to main program side.

I suppose you to first check in main program side to see if you include the correct header file.

Like right clicking the function name in main program "Go to Definition" will show you the header file. You can then see if the header file is the correct one.

Then you can check if you export functions correctly. You need to make sure main program link to the new compiled Object File Library(.lib). Every time you compile your DLL project you need to make sure main program link to the new compiled .lib and .dll and include the new header file.

Hope these helps.