Debug an imported dll with VS 2010

Question

I have a problem while I trying to call WinAPI functions from C# code. I have lots of imports, many of them works fine, but some of them not and leading to unexpected break main program, without any message, exception type, nothing, just fell down all its windows and exit.

I have two ways in code: via my developed library, where is more of winapi calls and I'm lazy to code specific structures, pointers, etc, and direct import from user32.dll, like this:

[DllImport(@"tradeInterop.dll")]
    public static extern void ChooseInstrumentByMouse(UInt32 hwnd, int baseX, int baseY, int idx, int _isDown);
    [DllImport(@"tradeInterop.dll")]
    public static extern void PutNumber(int num);
    [DllImport(@"tradeInterop.dll")]
    public static extern void PutRefresh();
    [DllImport(@"user32.dll")]
    public static extern UInt32 FindWindow(string sClass, string sWindow);
    [DllImport(@"user32.dll")]
    public static extern int GetWindowRect(uint hwnd, out RECT lpRect);
    [DllImport(@"user32.dll")]
    public static extern int SetWindowPos(uint hwnd, uint nouse, int x, int y, int cx, int cy, uint flags);
    [DllImport(@"user32.dll")]
    public static extern int LockSetForegroundWindow(uint uLockCode);
    [DllImport(@"user32.dll")]
    public static extern int SetForegroundWindow(uint hwnd);
    [DllImport(@"user32.dll")]
    public static extern int ShowWindow(uint hwnd, int cmdShow);
    [DllImport(@"tradeInterop.dll")]
    public static extern ulong PixelColor(uint hwnd, int winX, int winY); //tried (signed) long and both ints as return type, same result (WINAPI says DWORD as unsigned long, what about 64-bit assembly where compiled both lib and program?
    public struct RECT
    {
        public int Left;        
        public int Top; ...

As I said, many of this calls works perfectly, but have problem of last two of them: ShowWindow() and PixelColor() with following code:

extern "C" __declspec(dllexport) COLORREF __stdcall PixelColor(unsigned hwnd, int winX, int winY)
{
    LPPOINT point;
    point->x = winX;
    point->y = winY;
    ClientToScreen((HWND) hwnd, point);
    HDC dc = GetDC(NULL);
    COLORREF colorPx = GetPixel(dc, point->x, point->y);
    ReleaseDC(NULL, dc);
    return colorPx;
}

So, while I try to call directly imported ShowWindow() function, or library which calls api function(s), I got a program crash

Is there any way how to debug external libraries and its results?

What I¨m doing wrong?

Thanks a lot

Answer 1

You have several options for debugging issues.

1. Enable unmanaged code debugging in Visual Studio. Note: VS 2010 Express does not support mixed managed/unmanaged debugging. ( Instructions )
2. Use WinDbg . This would be my personal favorite option for debugging mixed applications. It is an incredibly powerful tool. Admittedly the learning curve is a little steep, but it's well worth the effort.
3. Use an external/third party debugger like OllyDbg. (As suggested by MrDywar )

The P/Invoke problems:

1. As IInspectable pointed out HWND should be passed to unmanaged code using IntPtr .
2. Windows API data types are well defined. DWORD is always 32 bit. Also, LONG (all caps) is not the same as long (lower case).

The C++ Problems:

1. As mentioned by IInspectable , the LPPOINT is never initialized, so when you call ClientToScreen() you're accessing uninitialized stack garbage as a pointer, and assigning values. To Fix it you could:
   1. Allocate the memory (your favorite scheme here, LocalAlloc , GlobalAlloc , malloc , calloc .
   2. Make the declaration POINT point; , assign values using point.x & point.y , and use it in the function call as &point .
2. Make the first parameter's type HWND . The API types, even though they are ultimately typedefs, carry extra meaning with them.