关闭dll中的Win32对话框时出现异常（从WPF应用程序）

Question

I have a C# application that is calling a C++ function from a DLL. The C++ function shows a dialog box and a exit button only.

The C++ function in DLL is looking like:

//the exported function for clients to call in DLL
HINSTANCE hInstance;
__declspec(dllexport) int __cdecl StartDialog(string inString)
{
    hInstance = LoadLibrary(TEXT("My.dll"));
    DialogBox(hInstance, MAKEINTRESOURCE(ID_DLL_Dialog), NULL, DialogProc);
    return 1;
}

BOOL CALLBACK DialogProc(HWND hwnd, UINT message, WPARAM wParam, LPARAM lParam)
{
    switch (message)
    {
    case WM_COMMAND:
        switch (LOWORD(wParam))
        {
        case IDD_BUTTON_EXIT:
            DestroyWindow(hwnd);
            return TRUE;            
        }
    }
    return FALSE;
}

If I call my StartDialog function in a simple C++ program, it works. I can show the dialog, and can close it properly when I click exit in my dialog.

typedef int(__cdecl *StartDialogFunc)(string);
StartDialogFunc StartDialog = nullptr;
HINSTANCE hDll = LoadLibrary(TEXT("My.dll"));
StartDialog = (StartDialogFunc)GetProcAddress(hDll, "StartDialog");

int i = StartDialog("hello");    //this is working
cout << i;

If I call it in my C# application, after I click the exit button, the dialog closes, and give me an exception. The C# code looks like:

[DllImport("My.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern int StartDialog(string s);

int i = StartDialog("hello");    //this causes a exception

The error message is like:

Debug Assertion Failed!

Program: (some paths...) My.dll

File: d:\\program files (x86)\\microsoft visual studio 14.0\\vc\\include\\xmemory0

Line: 100

Expression: "(_Ptr_user & (_BIG_ALLOCATION_ALIGNMENT - 1)) == 0" && 0

For information on how your program can cause an assertion failure, see the Visual C++ documentation on asserts.

How can I know what is exactly wrong in my DLL?

Answer 1

Try changing the C++ function signature to take a WCHAR* , as C++'s std::string is incompatible with C#. Also, to get a handle to the current DLL, use GetModuleHandle(NULL), not LoadLibrary.

HINSTANCE hInstance;

__declspec(dllexport) int __cdecl StartDialog(const WCHAR* inString)
{
    hInstance = GetModuleHandle(NULL);
    DialogBox(hInstance, MAKEINTRESOURCE(ID_DLL_Dialog), NULL, DialogProc);
    return 1;
}