通过DllImport调用非托管函数时损坏的堆

Question

I am using an unmanaged dll that is written in C/C++ from a C# application. I'm interested in using the following function from the dll:

    static void StorePath(const std::string& path, wchar_t *out_path,
     int *out_path_length){
      wcslcpy(out_path, c_str_w(path), *out_path_length);
      *out_path_length = path.size();
     }

    int WINAPI BrowseForDirectory(
     int allow_portable, int allow_online,
      wchar_t *t_directory, int *e_directory_length,
       wchar_t *m_directory, int *m_directory_length){
     .
     .
     . //initializing new forms and checking product keys

    StorePath(form->SelectedEDirectory().TopDir(), e_directory,
     e_directory_length);
    StorePath(form->SelectedMDirectory(), m_directory,
     m_directory_length);
    }

Header file:

    #if defined(_WIN32) && !BUILD_WITHOUT_DLLS &&!defined(ECLIPSE_CBUILDER_WORKAROUNDS)
    # if BUILDING_EXPORT_LIBRARY
    #  define EXPORT_DLL __declspec(dllexport)
    # else
    #  define EXPORT_DLL __declspec(dllimport)
    # endif
    #else
    #  define EXPORT_DLL
    #endif

    extern "C" {
        int WINAPI BrowseForDirectory(
         int allow_portable, int allow_online,
          wchar_t *t_directory, int *e_directory_length,
           wchar_t *m_directory, int *m_directory_length)
    }

Then, I am trying to invoke this function in my own managed, C# class library by doing the following:

    [DllImport("MyDLL.dll", CharSet = CharSet.Ansi)]
    public static extern int BrowseForDirectory(Int32 allowOnline, 
     Int32 allowPortable,
      [MarshalAs(UnmanagedType.LPStr)] StringBuilder eDirectory, 
       ref Int32 eDirLength, 
        [MarshalAs(UnmanagedType.LPStr)] StringBuilder mDirectory, 
         ref Int32 mDirLength);

Finally, I'm trying to use it in a C# application by calling it like:

    var eDir = new StringBuilder(260);
    var mDir = new StringBuilder(260);
    var eDirLength = eDir.Length;
    var mDirLength = mDir.Length;
    try
    {
        var result = Viewer.BrowseForDirectory(1, 1, eDir, 
         ref eDirLength, mDir, ref mDirLength);
    }
    catch(Exception ex)
    {
        MessageBox.Show(ex.ToString());
    }

However, I was getting a heap corruption, but now my application is exiting because of a STATUS_STACK_BUFFER_OVERRUN--something about an embedded breakpoint. Changing the C++ code is not an option. I have the proper reference and assemblies.

What am I doing wrong?

Answer 1

The problem that I can see is that your character sets do not match. The unmanaged code returns the text as UTF-16, but your p/invoke specifies ANSI encoded text. Change the p/invoke to:

[DllImport("MyDLL.dll", CharSet = CharSet.Unicode)]
public static extern int BrowseForDirectory(
    int allowOnline, 
    int allowPortable,
    StringBuilder eDirectory, 
    ref int eDirLength, 
    StringBuilder mDirectory, 
    ref int mDirLength
);

I'm assuming that c_str_w() takes an 8 bit encoded string and returns a pointer to null-terminated array of wchar_t .