I can't get the input parameter of a PInvoke to C++ DLL from C# to be used as ouput with IntPtr

Question

I have a function in a C++ DLL that takes one input. I'm trying to have that input be used as an output to the C# call.

Here is my C++ function:

MYAPI int testStuff3(unsigned char* str)
{
    printf("%s\n", str);
    str = (unsigned char*)malloc(9);
    str[0] = 'G';
    str[1] = 'o';
    str[2] = 'o';
    str[3] = 'd';
    str[4] = 'b';
    str[5] = 'y';
    str[6] = 'e';
    str[7] = '!';
    str[8] = '\0';
    return 1;
}

Here is the C# code:

public class Program
{
    [DllImport("NativeLib.dll")]
    private static extern int testStuff3([In, Out] IntPtr str);

    static void Main(string[] args)
    {
        IntPtr junk3 = IntPtr.Zero;
        int ret = testStuff3(junk3);
        Byte[] stuff3 = new byte[9];
        Marshal.Copy(junk3, stuff3, 0, 9);
    }
}

When the Marshal.Copy is called, it gives an error saying that the source (junk3) can not be null.

Will this not work, sending a null pointer to C++ DLL from C# and having the DLL allocate the memory and store something inside and return it to the caller? I want to keep it an IntPtr and not a StringBuilder because the data won't necessarily be a string in the final code. Just an unsigned char array in C++ and I want the IntPtr to point to it.

I've tried different variations of [In, Out], [Out], out and ref for the IntPtr passing.

Answer 1

Never ever allow memory allocations to cross a DLL boundary. That way lies madness, and/or Sparta.

(For the pedantic: you can allocate memory and then pass a pointer across, as long as you either pass ownership back to free it, or guarantee that the same allocator is used as part of a contract. But it's still something to avoid when possible.)

Typically to use a string output parameter you should pass a StringBuilder as the argument, setting its capacity to the maximum expected length. Then in the native code you simply fill this existing buffer.

See the "Fixed length string buffers" section here for an example.

Answer 2

Your C++ function doesn't modify the passed string. It allocates a new one with malloc, stores it in a local variable forgetting the passed value, then returns leaking the memory.

If for some reason you want to do manual marshalling, you probably want something like this (assuming this is for Windows):

MYAPI BOOL __stdcall testStuff3( char** pp )
{
    if( nullptr == pp )
        return FALSE;   // null pointer
    if( nullptr != *pp )
    {   // Print & release an old string
        printf( "%s\n", *pp );
        CoTaskMemFree( *pp );
        *pp = nullptr;
    }
    // Allocate a new one
    const char* const str = CoTaskMemAlloc( 9 );
    if( nullptr == str ) return FALSE;
    strncpy( str, "Goodbye!", 9 );
    *pp = str;
    return TRUE;
}

C#:

public class Program
{
    [DllImport( "NativeLib.dll" )]
    private static extern bool testStuff3( [In, Out] ref IntPtr str );

    static void Main( string[] args )
    {
        IntPtr ptr = IntPtr.Zero;
        if( testStuff3( ref ptr ) )
        {
            Console.WriteLine( Marshal.PtrToStringAnsi( ptr ) );
            Marshal.FreeCoTaskMem( ptr );
        }
    }
}

However, this is not something I recommend doing unless you have very good reasons. In most cases automatic marshalling is better. For C# -> C++ way it's trivially simple, const char* or const wchar_t* in C++, string (with correct attributes) in C#. For C++ -> C# you can allocate a StringBuilder in C#, pass char* or wchar_t* to C++, and buffer length in another argument.

Answer 3

Thanks for the help!

Here's what I ended up with.

C++ function:

MYAPI int testStuff4(wchar_t* str)
{
    unsigned char* stuff = (unsigned char*)malloc(10);
    stuff[0] = 'G';
    stuff[1] = 'o';
    stuff[2] = 'o';
    stuff[3] = 'd';
    stuff[4] = 'b';
    stuff[5] = 'y';
    stuff[6] = 'e';
    stuff[7] = '!';
    stuff[8] = '\0';

    mbstowcs(str, (const char*)stuff, 1024);
    free(stuff);

    return 1;
}

C# function:

public class Program
{
    [DllImport("NativeLib.dll")]
    private static extern int testStuff4(IntPtr str);

    static void Main(string[] args)
    {
        IntPtr junk4 = Marshal.AllocHGlobal(1024);
        int ret = testStuff4(junk4);
        string junkString = Marshal.PtrToStringUni(junk4);
        Console.WriteLine(junkString);
        Marshal.FreeHGlobal(junk4);
    }
}