client getting a byte array returned from C++ COM dll

Question

I have this declaration in C++ COM header and IDL files:

//Header file:
#define MAX_LENGTH      320
typedef BYTE            PRE_KEY [MAX_LENGTH];

//IDL file:
#define MAX_COUNT       10
HRESULT Save([in] DWORD dwCommand, [in]float fdata[MAX_COUNT], [out] PRE_KEY* phKey);

This is the C# client code:

//After C# interop compilation, the method's signature in C# becomes:
Save(uint dwCommand, float[] fdata, out byte[] phKey);

//The code to call the C++ COM server:    
uint dwCommand = 2;
float[] fdata = new float[dwCommand];

fdata[0] = 1; 
fdata[1] = 2;

byte[] phKey = new byte[320];

save(dwCommand, fdata, out phKey);

The code will crash in ntdll.dll before the call returns to C#, but the C++ server has already finished processing and no longer in the stack.

Anyone can figure out how to resolve this issue? And as I am using interop compilation to compile the idl file to generate the C# signaure, so I can't do something in the C++ IDL file and manually change the C# signature.

And what is funny about this is I have another similar call which returns the exact same phKey from C++ to C# and it works perfectly. The only difference is in that call phKey is in a structure and the entire structure is an '[out]' param. Really can't see why this can be returned within a structure but not directly as a param.

Answer 1

The [out] attribute on your IDL declaration is a serious interop problem. It means that your COM server will allocate the array and the caller needs to release it. That very rarely comes to a good end, there is no guarantee whatsoever that your server and your client use the same heap. And will always fail when you use the C runtime allocator with the malloc() function or new[] operator, the CRT uses its own private heap, one that the caller can never get to unless they share the exact same version of the CRT. Odds are very small that this is the case in general, zero when you interop through the CLR.

Which is why it bombs, the CLR knows it needs to release the array after copying it to a managed array. It will use CoTaskMemFree(), using the heap that's reserved for COM interop allocations. Surely you didn't use CoTaskMemAlloc() to allocate the array.

The general solution to this problem is to have the caller supply the array, callee fills it in. Which requires [in, out] on the parameter. And an extra parameter that indicates the size of that passed array, [sizeis] to tell the marshaller about it. Very efficient, no allocation is required. Using the automation SAFEARRAY type avoids having to specify that extra argument, the CLR knows about that type.