c# to unmanaged c++ dll access violation / “frame not in module”

Question

I have a C# application (in Unity3d mono) that is passing some data to c++ unmanaged DLL via C-API. It is passing in nested Struct that includes an array. Everything is fine until the C-API is closing when (at the } of the function) when I then get "Frame not in Module" message from VS2017, (if I "> Continue" I will get an access violation 0xC0000005 which the disassembly suggest could be missing a ptr - so probably a dereferencing error)

EDIT: I think this could be that my source struct has a pointer to an array //EDIT

what causes this "Frame not in Module" ? - it's not very informative as a message

and what can I do about it?

My code...

I have ac# hierarchy of structs.

private struct Vector3
{
    public float x;
    public float y;
    public float z;
}

private structure Location
{
    public Vector3 coords;
    public float distanceFromOrigin;
}

private struct Locations
{
    [MarshalAs(UnmanagedType.ByValArray, ArraySubType = UnmanagedType.Struct, SizeConst = 10)]
    public Location[] locations;
}

private struct Scene
{
    public Locations cam1;
    public Locations cam2;
    public float timeInMilliSecs;
}

I'm instantiating the Scene and with cam1 and cam2 each gets an array of 10 Locations. Everything traces fine. Works as expected to this point and the data structure is populated with the correct data.

I'm passing the instance of Scene to an unmanaged DLL

[DllImport(dllname)]
private static extern void updateScene(Scene scene);

In my c++ I have

extern "C" {
    DLL_EXPORT void updateScene(Scene scene);
}

and an overload

void updateScene(Scene scene) {
    setSecene(scene); // this calls function fine but with erroneous data
}

and signatures for the equivalent structs

struct Vector3
{
    float x;
    float y;
    float z;
}

struct Location
{
    Vector3 coords;
    float distanceFromOrigin;
}

struct Locations
{
    Location locations[10];
}

struct Scene
{
    Locations cam1;
    Locations cam2;
    float timeInMilliSecs;
}

The Scene struct is passed into the C with the correct data an goes to the c++. But then the problem occurs - i'm guessing this is a memory issue, but I'm not sure what to do about it.

Looking at the Disassembly is possibly shows it's a pointer issue?

0000000040BDAE57 49 89 0B mov qword ptr [r11],rcx

Can anyone help, please?

Answer 1

Last time you say you don't change CallingConvention , so change the value to Cdecl .

Default value for DllImportAttribute is WinApi which doesn't clear the stack after function call.

And default value for C code is __cdecl which also doesn't clear the stack. It waits the caller to clear it.

That occurs the exception.

Answer 2

In the end I marshalled the whole nested structure across in a pointer - all changes in the c#

[DllImport(dllname, CallingConvention = CallingConvention.Cdecl)]
private static extern void updateScene(IntPtr scene);

And then, before I call updateScene() I'm marshalling the scene struct

IntPtr  scenePtr = Marshal.AllocHGlobal(Marshal.SizeOf(scene));
Marshal.StructureToPtr(scene, scenePtr , true);
updateScene(scenePtr);

everything else remained the same.