Need Help Diagnosing a Failing P/Invoke Call Only on Windows XP

Question

I have the following P/Invoke defined:

[DllImport("helper.dll", CallingConvention=CallingConvention.Cdecl, 
    CharSet=CharSet.Ansi, EntryPoint="F_GetValue")]
private static extern Int32 _F_GetValue(String Formula, ref DATA_STRUCT Data,
    ref DATA_KEY DefaultKeyBuf, ref Double Result);

This call succeeds on Windows Vista and later, but fails on Windows XP with this memory exception:

A first chance exception of type 'System.AccessViolationException' occurred 
in helper.dll

I tried changing the first two "ref" modifiers to be [In, Out] but that did not solve the problem.

DATA_STRUCT and DATA_KEY are both structs that are instantiated and pre-populated.

Here is the C++ method definition that I am calling:

int F_GetValue(const char* pFormula, DATA_STRUCT* pData, 
    DATA_KEY* pDefaultKeyBuf, double* freturn)

I'm no P/invoke guru, so don't assume anything. Is there anything obviously wrong with the way this is defined? Is there more marshaling to be done (manually)? I feel like I might be missing something obvious.

EDIT: as requested, here are the structure definitions in .NET, the C++ F_GetValue() method, and the C++ structure definitions, respectively:

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Pack = 1)]
public struct DATA_STRUCT
{
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 260)]
    public String DataDir;
    [MarshalAs(UnmanagedType.U2)]
    public UInt16 LTType;
    [MarshalAs(UnmanagedType.U2)]
    public UInt16 FOMType;
    [MarshalAs(UnmanagedType.U2)]
    public UInt16 ResultType;
    [MarshalAs(UnmanagedType.ByValArray, SizeConst = 136)] // ( POS_BLOCK_SIZE + sizeof(int) + 4 )
    public Byte[] posBlock;
    public DATA_REC dataBuf;
    [MarshalAs(UnmanagedType.U2)]
    public UInt16 dataLen;
    [MarshalAs(UnmanagedType.U2)]
    public UInt16 keyNum;
    public DATA_KEY keyBuf;
    [MarshalAs(UnmanagedType.U2)]
    public UInt16 TNTC;
    [MarshalAs(UnmanagedType.I2)]
    public Int16 status;
}

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Pack = 1)]
public struct DATA_KEY
{
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 17)]
    public String LocName;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 17)]
    public String ParName;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 13)]
    public String DateTime;
}

int F_GetValue(const char* pFormula, DATA_STRUCT* pData, DATA_KEY* pDefaultKeyBuf, double* freturn)
{

    if (pFormula[0] == 0) // return quickly if nothing to do
    {
        *freturn = blank;
        pData->ResultType = sbit(DATA_BLANK);
        pData->status = B_NO_ERROR;
        return 0;
    }

    if ((_strnicmp(pFormula, "DOTNET_", 7) == 0) || (_strnicmp(pFormula, "(DOTNET_", 7) == 0)) // switch to/from dotnet
    {
        dotnetCalcs = (_strnicmp(pFormula + 7 + ((pFormula[0] == '(') ? 1 : 0), "ON", 2) == 0);
        *freturn = dotnetCalcs ? 1 : 0;
        pData->ResultType = sbit(DATA);
        pData->status = B_NO_ERROR;
        return strlen(pFormula);
    }

    BOOL bComingFromDotNet = (pData->dataLen == 65535);
    if (dotnetCalcs && (!bComingFromDotNet))
    {
        return F_GetValue2(pFormula, pData, pDefaultKeyBuf, freturn);
    }

    if (pSharedMem->bClient && ! bServer)
    {
        if (FromServer(ACTION_OPEN,NULL) == B_NO_ERROR)
        {
            ((CS_FORMULA *)pSharedMem->ClientServer)->nRecords = 1;
            strcpy(((CS_FORMULA *)pSharedMem->ClientServer)->DataDir,pData->DataDir);
            memcpy(&((CS_FORMULA *)pSharedMem->ClientServer)->Formula[0].DefaultKeyBuf,pDefaultKeyBuf,sizeof(DATA_KEY));
            strcpy(((CS_FORMULA *)pSharedMem->ClientServer)->Formula[0].Formula,pFormula);
            ((CS_FORMULA *)pSharedMem->ClientServer)->iType = CSTYPE_FORMULA;
            ((CS_FORMULA *)pSharedMem->ClientServer)->iAction = ACTION_READ;
            if (FromServer(ACTION_READ,NULL) == B_NO_ERROR)
            {
                *freturn = ((CS_FORMULA_RESULT *)pSharedMem->ClientServer)->Data[0].Data;
                pData->ResultType = ((CS_FORMULA_RESULT *)pSharedMem->ClientServer)->Data[0].ResultType;
                pData->status = ((CS_FORMULA_RESULT *)pSharedMem->ClientServer)->Data[0].status;

                FromServer(ACTION_CLOSE,NULL);
                return strlen(pFormula);
            }
            FromServer(ACTION_CLOSE,NULL);
        }
        *freturn = blank;
        return 0;
    }
    else
    {
        BOOL bOpenTemporaryData = bComingFromDotNet || (pData->dataLen == 0);
        if (bOpenTemporaryData)
        {
            DataOpenAndInitialize(pData,NULL);
        }

        int iReturn = F_DoGetValue(pFormula,pData,pDefaultKeyBuf,freturn);

        if (bOpenTemporaryData)
            DataExec(B_CLOSE,pData);

        return iReturn;
    }
}

typedef struct
{
    char DataDir[MAX_PATH];
    unsigned short LTType; 
    unsigned short FOMType; 
    unsigned short ResultType; 
    BTI_BYTE posBlock[POS_BLOCK_SIZE_];
    DATA_REC dataBuf;
    BTI_WORD dataLen;
    BTI_WORD keyNum;
    DATA_KEY keyBuf;
    unsigned short TNTC;
    BTI_SINT status;
} DATA_STRUCT;

typedef struct
{
char    LocName[LP_SIZE];
char    ParName[LP_SIZE];
char    DateTime[13];
} DATA_KEY;

For the sake of completeness, I also include this method, F_GetValue2(), which is called in F_GetValue(). And while this looks as though it might be running right back into managed code, it won't. This method exists for a different purpose, and I cannot assure you that it is not getting called in the case of my XP woes, as it would require that (dotnetCalcs && (!bComingFromDotNet)) be true, which it would not be.

One more thing, the other method that is called in there is F_DoGetValue(), and the set of parameters is then passed on to it. That method is huge, so I'll not post it here. But suffice it to say that it parses Formula and uses what it learns to call yet more methods that use the parsed string(s) to fetch data from the database, returning the Double member fReturn back up the chain until it ultimately is delivered back to the C# code via the marshaling.

Answer 1

   [StructLayout(..., Pack = 1)]

There is no #pragma pack visible in the C code, odds that the structure packing in your native code is actually 1 are low. The default is 8, same default for [StructLayout]. A minimum sanity check is that Marshal.SizeOf() on your structure types in C# returns the exact same value as sizeof() in your C code. It will not work correctly when there's a mismatch and a random AV is indeed likely.

And use the debugger to diagnose the AV. Project + Properties, Debug tab, tick the unmanaged code debugging option so you can debug both your C# and your C code. Set a breakpoint on first statement in the C function. And check that the debug view of the passed structure pointers matches the data you assigned in your C# code. Trouble is usually located near the end of the structure. Debug + Exceptions, tick the thrown box for Win32 exceptions to let the debugger stop when the AV exception occurs.

Answer 2

In the end, the problem turned out to be quite different from what evidence suggested (especially given my level of knowledge regarding P/Invoke). In fact, the problem was the result of using this declaration:

__declspec( thread ) BOOL dotnetCalcs = FALSE;

After stepping into the code, I found that it failed on this line:

if (dotnetCalcs && (!bComingFromDotNet))

The member "dotnetCalcs" is declared as thread local storage, and after doing some research, it appears that it is a known fail on XP. One example clue I found was the comment at the end of this MSDN page:

http://msdn.microsoft.com/en-us/library/9w1sdazb(v=vs.80).aspx

The part about delayed load is applicable in this instance, as the DLL in question is being loaded due to the DllImport.

Thank you to all who responded, and I'm sorry for having led a goose chase. But in the end, the trouble of setting up a special debug station proved worth the effort.

The fix:

Replacing the __declspec(thread) approach with calls to TLS methods. At DllMain() I establish a TLS index and hold it globally, setting the initial value with TlsSetValue(). All subsequent requesters are then using that index to retrieve the TLS value with TlsGetValue(). Whenever the value changes, just use TlsGetValue() again to set the value. Always remember to cast with LPVOID as that is the type these use.