Passing LPTSTR Parameter to DLL gives Access Violation in C++ project

Question

I'm calling a function in an undocumented DLL which unpacks a file.

There must be a mistake in the way the header is declared / allocated but can't figure out what is going wrong.

Project character set in VS 2010 is Unicode.

Can call the DLL function succesfully from C# with the snippet below (but I need to make it work in c++):

[DllImport("unpacker.dll", EntryPoint = "UnpackFile", PreserveSig = false)]
internal static extern IntPtr UnpackFile(byte[] file, int fileSize,
[MarshalAs(UnmanagedType.LPWStr)] StringBuilder header, int headerSize);

If I uncomment of move one the headers an Acccess Violation pops up. The function also returns 0 which it doesn't in C#.

Any thoughts?

Code in the VC++ 2010 project:

// unpacker.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include <windows.h> 
#include <fstream>

using namespace std;

typedef void* (*UnpackFile)(unsigned char*, int, LPTSTR, int);

int _tmain(int argc, _TCHAR* argv[])
{
    LPTSTR header; 
    //Move this and get a access violation on the _UnpackFile(filetounpack... line

    static unsigned char *filetounpack; //Buffer to byte array with the file to unpack
    int filelen; //variable to store the length of the file
    HINSTANCE dllHandle; // Handle to DLL
    UnpackFile _UnpackFile; // Function pointer
    ifstream filetoread; //Stream class to read from files
    static LPTSTR header2;  //Buffer for the header 2nd


    filetoread.open ("c:/projects/testfile.bin", ios::in | ios::binary|ios::ate); 
    filelen = filetoread.tellg(); //read the length
    filetounpack = new unsigned char [filelen]; //allocate space

    filetoread.seekg (0, ios::beg); //set beginning
    filetoread.read ((char *)filetounpack, filelen); //read the file into the buffer
    filetoread.close(); //close the file

    dllHandle  = LoadLibrary(_T("unpacker.dll"));

    _UnpackFile = (UnpackFile)GetProcAddress(dllHandle, "UnpackFile");

    //header = new _TCHAR[filelen]; //Allocate memory for header
    header2 = new _TCHAR[filelen]; //Allocate memory for header

    //Access violation reading location 0xffffffffffffffff!!!
    void* tmp = _UnpackFile(filetounpack ,filelen ,header2 ,filelen); 

    delete[] filetounpack;
    delete[] header;
    delete[] header2;

    FreeLibrary(dllHandle); 
    return 0;

}

Answer 1

 typedef void* (*UnpackFile)(unsigned char*, int, LPTSTR, int);

That does not match the CallingConvention property of your C# declaration. The default for C# is StdCall, the default for native C++ projects is __cdecl. Fix:

 typedef void* (__stdcall * UnpackFile)(unsigned char*, int, LPTSTR, int);

And keep in mind that error checking is never optional in C++, you really do need to check if LoadLibrary() and GetProcAddress() succeeded. Automatic in C#, not in C++. Both functions return NULL when they fail.