RPC_E_SERVERFAULT when calling Visual FoxPro COM object (out-of-process) from .Net?

Question

I need to call a Foxpro (VFP 8) COM object from C#

The vfp code looks like this:

Define Class x2 As session OlePublic
    Function testInteropServer()
    ENDFUNC
Enddefine

The C# code looks like this:

[TestFixture]
public class TestFixture
{
    [Test]
    public void TestXtmBase()
    {
        xtmbase.x2 xtmBaseX2 = new xtmbase.x2();
        xtmBaseX2.testInteropServer();
    }
}

The COM server is compiled as an executable (not a dll). I can load it from Fox. It loads the object into .Net but I can't seem to call any of it's methods. I am importing the COM reference through the GUI in VS 2005. It recognizes all the methods and properties of the object. I just can't access them. I am getting the following error:

Test 'TestProject/TestFixture/TestXtmBase' failed:
    Execute
    System.Runtime.InteropServices.COMException: The server threw an exception.
        (Exception from HRESULT: 0x80010105 (RPC_E_SERVERFAULT))
    ErrorCode: -2147417851
    at xtmbase.x2Class.testInteropServer()

To remove the possibility of it being related to a COM executable I created a MT dll with the following code:

Define Class MTx2 As session OlePublic
  Function testInteropServer()
  ENDFUNC
Enddefine

I then created a console app:

class Program
{
    static void Main(string[] args)
    {
        mtx2.mtx2 mtx2 = new mtx2.mtx2();
        mtx2.testInteropServer();
        Console.WriteLine("Done");
    }
}

and it fails:

System.Runtime.InteropServices.COMException was unhandled
Message="The server threw an exception. (Exception from HRESULT: 0x80010105 (RPC_E_SERVERFAULT))"
Source="Interop.mtx2"
ErrorCode=-2147417851
StackTrace:
   at mtx2.mtx2Class.testInteropServer()
   at ConsoleTest.Program.Main(String[] args) in E:\development\iisx2\ConsoleTest\Program.cs:line 12
   at System.AppDomain._nExecuteAssembly(Assembly assembly, String[] args)
   at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
   at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

Any ideas why?

Answer 1

You are looking for the problem in the wrong process. It was the COM server that fell over on an exception. The RPC_E_SERVERFAULT error is merely a notification of that, a pretty useless one since all info about the exception was lost.

Attach a debugger to the server process before running the statement and have it stop on all exceptions so you can diagnose this. In VS that's done with Debug + Exceptions, tick the Thrown box for "Win32 Exceptions". Tools + Attach to Process.

The default handling is of course incredibly unhelpful. You can fix this in Vista and up with the IGlobalOptions interface . Use COMGLB_EXCEPTION_HANDLING to force the server to crash. This needs to be done in the server code, ought to be a bit of a problem in Foxpro I imagine.

Answer 2

It was due to DEP (Data Execution Prevention) (thanks pst) which seems to be a problem with Foxpro code.

You can manually add your app to the DEP exception list via the GUI, but I needed a programmatic way of doing it so:

Add the exe to the exclusion list using a registry key:
HKLM\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\AppCompatFlags\\Layers
"fullPathToExe"="DisableNXShowUI"

If DEP is active you have to reboot to make it work.