Error Bad Calling Convention when debugging vb6 program

Question

I have a standard VB 6 exe (mailviewer).
This program has a "link" to a cobol DLL:

Declare Sub InkMvwMail Lib "inkvwm" Alias "INKMVWMAIL" ...

When starting the normal exe from windows,

EVERYTHING WORKS FINE,

but when I want to debug the call to the cobol DLL entry point in Visual Studio 6.0 (SP6) (on windows xp), I get

"Error 49, Bad Calling Convention"

Thanks for any help in advance

Wolfgang

Answer 1

EVERYTHING WORKS FINE,

No, it only looks that way. Everything is not fine, that Cobol function was designed to be called from a C program. It has the wrong calling convention, cdecl instead of stdcall. The stack imbalance that's caused by this can cause extremely hard to diagnose runtime failure, like local variables mysteriously having the wrong value and includes the hard crash for which this site is named.

When you run from the IDE, the debugger performs an extra check to verify that the stack pointer is properly restored across the function call. It is not, thus generating the error 49 diagnostic.

You'll need to follow the guidance in this KB article . It cannot be solved in VB6, this requires writing a little helper function in another language that can make cdecl calls, like C or C++. The KB article shows what such a function could look like, although they intentionally gave it the wrong convention to demonstrate the problem.