MSVC和gcc之间的调用约定

Question

I have a program build with MSVC which is loading a dll dynamically. The dll provides a function which is called from the main program. If both are build with either MSVC or gcc everything is fine, but when I compile eg main with MSVC and the dll with gcc something is wrong.

#  ifdef __GNUC__
#    define CDECL __attribute__ ((__cdecl__))
#  else
#    define CDECL __cdecl
#  endif

struct EXP result {
    uint32_t code;
};

#define SUCCESS result{0};

virtual result CDECL foo(char const* const*& target) const {
    target = (char const* const*)0xAFFE;
    return SUCCESS;
}

The problem is, after the call target is zero instead of 0xAFFE . The main program is compiled using __cdecl as calling convention. The struct is packed (no alignment), but I also tried to align to different sizes (1, 2, 4, 8, 16). I also tried to use __declspec/__atribute__(dllexport) and different combinations of both variants.

If I take a look into the assembler code, there are two big differences:

; MSVC                          |   gcc
;===============================|================================
; before calling                |
;-------------------------------|--------------------------------
                                |   sub     dword ptr [esp+4],8
                                |
; foo();                        |
;-------------------------------|--------------------------------
push    ebp                     |   push    ebp
mov     ebp,esp                 |   mov     ebp,esp
mov     eax,dword ptr [target]  |   
mov     dword ptr [eax],0AFFEh  |   
mov     eax,dword ptr [ebp+0Ch] |   mov     eax,dword ptr [ebp+0Ch]
mov     dword ptr [eax],0       |   mov     dword ptr [eax],0AFFEh
                                |   mov     eax,0
pop     ebp                     |   pop     ebp
ret                             |   ret

Why is this even if I use the same calling conventions on both compilers ? And how do I do fix it?

Answer 1

The calling convention is pretty meaningless in this case. The issue is things like vtable layout. MSVC ABI and Itanium disagree about a lot of things. You can't compile a C++ interface and mix and match between compilers, unless it is explicitly supported. Clang and G++ should be interoperable if you set the right settings, and Clang and MSVC may be interoperable depending on which exact features you use.

Generally speaking, do not mix and match C++ compilers for C++ interfaces. It won't work.

Answer 2

Even if it took me one year to answer my own question (I'm a realy lazy person ;) , I would like to clarify this. As Puppy mentioned in his answer , the ABI between the compilers differs. But this is just half true. Since Microsoft invented COM back in 1992 they created an interface which can be mapped into c++ vtable interfaces. So other big compiler vendors implemented the mapping between COM and C++ vtables :

[...] Since a Windows compiler that can't use COM is pretty limited, other compiler vendors enforced the mapping between COM vtables and C++ vtables. [...]

Also as Remy Lebeau mentioned in his comments above:

The function is returning a structure. More likely, msvc and gcc do not agree on how to pass that structure around, whether on the call stack or in registers , etc. Returning a struct from a function is not portable. The calling convention dictates how parameters are passed, but does not dictate how non-trivial return values are passed. The portable solution is to either return just the uint32_t by itself, or else pass the struct into the function as a pointer parameter and let the function fill it as needed.

[...] returning just a pointer by itself would be covered by the rules of the calling convention. Trivial built-in types (integers, floats/doubles, pointers, etc) are covered. User-defined types (classes/structs) are at the discretion of each compiler, typically due to differences in how they optimize their code.

So, as the return value is not covered by the rules of the convention, only trivial types should be used as return values.

Mentionable readings:

• https://msdn.microsoft.com/en-us/library/windows/desktop/ms680573(v=vs.85).aspx
• https://itanium-cxx-abi.github.io/cxx-abi/abi.html#vcall
• https://chadaustin.me/cppinterface.html